MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — part 2/3

Cleaned-text code points [15474, 31473). Bibliography omitted here; source.md includes it.

We find that o1-preview significantly outperforms all other models, achieving a medal on 16.9% of competitions - almost twice the number of medals on average as the next best model ( Table 2 ). We note that qualifying as a Kaggle Grandmaster 10 10 10 See Kaggle Progression System here: https://www.kaggle.com/progression requires 5 gold medals, while o1-preview achieves an average of 7 gold medals on MLE-bench. This comes with the following caveats: not all our chosen competitions are medal-granting, MLE-bench uses slightly modified datasets and grading, and agents have the advantage of using more recent technology than the participants in many cases.

##### Discussion

All agents often failed to create valid submissions, despite having access to the validation server. When analyzing agent transcripts we found that they did not always use the validation server, despite their prompts encouraging them to do so.

We found that MLAB and OpenHands tend to end their runs early, sometimes within the first few minutes, despite being told to optimize their scores as much as possible for the full 24-hour duration. In contrast, the AIDE scaffold repeatedly prompts models to improve their score until the full 24 hours is up, or when it has generated 500 nodes (the maximum we allow).

Small details in scaffold implementations can make a big difference. MLAB and OpenHands are given a variety of tools to solve open-ended tasks, though this flexibility also increases the risk surface area for failure. For example, MLAB often attempted to inspect files that were thousands of lines long, which ended up filling its context window. We fixed many of the most obvious failures in each agent (detailed in Appendix A.6 ), but expect failure modes to remain.

All three agents failed to effectively factor in compute and time limitations to their strategies. For example, they would execute commands that overload the machine’s disk or RAM, resulting in their process getting killed and their run finishing early. Additionally, agents rarely verbalized any consideration of how long their produced code would run for.

See Table 9 and Table 10 for our headline results broken down by complexity level and task category respectively. We also visualize performance on individual competitions as a function of the competition date for various models in Figure 9 .

### 3.2 Increasing number of attempts

To see how performance changes with more attempts, we evaluate GPT-4o (AIDE) and o1-preview (AIDE) using the pass@ k metric ( Chen et al., 2021 ) . We estimate the percentage of competitions in which the agent achieves a medal given k k attempts at each competition, drawn from n n seeds:

pass@ ​ k := 𝔼 Competitions ​ [ 1 − ( n − c k ) ( n k ) ] \text{pass@}k:=\underset{\text{Competitions}}{\mathbb{E}}\left[1-\dfrac{\binom{n-c}{k}}{\binom{n}{k}}\right]

The main result for k ∈ [ 1 , n 2 ] k\in\left[1,\frac{n}{2}\right] is shown in Figure 3 . We see a consistent increase in the percentage of medals achieved as the number of attempts increases for both agents.

### 3.3 Varying amount of compute available

Our main experiments give agents access to a single 24GB A10 GPU, whereas Kaggle provides a free 16GB P100 GPU to users, who often also use their own hardware to compete. In this experiment, we investigate how agents’ performance may be affected by our choice of hardware, or if they may even adapt their strategies depending on the hardware available (e.g. training smaller models when only CPU(s) are available, and training larger models when GPU(s) are available).

We compare the performance of GPT-4o (AIDE) on three different hardware setups, varying between our standard setup with a single A10 GPU, CPU-only, and two A10 GPUs.

Surprisingly, we find that GPT-4o (AIDE) performs similarly in all experiments. We manually viewed several logs from the “Extra GPU” experiment, and didn’t identify any attempts from GPT-4o AIDE to use the second GPU. See Table 3 for results.

### 3.4 Increasing time available

Our main experiments limit the amount of time per competition attempt to 24 hours; in contrast, human contestants often have weeks or months to work on Kaggle competitions. Increasing the time limit gives agents more time to iterate on their solutions, and permits more time for model-training. We run an experiment providing GPT-4o (AIDE) with a longer time limit of 100 hours per competition and increase the maximum nodes allowed by 10x to 5,000. We take snapshots of the agent’s best attempt every hour and grade each snapshot. See Figure 5 for the results.

We find that GPT-4o (AIDE) achieves a significant number of medals in the first couple hours of execution, then slowly accumulates more medals over the course of the run. Sometimes the number of medals GPT-4o (AIDE) achieves actually decreases, which is due to the imperfect method AIDE uses to select its “best" attempt.

## 4 Contamination & Plagiarism

Dekoninck et al. (2024) define contamination as artificially inflated and non-generalizing benchmark performance. For MLE-bench, we are most concerned about results being artificially inflated due to models developing solutions derived from public solutions of a given competition.

In the following sections, we investigate the effect of contamination by measuring the correlation between GPT-4o’s familiarity with competition winners’ discussion posts and its performance. We also run a variant of the benchmark with obfuscated instructions to make each competition less recognizable. Our experiments find no evidence of results being systematically inflated due to memorization.

In addition to investigating contamination, we run the plagiarism detector on all medal-winning submissions and find no evidence of plagiarism (Appendix A.4 ). We also run the log analysis tool and manually inspect any flagged violations but find no cases of rule-breaking (Appendix A.3 ).

### 4.1 Familiarity with top solutions

Blatant plagiarism can be detected using off-the-shelf detection tools, but contamination may have subtler effects if models have trained on discussions of winning solutions and adopt their high-level strategies, which could still lead to non-generalizing performance on new ML engineering tasks.

We investigate this effect in GPT-4o’s base model by measuring its familiarity with the competitions and their winning strategies. Previous work suggests that models place higher probabilities on the tokens of documents seen during training ( Carlini et al., 2023 ) . Thus, we define a model’s familiarity with a given document as the mean probability a model assigns to each token in that document, conditional on all preceding tokens. For each competition, we calculate the model’s mean familiarity with the main competition page and the 5 most popular discussion posts 11 11 11 Typically, these are posts by the competition winners sharing their approach. for that competition.

Figure 5 shows the result of this analysis. We find no correlation between the familiarity of GPT-4o’s base model with a competition and its performance on that competition.

### 4.2 Obfuscating competition descriptions

We run an additional experiment to investigate how contamination might affect our results: If models rely on matching familiar problems to memorized solutions, making the competitions unrecognizable may mitigate this effect.

We manually rewrite the competition descriptions of all 75 competitions in MLE-bench to obfuscate the provenance of each competition whilst retaining the key information. For example, we remove all references to Kaggle and the competition’s name, and cut out text that is not strictly required. See Appendix A.8 for an example obfuscated description.

We run GPT-4o (AIDE) with 10 seeds on these obfuscated descriptions. We see that GPT-4o (AIDE) achieves similar scores on both the original and obfuscated competition descriptions. See Table 4 for results.

In summary, our experiments suggest that GPT-4o’s familiarity with Kaggle competitions does not systematically inflate its scores. Furthermore, we find no evidence of GPT-4o being over-reliant on the original form of the competition descriptions. This does not rule out subtler effects of contamination, but our findings suggest that contamination effects are minimal on our results.

## 5 Related Work

Evaluating Software Engineering Capabilities . Chen et al. (2021) ; Hendrycks et al. (2021) ; Austin et al. (2021) ; Jain et al. (2024) evaluate models’ abilities to produce code following a natural language description. Frontier models are saturating many of these benchmarks 12 12 12 AgentCoder ( Huang et al., 2024a ) achieves 96.3% and 91.8% on HumanEval and MBPP respectively. , yet have failed to automate the job of a software engineer. SWE-bench ( Jimenez et al., 2024 ) tasks models to solve real-world pull requests from open-source repositories. Despite its challenging nature, performance on SWE-bench has been steadily increasing ( Zhang et al., 2024 ; factory.ai, 2024 ) . In contrast, the problems in MLE-bench are often more open-ended and difficult (for example, some are open research problems). However, MLE-bench may similarly see rapid progress as in SWE-bench, making it important to measure early.

Evaluating ML Engineering Capabilities . MLE-bench is not the first benchmark to use Kaggle competitions to measure autonomous ML engineering capabilities. MLAgentBench ( Huang et al., 2024b ) takes 13 tasks from Kaggle and bespoke ML tasks, provides a simple baseline solution for each, and evaluates how often agents can achieve at least a 10% improvement over the baseline solution. In contrast, MLE-bench provides significantly more tasks with more complexity, and requires agents to attempt the task from scratch.

Another benchmark, ML-Bench ( Tang et al., 2024 ) , tests agents’ abilities to generate code and execute commands to interact with popular ML repositories. Compared to MLE-bench, ML-Bench measures understanding and effective application of pre-existing codebases rather than developing ML solutions to open-ended problems.

Weco AI’s report of AIDE ( Schmidt et al., 2024 ) claims to beat > > 50% of human competitors on data science competitions from Kaggle. We find state-of-the-art models available at the time of AIDE’s announcement would only surpass the median score in MLE-bench ∼ \sim 10% of the time, far short of 50%. We take this as evidence that our selection of competitions is more difficult than Weco AI’s.

In work concurrent to ours, DSBench ( Jing et al., 2024 ) also introduces a benchmark of Kaggle competitions, but much like Weco AI’s dataset, DSBench focuses on data science tasks. There is some overlap between our datasets, but DSBench’s filtering criteria removes any competitions whose datasets do not fit a simple template that is used to automate task creation. This precludes many interesting competitions with non-standard formats. In contrast, each competition in MLE-bench has been manually ported over by our team, resulting in more diverse and challenging tasks.

Evaluating AI Agents . SWE-bench, MLAgentBench, and MLE-bench are multi-step benchmarks evaluating AI agents in the software domain. Here, components such as LMs, retrieval, and external tools are “scaffolded” together via code, unlocking new levels of autonomy unattainable via a single inference call ( Zaharia et al., 2024 ) . AgentBench ( Liu et al., 2023 ) provides environments for agents to complete multi-turn challenges, such as editing permissions on a Linux OS. GAIA ( Mialon et al., 2023 ) focuses on agent interactions with the real world, providing 466 questions that are conceptually simple for humans but challenging for current AI systems. Gioacchini et al. (2024) propose AgentQuest, a modular agent evaluation framework designed for extensibility, and Kapoor et al. (2024) provide an analysis of agent evaluation efforts so far.

## 6 Limitations

Contamination & plagiarism. Since our dataset consists of public Kaggle competitions (Appendix A.7 ), it’s possible that models have trained on all public Kaggle material including competition details, solutions, and even the datasets including our test set 13 13 13 In early experiments, we found GPT-4’s base model could reproduce several rows from the dataset of the “Titanic - Machine Learning from Disaster” competition when given the first few rows as a prompt. . There is therefore a risk that models have memorized answers or intuitions about the solutions such that MLE-bench over-represents model capabilities. We have mitigations in place to prevent plagiarism of the top participants’ code or test labels (log analysis and plagiarism detector), but it is difficult to detect the reuse of high-level strategies. Our experiments (Section 4 ) find no systematic effect of contamination for GPT-4o, but make no guarantees about future models. Future work may seek to regularly update MLE-bench with new Kaggle competitions to stay ahead of contamination issues.

Coverage of AI R&D capabilities. We built MLE-bench to better understand the risk of AI R&D acceleration via automated ML engineers, but the tasks included in MLE-bench don’t cover the full spectrum of capabilities required for AI R&D. MLE-bench selects for Kaggle competitions that provide clear problem statements, datasets that are clean and well-documented, and have clear metrics for optimization. On the other hand, real-world AI R&D often may not even have a clear problem statement, and figuring out the dataset and metrics is part of the problem. Nevertheless, MLE-bench evaluates many core competencies involved in AI R&D, including preparing large multi-modal datasets, managing long-running training scripts, and debugging poor-performing models.

Differences to real competitions. MLE-bench uses different train-test splits to the original competitions on Kaggle and re-implements their grading code. This raises concerns about how comparable our scores are to the human leaderboards from Kaggle. We have been careful to implement our competitions in a way that the new train and test sets retain a similar distribution as the original sets, and confirm that sample and gold submissions lead to results consistent with the human leaderboard. A further concern is that algorithmic progress may result in older competitions being easier, as agents with today’s knowledge and tools have advantages over the original participants. To account for this, we label competitions with complexity levels from the point of view of an ML engineer today, and we may yet need to update the complexity annotations as capabilities advance.

Accessibility : MLE-bench is a particularly resource-intensive benchmark to run. A single run of our main experiment setup of 24 hours per competition attempt requires 24 ​ hours × 75 ​ competitions = 1800 ​ GPU hours of compute 24\text{ hours}\times 75\text{ competitions}=1800\text{ GPU hours of compute} . Furthermore, running agents for the whole duration is very token-intensive. In our experiments, o1-preview with AIDE used 127.5M input tokens and 15.0M output tokens on average for one seed of 75 competitions.

## 7 Conclusion

We introduce MLE-bench, a benchmark designed to evaluate AI agents on ML engineering tasks using challenging Kaggle competitions. By closely simulating the experience of participating in a Kaggle competition, MLE-bench enables a direct comparison between agents and human competitors. Our experiments show that frontier models combined with agent scaffolding – specifically, o1-preview with AIDE – can achieve a medal in 16.9% of competitions. By open-sourcing MLE-bench, we aim to facilitate further research in evaluating ML engineering capabilities of agents. Ultimately, we hope our work contributes to a deeper understanding of the capabilities of agents in autonomously executing ML engineering tasks, which is essential for the safe deployment of more powerful models in the future.