MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — part 3/3

Cleaned-text code points [31473, 43161). Bibliography omitted here; source.md includes it.

##### Ethics Statement

If AI agents become capable of autonomously performing ML research, they could have numerous positive impacts, such as accelerating scientific progress in healthcare, climate science, and other domains, accelerating safety and alignment research of models, and fostering economic growth through the development of new products. The capacity of agents to perform high-quality research could mark a transformative step in the economy.

However, agents capable of performing open-ended ML research tasks, at the level of improving their own training code, could improve the capabilities of frontier models significantly faster than human researchers. If innovations are produced faster than our ability to understand their impacts, we risk developing models capable of catastrophic harm or misuse without parallel developments in securing, aligning, and controlling such models.

We believe a model capable of solving a large fraction of MLE-bench likely possesses the capability to execute many open-ended ML tasks. We are open-sourcing MLE-bench to aid research into the agentic capabilities of language models and increase transparency into acceleration risks at research labs. As we do so, we recognize the limitations of MLE-bench and strongly encourage the development of more evaluations of automated ML research capabilities, especially those more specific to the workflow of researchers training large language models.

Our benchmark uses publicly available Kaggle competitions. No sensitive data is used, and code is provided to allow users to reproduce datasets in a way that complies with relevant licenses.

##### Reproducibility Statement

We have taken care to ensure that our setup is fully reproducible. We provide all necessary details for reproducing our results, including dataset curation, evaluation metrics, and experimental setup. Our codebase is publicly available, including code for reproducing the full benchmark and experiments. The code for scalably running agents is infrastructure-specific so not included, but we provide examples for running agents on MLE-bench which can be adapted to the user’s own infrastructure. As discussed in Section 6 , users may find it difficult to fully reproduce our experiments due to compute and token costs.

## Appendix A Appendix

### A.1 Dataset Curation Criteria

We manually filter candidate competitions according to the following criteria. Each competition in our final set was screened by at least two ML engineers working at leading AI companies.

1. The competition requires ML engineering capabilities in order to achieve a medal, specifically those relevant for modern-day ML.

2. The competition description is well-specified enough to be solvable, i.e. there are no obvious missing components or crucial information that is inaccessible. We found the description to be detailed and thorough without any major ambiguities about how to approach that might only be resolved in the Discussion tab or external materials.

3. The competition’s evaluation metric can be computed locally.

4. The competition must have finished and is therefore unlikely to change (and, for some competitions, the test set is now publicly available).

5. The dataset isn’t used extensively outside of Kaggle (e.g. we avoid datasets like MNIST).

6. The train and test sets are from the same distribution, such that it is feasible to create a new train and test split from the public training data.

7. The final submission must be a CSV file (or, in the case of code competitions, must produce a CSV when the submitted notebook is run).

8. The competition doesn’t require downloading data from websites other than Kaggle.

9. The dataset’s license doesn’t restrict its inclusion in our benchmark.

### A.2 Distribution of Competitions

We provide a high-level overview of the problem category and complexity level distributions in MLE-bench in Figure 6 . Both the problem category and complexity were manually labeled.

### A.3 Runs analysis

We provide a code and log analysis tool that processes logs and code outputs from agent runs with GPT-4o mini 14 14 14 gpt-4o-mini-2024-07-18 using the rubric of questions in Table 5 .

We run the code and log analysis tool on all medal-winning submissions. See Table 6 for results of the analysis tool.

We found gpt-4o-mini to be overly cautious, flagging valid behaviors as violations even when they were not. The tool detected violations in the o1-preview AIDE and gpt-4o OpenHands runs; however each violation was determined to be a false positive after human review.

### A.4 Plagiarism detection tool

We use the source code plagiarism detection tool Dolos ( Maertens et al., 2024 ) to check submissions for plagiarism of top associated notebooks from each Kaggle competition.

The Dolos algorithm is explained in https://dolos.ugent.be/about/algorithm.html . To summarize: Dolos tokenizes code to make plagiarism detection invariant to the specific naming of variables and functions. It then fingerprints files according to sub-sequences of k-length tokens (we use the default setting of k=23). Pairs of files (in our case, a file from a submission and a notebook from Kaggle) are given a similarity score based on the fraction of shared fingerprints between the two files.

We surface any submission file with a similarity score above 60% for human review of potential plagiarism, finding no detected cases of plagiarism.

### A.5 Setup details

In our experiments, unless otherwise stated, each agent is executed within a Microsoft Azure Standard_NV36ads_A10_v5 virtual machine, which has 36 AMD EPYC 74F3v (Milan) [x86-64] vCPUs, 440GB memory, and one Nvidia A10 GPU (24GB). Each agent has full write access to a 4095 GiB SSD which it uses to produce its final submission and create any intermediate files. We use Docker bind mounts to give agents access to the relevant datasets for the competition they are solving.

We execute agents within Docker containers with the sysbox runtime. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security, and our cluster environment is itself designed to be robust. Each agent is executed using a Python virtual environment containing packages necessary and useful for the agents. We passed the instructions directly to AIDE and OpenHands on execution, and instructed MLAgentBench to read a file including the instructions.

### A.6 Scaffold details

Table 7 details the hyperparameters for each of our 3 tested scaffolds: the data science agent AIDE from Schmidt et al. (2024) , OpenHands’ CodeActAgent ( Wang et al., 2024 ) , and MLAgentBench’s ResearchAgent ( Huang et al., 2024b ) .

For all scaffolds we ensure that the agent is prompted with the total time available to complete the task, the timeout any code execution has before being forcibly terminated (if any), and the hardware available. See Figure 8 for the prompt used for all agents. We additionally insert information about the time and steps remaining into the end-of-turn prompt for all scaffolds.

In the following, we describe any further scaffold-specific modifications to address common pitfalls and improve the robustness of the agents.

#### A.6.1 AIDE modifications

• Implement exponential backoff on API call rate limits to handle high traffic scenarios.

• Add strict: True to Function Calling outputs to ensure stricter enforcement of output format rules, preventing invalid feedback responses.

• Add support for the Gemini and OpenRouter backends, and improve the Anthropic backend to enhance overall backend logic and response handling.

• Update internal prompts to emphasize the importance of creating a valid submission.csv file, as this was previously under-emphasized.

• Truncate excessively long data previews to prevent overwhelming the agent.

• Handle cases where input files are saved as .json instead of the required .jsonl format to prevent errors when processing submissions.

• Actively track whether a solution generates a submission.csv file; flag solutions that fail to produce this file as buggy.

• Modify solution selection criteria to consider not only performance metrics but also whether a valid submission.csv file was created.

• Add an option to obfuscate references to Kaggle, used in subsection 4.2 .

#### A.6.2 OpenHands modifications

As our cluster infrastructure requires agents to be packaged within Docker containers, and OpenHands also manages its own sub-containers, we end up with a Docker-in-Docker situation when running OpenHands. We make modifications to the OpenHands Docker configuration to enable GPU passthrough to the code agent, enable full use of the host disk space, and set the RAM allowance to 100GiB. We make no modifications to the scaffold in terms of tooling or behavior.

#### A.6.3 MLAB modifications

• Add "Validate Submission" tool as a low-level action available to the agent, with the following description: "Use the benchmark-provided tool to validate the format of your submission. You must provide the path to a submission file."

• Add automatic retries with the tenacity library to the complete_text_openai function for any OpenAI API Errors; return the error message to the agent after a maximum of 10 retries.

• To discourage agents from ending their runs early, we modify the description of the “Final Answer” tool to make it clear that the agent should not use this tool unless it has exhausted all avenues for improving their solution.

• Include full error messages in all EnvException exceptions so that the agent can debug more effectively.

• Fixed an edge-case bug in the parse_action_input method.

• Truncate observations with "…TRUNCATED" when the “List Files” tool exceeds 10,000 characters.

• Truncate observation with "File too large, only showing the first 10 blocks." when the “Understand File” tool exceeds 10 blocks.

• Truncate summarize_observation function with ""WARNING: Reached maximum number of chunks (100), this summary of the observation will be incomplete. Please consider trimming down your action request to avoid overloading the observation response."" when it exceeds 100 summary chunks.

### A.7 Dataset

We provide the full list of competitions in Table 8 , with notes on how we created a new test split from the publicly available training data.

### A.8 Obfuscated Descriptions

Below we compare the description for a arbitrarily chosen competition (champs-scalar-coupling) in MLE-bench to the obfuscated version of the description used in the obfuscating competition descriptions experiment of Section 4.2 .

### A.9 Performance by complexity, category, and competition date

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .