MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

arxiv:2410.07095

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2024-10-09

Cites (0) · Cited by (8)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• Abstract
• 1 Introduction
• 2 MLE-bench
• 2.1 Dataset Curation
• 2.2 Metrics
• Leaderboards
• Medals
• Headline metric
• Raw Scores
• 2.3 Setup
• Validating submissions
• 2.3.1 Rules
• Rule-breaking detection
• Plagiarism detection
• 3 Experiments and Results
• 3.1 Main experiment
• Varying scaffolding
• Varying models
• Discussion
• 3.2 Increasing number of attempts
• 3.3 Varying amount of compute available
• 3.4 Increasing time available
• 4 Contamination & Plagiarism
• 4.1 Familiarity with top solutions
• 4.2 Obfuscating competition descriptions
• 5 Related Work
• 6 Limitations
• 7 Conclusion
• Ethics Statement
• Reproducibility Statement
• Appendix A Appendix
• A.1 Dataset Curation Criteria
• A.2 Distribution of Competitions
• A.3 Runs analysis
• A.4 Plagiarism detection tool
• A.5 Setup details
• A.6 Scaffold details
• A.6.1 AIDE modifications
• A.6.2 OpenHands modifications
• A.6.3 MLAB modifications
• A.7 Dataset
• A.8 Obfuscated Descriptions
• A.9 Performance by complexity, category, and competition date
• Instructions for reporting errors