What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

arxiv:2511.15593

Original source · Paper JSON

Agents, open-endedness, AGI · 2025-11-19

Cites (1) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity
• Abstract
• 1 Introduction
• 1.1 Contributions
• 2 Research Agents and MLE-bench
• Agents
• MLE-Bench
• 3 Methods
• 3.1 General Setup
• 3.1.1 Dataset
• 3.1.2 Metrics
• 3.1.3 Agentic Scaffolds
• 3.1.4 LLM Backbones
• 3.2 Measuring Ideation Diversity
• 3.3 Controlling Ideation Diversity
• 3.3.1 Baseline agents
• 3.3.2 Agents with ablated diversity
• 4 The Ideation Diversity Bottleneck
• 4.1 Deep-dive on Agentic Trajectories
• The agent scaffold choice impacts ideation diversity
• Diversity correlates with performance on MLE-bench
• 4.2 Impact of Diversity: A Controlled Experiment
• 4.2.1 Are We Actually Influencing Diversity?
• 4.2.2 Results
• 4.3 Evaluating with Alternative Metrics
• 4.3.1 Alternative Metrics
• 4.3.2 Data Analysis Results with Additional Metrics
• 4.3.3 Controlled Experiment Results with Additional Metrics
• 5 Discussion
• What does it take to be a good AI research agent?
• Importance of the implementation bottleneck.
• Generalization to other benchmarks.
• Limitations of MLE-bench evaluations.
• Limitations of this study.
• 6 Related Work
• Generation diversity in language models.
• Diversity in reinforcement learning and population-based reinforcement learning.
• Diversity in multi-agent systems.
• Automated machine learning and AI research agents.
• 7 Conclusions
• Appendix A Appendix
• A.1 Controlling Ideation Diversity Using Temperature
• A.2 Alternative Metrics
• A.2.1 Limitations of MLE-Bench Medal System
• Variable medal thresholds
• Narrow thresholds between medals and top scores
• Different test sets for evaluation
• Competition age
• A.2.2 Evaluation Principles
• Limited Set of Metrics
• Inclusion of all attempts
• Independence from Human Score Distributions
• Capturing the complexity of hill-climbing
• A.3 Models Used By Agents For Image Classification Tasks
• Instructions for reporting errors