What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity — part 3/3

Cleaned-text code points [31783, 42783). Bibliography omitted here; source.md includes it.

##### Diversity in reinforcement learning and population-based reinforcement learning.

Trajectory diversity is synonym of increased exploration in reinforcement learning models. Hong et al. (2018) investigate a diversity-driven exploration strategy for training reinforcement learning models. Eysenbach et al. (2019) propose a diversity objective to learn skills without reward functions. Parker-Holder et al. (2020) improve diversity in population-based reinforcement learning, while Conti et al. (2018) propose a novelty objective in a population of agents to improve exploration in reinforcement learning. More recently, Yao et al. (2025) propose a diversity-aware policy optimization algorithm; unlike the works cited above, it does so in the context of LLMs. Zeng et al. (2025) propose B-star, a reinforcement learning approach for reasoning LLMs that balances exploration and exploitation.

##### Diversity in multi-agent systems.

Diversity has also been studied in the context of multi-agent foundation models ( Tuyls, 2023 ) . Bettini et al. (2025) study the impact of behavioral diversity in multi-agent reinforcement learning. Li and Zhu (2025) address how agents in multi-agent RL often end up learning nearly identical behaviors when they share network parameters, which hurts exploration. The authors propose CTEM, a method that uses trajectory entropy maximization to push agents toward more diverse behaviors without needing complex density models. Li et al. (2025a) find that generating diverse teammates in multi-agent training can lead to random, semantically meaningless behaviors, reducing training efficiency. Their SemDiv approach leverages large language models to describe coordination strategies in natural language, then converts these into reward functions for training meaningful teammate policies. In LLM-agent world simulations, Chu et al. (2025) investigate prompt design’s impact on conversational diversity and introduce a prompt-tuning mechanism that controls diversity via a single parameter.

##### Automated machine learning and AI research agents.

In the era of tool-use LLM-based agents ( Schick et al., 2023 ; Kaddour et al., 2023 ) , Shen et al. (2023) ; Nathani et al. (2025) evaluate agents at implementing machine learning tasks. MLE-bench ( Chan et al., 2025 ) , the benchmark used in this work, is a machine learning benchmark consisting of 75 Kaggle tasks. Zhao et al. (2025) propose a self-contained machine learning task with a focus on language models. AIRA ( Toledo et al., 2025 ) , studies AI research agents by formalizing AI research as search policies over a space of candidate solutions. This development is parallel to agentic benchmarks and scaffolds for other fields, such as software engineering ( Jimenez et al., 2024 ; Jiang et al., 2025b ) .

## 7 Conclusions

This work started from the hypothesis that ideation diversity is a key bottleneck in AI research agents’ performance. We have confirmed this hypothesis by conducting a large-scale analysis on AI research agents’ trajectories and performing a controlled experiment. Our findings hold across several evaluation metrics.

In future work, we suggest focusing on diversity-aware methods, as other bottlenecks such as implementation quality will decrease in importance when AI systems keep improving. We also recommend considering multiple evaluation metrics and extending the existing benchmarks to more recent machine learning tasks.

## Appendix A Appendix

### A.1 Controlling Ideation Diversity Using Temperature

With the intuition that the temperature sampling parameter has an impact on ideation diversity and performance, we run several experiments intervening on the sampling temperature. To isolate the impact of temperature, we use AIRA Greedy \text{AIRA}_{\text{Greedy}} but remove all mechanisms that enhance ideation diversity (sibling memory, prompt-adaptive complexity, mention of diversity in the system prompt), as described in section 3.3 , and experiment with different temperatures, above and below the recommended temperature of 0.6 for DeepSeek-R1 ( DeepSeek-AI et al., 2025 ) .

Figure 12 shows that changing temperature does not have an impact on performance (neither beneficial nor detrimental), assessed as medal rate. Results hold across alternative metrics like valid submission rate, average normalized score, and percentile, except for Elo, the only metric where increased temperature significantly leads to improved performance. Despite our effort in isolating ideation diversity, we hypothesize that the reason for these negative results is that modifying the temperature parameter (instead of using the recommended one) also affects the agent in additional manners ( Renze, 2024 ; Wu et al., 2025 ; Li et al., 2025b ) . For example, we would expect the implementation capabilities to also be affected, 2 2 2 Surprisingly, Renze (2024) found no statistically significant effect in problem solving skills of LLMs for certain temperature ranges. and there could be second-order effects that are hard to reason about. We leave further investigating the temperature-based results as future work.

### A.2 Alternative Metrics

#### A.2.1 Limitations of MLE-Bench Medal System

Performance on MLE-bench has traditionally been evaluated using Kaggle’s medal system, where medals are awarded based on score percentiles (see details here). For instance, in competitions with fewer than 99 teams, gold medals are given to the top 10% of submissions. However, several limitations are inherent to this medal-based evaluation framework.

##### Variable medal thresholds

Kaggle medal criteria vary with the number of submissions (e.g., bronze is the top 10% for competitions with 1000+ teams vs. top 40% for those with 1-249 teams; see Figure 13 ). This inconsistency means that medals do not equate to the same performance level across different competitions.

##### Narrow thresholds between medals and top scores

In some competitions, there is only a minimal difference between the threshold for earning a medal and the highest score achieved. Specifically, as shown in Figure 13 , in about 30% of MLE-bench competitions, the score (relative) difference between the best score and the bronze medal threshold is less than 3%. In around 50% of all MLE-bench lite competitions, this gap is similarly small.

##### Different test sets for evaluation

AI research agents are tested on custom sets designed by MLE-bench, not the private test sets used for human submissions on Kaggle. Human submissions leading to medals or best scores would perform differently on custom test sets for AI research agents, while the medal thresholds are still computed based on submissions on Kaggle private test sets. This variability is an issue, especially when medal thresholds are set close to each other, and could make medal thresholds unreachable by AI research agents.

##### Competition age

Some competitions are more than 10 years old. In this rapidly evolving space, human score distributions don’t reflect what they would be now in 2025 (for example agents perform better than top human scores on ’detecting-insults’ competition). For recent competitions, after 2022, we see a substantial decrease in performance, and all agents tested are not able to get any medal for most competitions after 2022.

These limitations suggest that the medal rate metric, by itself, may not offer a comprehensive evaluation of agent performance on MLE-bench. Therefore, in the following section, we describe alternative metrics to provide a more complete assessment of performance on MLE-bench. We used these metrics for the results in Section 4.3 .

#### A.2.2 Evaluation Principles

Evaluations serve as a measure of a model’s capabilities. For MLE-bench, we aim to evaluate our AI research agent’s ability to address a wide range of machine learning tasks, and manage key stages of the machine learning lifecycle in order to get the best performance, including data cleaning & preprocessing, model development & tuning, validation. We describe here a list of principles to consider for additional metrics.

##### Limited Set of Metrics

We have shown that relying solely on medal rates was hindering our assessment of performance on MLE-bench. We need more metrics to get a comprehensive view of the performance of AI research agents. However, relying on a large set of metrics can make the assessment of an agent difficult, and hard to compare with others.

##### Inclusion of all attempts

Failed submissions need to be taken into account in the evaluation. If an agent gets a perfect score on 50% of tasks and fails to get a valid submission on the other 50% of tasks, the agent should theoretically not get a perfect score.

##### Independence from Human Score Distributions

Agents and humans are not evaluated on the same test sets.

Medal rates only value marginal improvements on good scores.

##### Capturing the complexity of hill-climbing

In some competitions, the complexity resides in doing the last mile of optimization.

Table 2 shows the additional metrics used in this work and how they relate to these principles. None can actually follow all these principles at the same time, demonstrating the importance of having multiple metrics to describe the performance of AI research agents.

### A.3 Models Used By Agents For Image Classification Tasks

Focusing on the neural architectures used by the agents in image classification tasks in Figure 14 , 3 3 3 Image classification is the largest category in MLE-bench (8 tasks out of 22 in MLE-bench lite) we observe that AIDE relies on the EfficientNet architecture (and its variants) for almost 40% of the tasks. For 75% of its attempts, AIDE uses only 3 different architectures: EfficientNet, ResNet ( He et al., 2016 ) and LightGBM. AIRA Greedy \text{AIRA}_{\text{Greedy}} uses a wider range of architectures, with EfficientNet, ConvNeXt ( Liu et al., 2022 ) , and ViT ( Dosovitskiy et al., 2020 ) making only 38% of the agents’ initial ideas.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .