What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity — part 2/3

Cleaned-text code points [15806, 31783). Bibliography omitted here; source.md includes it.

By changing the parts of the prompt mentioning diversity, we intend to only impact the diversity of ideas generated by the agent, and not other solution aspects, such as implementation quality.

## 4 The Ideation Diversity Bottleneck

This section presents the core results of this paper. First, in Section 4.1 , we analyze a large sample of agentic trajectories in MLE-bench. In Section 4.2 , we show the controlled experiment to validate the diversity hypothesis. Finally, in Section 4.3 , we incorporate additional metrics in our analysis.

### 4.1 Deep-dive on Agentic Trajectories

##### The agent scaffold choice impacts ideation diversity

Figure 3 illustrates a comparison between AIDE and AIRA Greedy \text{AIRA}_{\text{Greedy}} agents, both using o3 as the backbone. Observing the model architectures and general machine learning approaches used by the agent, we can see from Figure 3 (a) that AIDE agents prefer Gradient Boosting Decision Trees (GBDT) and Convolutional Neural Networks (CNN) in 70% of the initial draft nodes. In contrast, AIRA Greedy \text{AIRA}_{\text{Greedy}} agents generate a greater diversity of ideas. The most common architectures among these agents are CNN, Transformers, GBDT, and Hybrid models that combine multiple approaches. Collectively, these four architectures represent 68% of the ideas produced, as shown in Figure 3 (b).

Looking at the models trained by the agent in Figure 3 (c) and (d), LightGBM and EfficientNet represent 43% of models AIDE agents intend to train in its initial draft nodes, while in the case of AIRA Greedy \text{AIRA}_{\text{Greedy}} as many as 9 models represent this percentage. This difference in diversity highlights the importance of the design of agents (system prompt, search mechanism, operators), in influencing the variety of models the agent intends to train or use. Next, we study how this ideation diversity correlates to performance on MLE-bench.

##### Diversity correlates with performance on MLE-bench

Figure 1 shows the correlation of the MLE-bench score (measured as medal rate) with ideation diversity (measured as architecture choice distribution entropy), for each run of the AI research agents included in our study on the 75 tasks of MLE-bench. Here, a point in the plot refers to one agent’s performance on the full set of tasks.

We observe a good correlation between diversity and performance, with two distinct clusters, one including high-performing agents (using o3, gpt-oss 120b, and gpt-oss 20b as backbones) and the other using open-source LLMs in our study (Llama Maverick, Devstral, CWM). Agents that utilize a wider range of techniques tend to achieve higher performance. Additionally, we observe in Figure 4 how diversity changes for different agents, by measuring how many model architectures on average are used in the first 5 nodes of the agent’s exploration, a metric we refer to as tree-level diversity.

Figure 4 shows how high-performing models considered here (o3, gpt-oss 120b, gpt-oss 20b) use more diverse architectures in the 5 initial ideas (3.5 distinct architectures on average) compared to Llama Maverick, Devstral and CWM (2.8 distinct architectures on average). Like diversity measured as entropy, tree-level diversity also correlates with performance.

### 4.2 Impact of Diversity: A Controlled Experiment

We have seen that better agents are usually having more diversity of ideas in their trajectories. To understand whether diversity has a causal relationship with performance, we perform a controlled experiment, as described in Section 3.3 , by removing the different mechanisms for diversity, and directly prompting the agents to generate similar ideas to solve a single task.

#### 4.2.1 Are We Actually Influencing Diversity?

Prompting the agent to come up with similar ideas negatively impacts the diversity of ideas generated by the agent. Figure 5 shows that agents with less diversity use a decreased number of unique architectures and general ML approaches. Baseline agents AIRA Greedy \text{AIRA}_{\text{Greedy}} and AIRA MCTS \text{AIRA}_{\text{MCTS}} use no more than 2 different architectures in their 5 initial drafts in only 40% of tasks. However, the AIRA Greedy ​ - Low Diversity \text{AIRA}_{\text{Greedy}}\text{- Low Diversity} and AIRA MCTS ​ - Low Diversity \text{AIRA}_{\text{MCTS}}\text{- Low Diversity} agents that are prompted to come up with similar ideas, use no more than 2 distinct architectures or approaches in 70% of tasks. These different behaviors highlight the actual impact of diversity mechanisms on ideation.

#### 4.2.2 Results

When it comes to performance measured as medal rate, Figure 6 demonstrates that reducing ideation diversity - by prompting the agent differently in order to generate similar ideas - leads to a decline of performance on MLE-bench lite. This applies for both agentic scaffolds AIRA Greedy \text{AIRA}_{\text{Greedy}} and AIRA MCTS \text{AIRA}_{\text{MCTS}} , with a 6.9 6.9 and 8.4 8.4 absolute points decrease, respectively. By modifying the system prompt to isolate the effect of ideation diversity, the results indicate that diversity is an important factor limiting performance.

### 4.3 Evaluating with Alternative Metrics

While relevant, the differences in performance observed so far are constrained to the medal rate metric, used by default. The medal rate metric, by itself, may not offer a comprehensive picture of agent performance on MLE-bench. In the current section, we introduce alternative metrics to provide a more complete assessment of performance on MLE-bench.

#### 4.3.1 Alternative Metrics

Each of the alternative metrics we consider offers a distinct perspective: while some, like medal rate, emphasize marginal improvements, others account for all performance gains. Additionally, certain metrics are based entirely on human score distributions, whereas others operate independently of them. We consider 4 additional metrics. 1. Valid Submission Rate : The percentage of tasks in which the agent is able to make a valid submission. This metric captures the ability of the agent to ideate, implement, and debug until reaching at least one valid submission. 2. Average Normalized Score : For each agent attempt at a task, we compute a normalized score: a score of 0 represents the lowest human score achieved on the task, and 1 the highest. The metric captures how good agents submissions are, independently from human score distributions. 3. Percentile : The metric captures the ability of the agent to outperform humans at machine learning engineering. Compared to medal rate, this metric still relies on human score distributions, and offer a less discrete assessment of the performance. Like average normalized scores, and unlike medal rates, improvements of percentile in poor and strong scores are equally valued. 4. ELO-Based Agent Ranking : We create an ELO system ( Bradley and Terry, 1952 ) using all possible heads-to-heads between agents’ scores. ELO rankings are agnostic of the human score distribution on MLE-bench tasks. ELO difference of 100 points corresponds to about a 64% expected win probability for the higher-rated agent.

#### 4.3.2 Data Analysis Results with Additional Metrics

We use the new set of metrics to gain a deeper understanding of the correlation between diversity and performance. When measuring performance using either the percentile or the average normalized score, instead of the medal rate, our correlation results remain consistent and, in fact, show even higher correlations (Figures 8 and 8 ).

#### 4.3.3 Controlled Experiment Results with Additional Metrics

Using this additional set of metrics, we can perform a more comprehensive analysis of the controlled experiment. Figure 9 shows the performance gap between baseline agents ( AIRA Greedy \text{AIRA}_{\text{Greedy}} , and AIRA MCTS \text{AIRA}_{\text{MCTS}} ) and agents with ablated diversity. The performance loss is observed across all different metrics, validating our hypothesis that agents perform worse when ideas are less diverse.

A notable observation is the drop in valid submission rates, which fall from 98% to 92% for AIRA Greedy \text{AIRA}_{\text{Greedy}} and to 90% for AIRA MCTS \text{AIRA}_{\text{MCTS}} . This indicates that, for certain tasks, Low Diversity agents are sometimes unable to produce even a single valid submission during their search. Our analysis reveals that this decline is primarily driven by two competitions: ’text-normalization-challenge-english-language’ and ’text-normalization-challenge-russian-language’. Upon examining agent trajectories, we find that Low Diversity agents repeatedly attempt to implement the same model, T5 ( Raffel et al., 2020 ) , but consistently fail, resulting in timeouts. In contrast, baseline agents implement a wider range of solutions and are more often able to make correct submissions. Notably, baseline agents also occasionally attempt to implement T5 and encounter similar failures, but their greater ideation diversity allows them to succeed elsewhere. These two competitions also account for a significant portion (estimated at around half) of the observed decline in medal rates.

Our findings suggest that one reason for the drop in performance is that Low Diversity agents tend to focus on similar ideas that they sometimes cannot successfully implement. In other words, ideation diversity is crucial for performance, as it increases the likelihood that an AI research agent will attempt solutions it is actually capable of executing.

## 5 Discussion

##### What does it take to be a good AI research agent?

We can imagine a hypothetical future scenario where excellent AI research agents ideate brilliant experiments and have outstanding coding skills to implement them. Until we get to this ideal situation, in practice, even state-of-the-art AI research agents will exhibit limited ideation and implementation capabilities, particularly when evaluated in challenging, real-world settings. In this imperfect, yet realistic, scenario, given the same level of capabilities, we prefer agents with greater ideation diversity. First, because it de-risks implementation pitfalls. Our analysis of the controlled experiment shows that one reason why diversity is important is that it helps agents design solutions they are actually able to execute, highlighting the interplay between ideation and implementation. If the different proposed plans by the agent rely on similar approaches, and those happen to be hard to implement by the agent (in the context of the particular task), then we risk low implementation accuracies. Intuitively, a potential second argument for ideation diversity is that given the difficulty of coming up with creative, yet feasible research ideas, exploring significantly different paths hedges against pursuing a single unproductive direction (even if the agent knows how to implement it), and enables agents to more effectively explore the solution space of machine learning problems. We want to invest the allocated compute in a diversified, yet plausible, set of ideas. However, this second reason is hard to evaluate given the implementation bottleneck. Ultimately, a good experimentation plan could fail due to the agent being unable to implement it. Repeating these controlled experiments as LLMs’ coding capabilities get increasingly more powerful may yield valuable insights.

##### Importance of the implementation bottleneck.

Unsurprisingly, implementation quality is an important bottleneck of AI research agents. We observe a strong correlation between AI research agents’ performance and the ability to implement sufficiently complex solutions. By aggregating performances of AIRA (Greedy and MCTS) for each LLM, Figure 11 shows that, on average, the more time an agent spends on each successfully implemented solution (including ideation, implementation, and model training), the more medals it earns. This suggests that performance increases with the agents’ ability to implement more complex solutions. Furthermore, Figure 11 shows that agents perform better when, out of the 24 hours allotted to complete a task, they spend a higher proportion of time on successfully implemented solutions. However, since LLMs and coding agents are improving rapidly ( Kwa et al., 2025 ) , particularly in verifiable tasks ( DeepSeek-AI et al., 2025 ) , we hypothesize that the relative importance of the ideation and planning phase might increase over time, not to de-risk implementation pitfalls, but to efficiently explore the solution space.

##### Generalization to other benchmarks.

The findings presented in this study are based on experiments conducted using MLE-bench only. Given the range of machine learning tasks included in this benchmark, we hypothesize that our results are likely to generalize to other machine learning tasks. Additional benchmarks could be examined in future research.

##### Limitations of MLE-bench evaluations.

Performance on MLE-bench has traditionally been evaluated using Kaggle’s medal system, where medals are awarded based on score percentiles. For example, in competitions with fewer than 99 teams, gold medals are given to the top 10% of submissions. However, this medal-based evaluation framework has several important limitations.

First, medal criteria change with the number of submissions (e.g., bronze goes to the top 10% for 1000+ teams, but top 40% for 1–249 teams), so earning a medal does not indicate consistent performance across competitions. Second, as shown in the Appendix, the gap between the bronze threshold and the best score is often extremely small—frequently below 3%.

Since agents are evaluated on custom test sets, while medal thresholds are computed using Kaggle private test sets, the data split variance introduces some score variability, and can affect medal outcomes. Third, some competitions are over a decade old, and human score distributions from these may not represent current machine learning standards. In some older competitions like ’detecting-insults-in-social-commentary’, AIRA agents are able to outperform best human submissions. In recent competitions (post-2022), agent performance drops sharply, with most agents unable to earn medals. To address these different issues, we also used additional metrics, in Section 4.3 . In the Appendix, we provide additional information on these alternative metrics.

##### Limitations of this study.

Despite the efforts to isolate ideation diversity, it is difficult to track the potential second-order effects of modifying the system prompt. To better isolate the effect of ideation diversity, future work could focus on disentangling the LLM responsible for ideating, and the one responsible for implementing. In this work, we also experimented with controlling diversity through temperature, as detailed in the Appendix.

## 6 Related Work

##### Generation diversity in language models.

Generation diversity in language models has been studied and even explicitly promoted since the statistical machine translation era ( Macherey and Och, 2007 ; Gimpel et al., 2013 ; Xiao et al., 2013 ) , where selecting ( Devlin and Matsoukas, 2012 ) or combining ( Macherey and Och, 2007 ) a set of diverse yet plausible generations lead to improved translation quality. Similar observations were later identified in neural machine translation and other sequence-to-sequence settings ( Li et al., 2015 ; Vijayakumar et al., 2016 ; Ippolito et al., 2019 ) . Holtzman et al. (2020) developed nucleus sampling with the goal of generating both coherent and diverse text. More recently, Murthy et al. (2025) and Kirk et al. (2024) study the effect of RLHF ( Ouyang et al., 2022 ) on LLMs with a focus on the (decreased) generation diversity. Chen et al. (2024) study effect of diversity of synthetic data in training LLMs. Li et al. (2025c) propose a diversity-preserving algorithm for supervised fine-tuning of LLMs.