Towards Execution-Grounded Automated AI Research — part 2/4

Cleaned-text code points [15323, 31065). Bibliography omitted here; source.md includes it.

Our search method is inspired by prior evolutionary search approaches for code optimization, such as AlphaEvolve ( Novikov et al., 2025 ) . Our algorithm is detailed in Algorithm 1 . At the first search epoch, we sample a full batch of new ideas. In all subsequent epochs, we split the idea generation into exploitation and exploration subsets. For exploitation, we choose ideas from previous epochs that outperform the baseline and append them to the idea generation prompt to ask the ideator model to generate new variants that combine their strengths. For exploration, we randomly sample ideas from previous epochs to append to the idea generation prompt until reaching the max context length and instruct the ideator model to generate completely new ideas different from them. We start with 50% exploitation and 50% exploration at epoch 1 and gradually anneal the exploration rate and increase the exploitation ratio in later epochs. We use a batch size of 50 for the GRPO environment and a batch size of 80 for the nanoGPT environment.

### 4.2 Experiment Results

For each environment, we perform execution-guided search with three different models: Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5. For each experiment, we use the same model as both the ideator and executor (i.e., self-execution). We plot the progression of the best performance at each search epoch in Figure 3 . We summarize several notable trends below.

First, we observe a scaling trend with Claude-4.5-Opus, where searching for more epochs leads to a higher upper bound. In contrast, Claude-4.5-Sonnet and GPT-5 tend to saturate early. Second, all models can find ideas that significantly outperform the baselines. On GRPO, Claude-4.5-Sonnet finds that using vanilla policy gradient with the group-average baseline without importance reweighting or clipping outperforms the standard GRPO objective in this particular experiment setup and exploits this finding in all subsequent search epochs, resulting in the best solution of 69.4% at epoch 2 with precise hyper-parameter tuning. On nanoGPT, Claude-4.5-Opus achieves the min validation loss of 3.1407 at epoch 9 by combining various architectural modifications, hyper-parameter tuning, and applying exponential moving average of intermediate checkpoints during validation (see Appendix A.2 for the full idea). We run this top solution on 8 H100s to follow the same setup as the nanoGPT speedrun, and it reaches the 3.28 target validation loss in 19.7 minutes, a significant speedup as compared to the baseline codebase, which takes 35.9 minutes of training time to reach the same target validation loss.

To better contextualize these solutions optimized by the model, we also compare the top performance of our execution-guided search to human experts (Table 1 ). For the GRPO environment, we compare with the leaderboard of the Stanford CS336 graduate-level LLM class, which hosted the same environment as an assignment for all students to optimize the validation accuracy under the same training time budget. The best student solution 2 2 2 https://github.com/stanford-cs336/assignment5-alignment-leaderboard achieved an accuracy of 68.8%, lower than Claude-4.5-Sonnet’s top solution using our execution-guided search. For the nanoGPT environment, we directly compare with the nanoGPT speedrun leaderboard, 3 3 3 https://github.com/KellerJordan/modded-nanogpt where the top human solution as of December 2025 can reach the target validation loss under 2.1 minutes, indicating significant headroom for further model capability and search method improvement on this environment.

### 4.3 Comparison with Best-of-N

To demonstrate the effectiveness of our search scaffold, we compare our execution-guided search with the best-of-N baseline with the same sampling budget on the nanoGPT environment. Since the batch size for our search is 80, we compare the first 3 epochs of the execution-guided search using the GPT-5 backbone with the best-of-N results of GPT-5 with N ∈ { 80,160,240 } N\in\{80,160,240\} . As shown in Figure 4 , search and best-of-N start from similar performance at epoch 0 (they are not exactly the same due to variances from sampling), but evolutionary search significantly outperforms best-of-N from epoch 1 onward, demonstrating that the model is effectively leveraging trajectories from previous epochs to generate more effective ideas in future epochs.

### 4.4 Analysis of Generated Ideas

Hyper-parameter vs Algorithmic To quantitatively understand the types of ideas that models generate during the execution-guided search, we perform a stratified analysis by classifying all generated ideas into either hyper-parameter tuning (including any ideas that can be implemented via changing existing configs) or algorithmic (including all ideas that involve implementing new changes not originally supported by the baseline codebase) by using an LLM-judge. Based on Table 2 , all three models generate a substantial amount of algorithmic ideas apart from hyper-parameter tuning. Interestingly, different models exhibit different patterns, where Claude-4.5-Sonnet generates significantly more hyper-parameter ideas than both Claude-4.5-Opus and GPT-5. Moreover, the most effective ideas come from algorithmic ideas in most cases, except when using Claude-4.5-Sonnet.

Qualitative Examples To complement the quantitative analysis, we provide several executed ideas on the GRPO environment in Table 3 and provide example ideas on the nanoGPT environment in Appendix A.2 . When sampling ideas, models would generate a thinking trace, followed by the natural language idea and a brief description of all the code changes needed to implement the idea. For brevity, we only include the natural language ideas in the table, but we present additional examples in Appendix A.3 with more details, including full code execution trajectories. Based on the qualitative examples in Table 3 , different models generate different styles of ideas. For example, Claude-4.5-Sonnet generates more intuitive ideas while Claude-4.5-Opus and GPT-5 are more mathematically inclined.

Recovering Recent Research Papers We observed multiple cases where the model-generated ideas (without any RAG) are highly similar to research papers released within the three months prior to the writing of this paper. For example, Claude-4.5-Sonnet proposed: “ Implement response diversity rewards within groups where responses to the same prompt receive bonus rewards for being dissimilar to other responses in their group, encouraging exploration of different solution paths. ”, which is similar to Li et al. (2025) . For pre-training, Claude-4.5-Opus proposed: “ Causal Context Compression: Before each attention layer, apply a learned compression that mixes local context (previous 2-3 tokens) into the current representation, providing implicit local context without convolutions. ”, which is similar to the “canon layer” described in Allen-Zhu (2025) . Although assessing the novelty of LLM-generated ideas is beyond the scope of this work, models’ ability to rediscover ideas from recent research papers nevertheless indicates that automated AI researchers could plausibly support work at the frontier of LLM research.

## 5 Reinforcement Learning from Execution Reward

Different from evolutionary search, reinforcement learning is an alternative learning algorithm that shapes model behaviors through gradient updates. Despite much recent success on verifiable domains like math and coding ( DeepSeek-AI et al., 2025 ) , RL’s effectiveness on open-ended AI research remains unclear. For the first time, we explore whether we can leverage the automated executor as a reward function to directly finetune LLMs to generate more effective ideas via RL. We detail our implementation, experiment setup, and analysis of the training dynamics.

### 5.1 Reward Design and Experiment Setup

We use Qwen3-30B-A3B ( Yang et al., 2025a ) as the base model and finetune it using the standard GRPO algorithm ( Shao et al., 2024 ) , motivated by its consistent empirical success on other verified domains. Our prompt batch size is one since we only have one prompt for each environment. In the prompt, we provide the baseline codebase and ask the model to generate new ideas to improve the baseline (GRPO or nanoGPT). This experiment setup is similar to prior work exploring RLVR from one training example ( Wang et al., 2025 ) .

We use large group sizes to stabilize training. For the post-training environment, we use a group size of 256; for the pre-training environment, we use a group size of 128. Since each idea on the GRPO environment runs on one single GPU and each idea on the nanoGPT environment runs on 8 GPUs, these group sizes correspond to parallel execution on 256 GPUs (for GRPO) or 1024 GPUs (for nanoGPT) to obtain the execution reward on each batch of rollout ideas. Each rollout is a thinking trace followed by the natural language idea. We set a max output length of 8192 tokens for rollout sampling and only feed the extracted ideas to the automated executor without the preceding thinking trace.

For the post-training environment, we directly use the validation set accuracy of each rollout idea after execution as the reward. For ideas without a valid accuracy (i.e., when the execution failed due to code generation errors), we assign a reward of 0. For the pre-training environment, we use the reciprocal of the validation loss as the reward ( 1 l ​ o ​ s ​ s \frac{1}{loss} ) and assign a reward of 0 to ideas with failed execution. Our experiments are based on the Tinker API ( Thinking Machines Lab, 2025 ) .

### 5.2 Experiment Results

#### Positive Training Curves for Average Reward

We plot the average reward of all rollouts of each training epoch in the upper row of Figure 5 . For the first time, we demonstrate that the average performance of the generated ideas can increase after sufficient training epochs for open-ended research environments. For instance, the average accuracy on the GRPO environment increases from 0.253 at the beginning to 0.343 after 40 training epochs (top left plot of Figure 5 ); and the average reward on the nanoGPT environment increases from 0.194 at the beginning to 0.246 after 68 epochs (top right plot of Figure 5 ), corresponding to a decrease in the average validation loss from 5.150 to 4.066. Such training curves are similar to prior findings on the effectiveness of one-shot RVLR on other verified domains like math ( Wang et al., 2025 ) .

#### The Case of Max Reward

Despite successfully reproducing the positive training curves observed in other domains, we argue that there is a distinction between idea generation and other verifiable domains. For advancing scientific discovery, we often care about the upper-bound of idea generation, rather than the average quality. In our particular case, we care more about having one breakthrough idea that dominates the baselines, rather than having many safe ideas with a high average. Thus, we plot the max reward of all rollouts at each training epoch in the lower row of Figure 5 . The trend here is very different – the max reward is fluctuating throughout RL training without a clear upward trend . This reveals the crucial limitation of the standard GRPO algorithm for improving idea generation. In the next subsection, we analyze why RL from execution reward improves the average reward but not the max.

### 5.3 Analysis of Training Dynamics

#### Thinking Length

We first plot how the lengths of the thinking traces evolve over RL training in the upper row of Figure 6 . In both environments, the thinking traces rapidly decrease in length while the idea lengths stay roughly constant. This is the opposite of the thinking emergence trend from prior RLVR work, such as DeepSeek-R1 ( DeepSeek-AI et al., 2025 ) . To further understand why the thinking length decreases, we investigate the correlation between the idea execution rate and the thinking trace length. In the bottom row of Figure 6 , for the first 20 epochs of the RL training, we sort all ideas in each epoch by their thinking trace lengths and plot the average execution rate of the top-30% longest thinking ideas (red line) and the bottom-30% shortest thinking ideas (blue line). We see a clear trend where ideas with longer thinking consistently have a lower execution rate. We thus hypothesize that longer thinking correlates with more complex ideas with lower execution rates, leading the model to prefer shorter thinking instead in order to maximize the reward.

#### Diversity Collapse

Upon manual investigation of all the rollouts being sampled throughout the RL training, we also observed a diversity collapse. Specifically, the model learned to converge on a few simple ideas that can consistently get a positive reward. For example, in the nanoGPT environment, the model learned to converge towards two common ideas: (1) replacing RMSNorm with LayerNorm; and (2) performing exponential moving average (EMA) over intermediate model checkpoints. As shown in Figure 7 , out of a batch of 128 sampled ideas per epoch, 51 ideas sampled from Qwen3-30B at epoch 0 are one of the two common ideas above. Towards the end of the RL training, 119 out of 128 sampled ideas at epoch 68 are one of the two common ideas, indicating a severe diversity collapse.

The above analyses reveal that RL causes the models to converge on a few simple-to-implement ideas, accompanied by shrinking thinking lengths. These lead to an increase in the average reward, but do not push the upper-bound due to the lack of exploration. This phenomenon is analogous to mode-collapse observations on other verifiable domains, where the pass@k performance stagnates or even decreases after RL ( Yue et al., 2025 ; Wu et al., 2025 ) . Avoiding such convergence and collapse is an open problem and likely requires new algorithmic interventions beyond standard GRPO, which is beyond the scope of this work. However, we do share several preliminary attempts, including: sampling and appending previous epochs’ trajectories into the current epoch’s prompt for rollout sampling, adding a weighted length reward in the total reward, and adding a weighted similarity penalty in the total reward. We did not observe clear gains in the initial epochs and thus early-stopped them, but we document all these results in Appendix A.1 to inform future work.

## 6 Related Work

#### AutoML

Our work has deep connections to the AutoML literature. For example, the Neural Architecture Search (NAS) line of work typically defines a constrained set of neural network operators and optimizes for architectures based on validation set performance through reinforcement learning ( Zoph & Le, 2017 ; Zoph et al., 2017 ) or search ( Liu et al., 2018 ; So et al., 2019 ) . In the modern era, recent works also explored directly using LLMs to propose architecture variants and implement them for validation ( Liu et al., 2025 ; Cheng et al., 2025 ) . Beyond architectures, similar automatic optimizations have been applied to improve hyperparameter tuning ( Zhang et al., 2023 ) , discover machine learning algorithms ( Real et al., 2020 ) , improve post-training objectives ( Lu et al., 2024a ) , discover better neural network optimizers ( Chen et al., 2023 ) , and design agent scaffolds ( Hu et al., 2025 ) . Different from this line of work, we tackle automated AI research in a fully open-ended setting without any constraint on the type of ideas. Moreover, our goal is to improve the effectiveness of idea generation, where natural language ideas represent a higher level of abstraction than specific architecture variants or code optimizations.

#### LLM-based Research Agents