Learning to Discover at Test Time — part 1/5

Cleaned-text code points [0, 15416). Bibliography omitted here; source.md includes it.

# Learning to Discover at Test Time

###### Abstract

How can we use AI to discover a new state of the art for a scientific problem? Prior work in test-time scaling, such as AlphaEvolve, performs search by prompting a frozen LLM. We perform reinforcement learning at test time, so the LLM can continue to train, but now with experience specific to the test problem. This form of continual learning is quite special, because its goal is to produce one great solution rather than many good ones on average, and to solve this very problem rather than generalize to other problems. Therefore, our learning objective and search subroutine are designed to prioritize the most promising solutions. We call this method Test-Time Training to Discover (TTT-Discover). Following prior work, we focus on problems with continuous rewards.

We report results for every problem we attempted, across mathematics, GPU kernel engineering, algorithm design, and biology. TTT-Discover sets the new state of the art in almost all of them: (i) Erdős’ minimum overlap problem and an autocorrelation inequality; (ii) a GPUMode kernel competition (up to 2 × 2\times faster than prior art); (iii) past AtCoder algorithm competitions; and (iv) denoising problem in single-cell analysis. Our solutions are reviewed by experts or the organizers.

All our results are achieved with an open model, OpenAI gpt-oss-120b, and can be reproduced with our publicly available code , in contrast to previous best results that required closed frontier models. Our test-time training runs are performed using Tinker, an API by Thinking Machines, with a cost of only a few hundred dollars per problem.

## 1 Introduction

To solve hard problems, humans often need to try, fail, stumble upon partial successes, and then learn from their experiences. Consider your first really hard programming assignment. You read the textbook and trained yourself on the book exercises, but this assignment just asked for so much beyond the basics in the book. You tried to guess the solution, but these attempts merely produced small signs of life. So you had to take a deep breath and learn from your failed attempts, which made your future attempts more intelligent. Finally, after hours of trying and learning, you understood the new ideas behind the assignment. And indeed, the next attempt worked!

In this example, the assignment was hard because it required new ideas beyond your training data (the text and exercises in the book). Now consider using AI to solve scientific discovery problems. This goal is even harder: By definition, discovery problems require ideas not only beyond the model’s training data but also all existing knowledge of humanity. And out-of-distribution generalization is no easier for AI than for humans Miller et al. (2020) ; Hendrycks et al. (2021) ; Ribeiro et al. (2020) ; Koh et al. (2021) .

To offset this hardness, prior work has focused on test-time search in the solution space by prompting a frozen LLM to make many attempts, similar to how we tried to guess the solution to the assignment. In particular, evolutionary search methods, such as AlphaEvolve, store past attempts in a buffer and use them to generate new prompts via hand-crafted and domain-specific heuristics Novikov et al. (2025) ; Lange et al. (2025) ; Sakana AI (2026) ; Yuksekgonul et al. (2025) . While these prompts can help the LLM improve previous solutions, the LLM itself cannot improve, similar to a student who can never internalize the new ideas behind the assignment.

The most direct way for the LLM to improve is through learning. And indeed, while both learning and search scale well with compute Sutton (2019) , learning has often superseded search in the history of AI for hard problems such as Go and protein folding Silver et al. (2017) ; Jumper et al. (2021) . We believe that this observation from history is still relevant today, as we scale compute at test time. So we continue to train the LLM, while it attempts to solve this very test problem. And these attempts, in turn, provide the most valuable training data: Recall that the test problem was hard because it was out-of-distribution. Now we have a data distribution specific to this problem.

At a high level, we simply perform Reinforcement Learning (RL) in an environment defined by the single test problem, so any technique in standard RL could be applied. However, our goal has two critical differences from that of standard RL. First, our policy only needs to solve this single problem rather than generalize to other problems. Second, we only need a single best solution, and the policy is merely a means towards this end. In contrast, the policy is the end in standard RL, whose goal is to maximize the average reward across all attempts. While the first difference is a recurring theme in the field of test-time training Sun et al. (2020) , the second is unique to discovery problems.

To take advantage of these differences, our learning objective and search subroutine strongly favor the most promising solutions. We call this method Test-Time Training to Discover (TTT-Discover). We focus on problems with continuous rewards, in mathematics (§ 4.1 ), GPU kernel engineering (§ 4.2 ), algorithm design (§ 4.3 ), and biology (§ 4.4 ). We report results for every problem we attempted, and TTT-Discover sets the new state of the art in almost all of them, using only an open model.

There are three pieces of concurrent work that share our high-level idea: EvoTune (Surina et al.) Surina et al. (2025) , MiGrATe (Phan et al.) Phan et al. (2025) , and recently ThetaEvolve (Wang et al.) Wang et al. (2025a) , which is especially relevant. Compared to ThetaEvolve, TTT-Discover using the same model and compute budget still produces significant improvements (Table 2 ), due to its special learning objective and search subroutine.

## 2 Preliminaries

All methods in this paper, including the baselines, share a common goal: Given a scientific problem at test time, the goal is to discover a new state-of-the-art solution with an LLM policy π θ \pi_{\theta} , whose weights θ \theta have already been trained (at training time). To formalize this goal, we first introduce how each scientific problem defines an environment, i.e., a Markov Decision Process (§ 2.1 ), which can then be used for search (§ 2.2 ) and learning (§ 3 ).

### 2.1 Discovery Problem

Our definition of the environment follows prior work in test-time scaling, such as AlphaEvolve Novikov et al. (2025) : A scientific problem comes in the form of a text description d d , which we always feed as context to the policy. We define a state s s as a candidate solution, such as a kernel implementation of the PyTorch code in d d . In our applications, the problem description also induces a continuous reward function R ⁡ ( s ) ∈ R R(s)\in\mathbb{R} , such as the inverse runtime of the kernel.

We denote s sota s_{\text{sota}} as the best-known solution among all existing candidates, and r sota = R ⁡ ( s sota ) r_{\text{sota}}=R(s_{\text{sota}}) as the best-known reward. And in case there is no existing solution, s sota s_{\text{sota}} can be the empty string <empty> . For example, s sota s_{\text{sota}} can be the kernel currently at the top of the leaderboard. These notations allow us to formalize the notion of a discovery:

###### Definition (Discovery) .

A discovery is an event where a state s s is found such that R ⁡ ( s ) > r sota R(s)>r_{\text{sota}} . The larger the difference, the more significant the discovery.

Under this formalism, we define a discovery problem as finding such a state s s with large R ⁡ ( s ) − r sota R(s)-r_{\text{sota}} within the environment defined by the scientific problem.

To produce a better solution, both search and learning methods use the LLM policy to generate an action a ∼ π θ ( ⋅ ∣ d , s ) a\sim\pi_{\theta}(\cdot\mid d,s) , where the choice of the initial solution s s (e.g., = s sota =s_{\text{sota}} ) is an important part of the method’s design. Similar to the reward function, the transition function ( s , a ) → s ′ (s,a)\rightarrow s^{\prime} of the environment is also induced by the problem description. Here, we consider only a single timestep since state reuse, which we will introduce soon, effectively subsumes multiple timesteps.

In all our applications, a valid action contains a piece of code and optionally some thinking tokens. For coding problems (e.g., kernel engineering), the environment produces s ′ s^{\prime} by simply parsing the code out of a a . For problems in mathematics, the environment also needs to execute the code in a a after it is parsed. Table 1 provides an overview of the environments for all our applications.

### 2.2 Search Methods

The simplest search method, known as Best-of- N N , samples i.i.d. rollouts from π θ \pi_{\theta} : Best-of- N : s = s sota or <empty> , a i ∼ π θ ( ⋅ ∣ d , s ) , \textbf{Best-of-$N$:}\quad s=s_{\text{sota}}\text{ or }\texttt{<empty>}{},\;\;a_{i}\sim\pi_{\theta}(\cdot\mid d,s), where the subscript, i = 1 , … , N i=1,\dots,N , denotes the index of the rollout. By using i i instead of t t for the index, we indicate that the rollouts here are independent. One reasonable choice of the initial state s s is s sota s_{\text{sota}} , assuming that a previous solution exists. But s sota s_{\text{sota}} might be too strong a prior towards exploitation. For example, conditioning on s sota s_{\text{sota}} might prevent the policy from exploring very different, but more promising directions that would ultimately produce better solutions under a large compute budget. To address this concern, we usually set s = <empty> s=\texttt{<empty>}{} , the empty (or trivial) solution.

On the other hand, the policy might also explore a promising direction using s = <empty> s=\texttt{<empty>}{} , but fail to fully exploit it. One technique to address this opposite concern is state reuse , which warm starts the policy with some of the previous solutions. Specifically, it maintains a buffer ℋ i \mathcal{H}_{i} of the previous solutions, and samples the initial solution s i s_{i} from ℋ i \mathcal{H}_{i} using a search heuristic, reuse , which favors high-reward solutions but still assigns nontrivial likelihood to low-reward ones: State reuse: s i ∼ reuse ( ℋ i ) , a i ∼ π θ ( ⋅ ∣ d , s i ) , ℋ i + 1 = ℋ i ∪ { ( s i ′ , r i ) } . \textbf{State reuse:}\quad s_{i}\sim\texttt{reuse}(\mathcal{H}_{i}),\;\;a_{i}\sim\pi_{\theta}(\cdot\mid d,s_{i}),\;\;\mathcal{H}_{i+1}=\mathcal{H}_{i}\cup\{(s^{\prime}_{i},r_{i})\}.\vskip-4.30554pt When we reuse a previous solution s i ′ s^{\prime}_{i} , we have effectively added an extra timestep to its trajectory.

Prior work, such as AlphaEvolve Novikov et al. (2025) , also reuses the actions, which can contain thinking tokens and intermediate results (e.g., code for math problems) that are not part of the states. As a consequence, the reuse heuristic also needs to convert the information from previous actions into natural language context c i c_{i} that can be ingested by the LLM policy: State-action reuse: s i , c i ∼ reuse ( ℋ i ) , a i ∼ π θ ( ⋅ ∣ d , s i , c i ) , ℋ i + 1 = ℋ i ∪ { ( s i , a i , s i ′ , r i ) } . \textbf{State-action reuse:}\quad s_{i},c_{i}\sim\texttt{reuse}(\mathcal{H}_{i}),\;\;a_{i}\sim\pi_{\theta}(\cdot\mid d,s_{i},c_{i}),\;\;\mathcal{H}_{i+1}=\mathcal{H}_{i}\cup\{(s_{i},a_{i},s^{\prime}_{i},r_{i})\}. Prior work Novikov et al. (2025) ; Lange et al. (2025) ; Yuksekgonul et al. (2025) ; Liu et al. (2024b) refers to state-action reuse as evolutionary search , because the reuse heuristic usually involves sophisticated designs motivated by evolution, including hand-crafted operations for mutation and cross-over, and domain-specific measurements of fitness and diversity.

## 3 Learning to Discover at Test Time

So far, the policy’s experience with the test problem can only improve the next prompt ( d , s i , c i ) (d,s_{i},c_{i}) , but not the policy π θ \pi_{\theta} itself, since θ \theta remains frozen. We use this experience to improve the policy in an online fashion, by training π θ \pi_{\theta} on its own search attempts accumulated in the buffer ℋ i \mathcal{H}_{i} .

Algorithm 1 outlines the general form of our method, where the two key subroutines to instantiate are reuse and train .

### 3.1 Naive RL at Test Time

Algorithm 1 falls under the formulation of reinforcement learning (RL). A natural baseline is to use a standard RL algorithm: train : θ i + 1 = θ i + η ∇ θ E a ∼ π θ i ( ⋅ ∣ s ) [ R ( s , a ) ] , reuse ( ℋ i ) = δ <empty> , \texttt{train}:\quad\theta_{i+1}=\theta_{i}+\eta\nabla_{\theta}\mathbb{E}_{a\sim\pi_{\theta_{i}}(\cdot\mid s)}\!\left[R(s,a)\right],\qquad\texttt{reuse}(\mathcal{H}_{i})=\delta_{\texttt{<empty>}{}}, i.e., optimize for expected reward with no reuse, where δ <empty> \delta_{\texttt{<empty>}{}} is a delta distribution with mass only on the initial state <empty> . We will use θ i \theta_{i} to denote the model weights for rollout i i . We can straightforwardly apply popular RL algorithms, such as PPO or GRPO Schulman et al. (2017) ; Guo et al. (2025) , only in the environment defined by the single problem.

However, these algorithms are designed with the standard RL problem in mind. Discovery problems have important distinctions from standard RL problems.

In standard RL problems, the goal is to find a policy that maximizes the expected reward. This policy is to be deployed repeatedly in the same environment. The primary artifact is the policy.

In discovery problems, the goal is to find a single state that improves upon the state-of-the-art. We do not care about the average performance. There is no separate deployment phase and thus the policy need not maintain robust performance in many states it may encounter starting from the same initial state distribution. In fact, a policy can have very low expected reward, so long as it reaches a new state-of-the-art once.

Due to these differences, the naive RL instantiation has important shortcomings.

Objective function. Naive RL optimizes average performance, and is indifferent to the state of the art. In discovery, however, success is determined by the maximum, and whether it improves upon the state of the art. Consider a kernel engineering problem where the state-of-the-art runtime is 2000 ​ μ ​ s 2000\,\mu\text{s} . Achieving 1900 ​ μ ​ s 1900\,\mu\text{s} would require substantial optimization and perhaps a breakthrough. Yet, without complicated reward shaping, both would receive nearly the same reward.

Short effective horizon. Starting each attempt from scratch limits how far the policy can reach in an attempt. Reusing a previous solution effectively adds extra timesteps to an attempt, extending the horizon. As a result, more complex solutions can emerge during training. In standard RL, a fixed initial state distribution makes sense as the policy must perform robustly from states it will encounter at deployment. Discovery has no such deployment phase.

Exploration. Exploration requires care at two levels. Optimizing for expected reward, the policy can collapse to safe, high-reward actions rather than risky ones that might achieve discovery. At the reuse level, naive prioritization can over-exploit a few promising states at the expense of diversity.

### 3.2 TTT-Discover

To address these shortcomings, we introduce two simple components.