Learning to Discover at Test Time — part 4/5
Cleaned-text code points [47022, 62247). Bibliography omitted here; source.md includes it.
### 5.1 Continual Learning
Most of today’s AI systems remain static after deployment, even though the world keeps changing. The high-level goal of continual learning is to enable AI systems to keep changing with the world, similar to how humans improve throughout their lives Hassabis et al. (2017) ; De Lange et al. (2021) .
Conventionally, continual learning as a research field has focused on learning from a distribution that gradually changes over time Lopez-Paz and Ranzato (2017) ; Van de Ven and Tolias (2019) ; Hadsell et al. (2020) . For example, one could update a chatbot model every hour using new knowledge from the Internet, while typical use cases of the model may require knowledge from both the past and the present Scialom et al. (2022) ; Ke et al. (2023) ; Wang et al. (2024) . More formally, at each timestep, we sample new training and test data from the current distribution, update our model using the new training data, and then evaluate it on all the test data up to the current timestep. Under this setting, most algorithms focus on not forgetting the past when learning from the present Santoro et al. (2016) ; Li and Hoiem (2017) ; Kirkpatrick et al. (2017) ; Gidaris and Komodakis (2018) .
### 5.2 Test-Time Training
The algorithmic framework of test-time training has the same high-level goal as continual learning, but it focuses on two aspects where human learning stands out from the forms of continual learning in the conventional literature.
First, each person has a unique brain that learns within the context of their individual life. This personalized form of continual learning is quite different from, for example, the chatbot model that is fine-tuned hourly using the latest information available worldwide. While such a model does change over time, it is still the same at any given moment for every user and every problem instance.
Second, most human learning happens without a boundary between training and testing. Consider your commute to work this morning. It is both "testing" because you did care about getting to work this very morning, and "training" because you were also gaining experience for future commutes. But in machine learning, the train-test split has always been a fundamental concept.
The concept of test-time training is introduced to realize these two special aspects of human learning. Training typically involves formulating a learning problem (such as empirical risk minimization) and then solving it. Following Sun et al. (2023) , test-time training is defined as any kind of training that formulates a potentially different learning problem based on each individual test instance.
This concept has a rich history in AI. A well-known example in NLP is dynamic evaluation, pioneered by Mikolov et al. Mikolov et al. (2013) and extended by Krause et al. Krause et al. (2018) . In computer vision, early examples have also emerged in applications such as face detection Jain and Learned-Miller (2011) , video segmentation Mullapudi et al. (2018) , super-resolution Shocher et al. (2018) , and 3D reconstruction Luo et al. (2020) . Next, we discuss three popular forms of test-time training today, with an emphasis on their connections to each other and to historical examples.
#### 5.2.1 TTT on Nearest Neighbors: Larger Effective Capacity
One simple form of test-time training was called locally weighted regression in the 1970s Stone (1977) ; Cleveland (1979) , local learning in the 1990s Bottou and Vapnik (1992) , and KNN-SVM in the 2000s Zhang et al. (2006) : Given a test instance, find its nearest neighbors in the training set, and then train (or fine-tune) the model on these neighbors before making a prediction. This procedure can significantly increase the effective capacity of the model; for example, it allows a linear model to fit a highly nonlinear ground truth Stone (1977) .
This simple form captures one of the key intuitions of test-time training. In the conventional view of machine learning, a model, once trained, no longer changes at test time. As a consequence, it must prepare to be good at all possible inputs in the future. This task can be very hard, because being good at all possible futures limits the model’s capacity to be good at any particular one. But only one future is actually going to happen. So why not train our model once this future happens?
Recently, Hardt and Sun (2023) extended this idea to modern language models and observed a similar benefit of larger effective model capacity after test-time training, and Hübotter et al. (2024) further improved these results through better strategies for neighbor selection. In addition, Hübotter et al. (2025) showed that test-time training on neighbors from the training set is also effective with RL for reasoning tasks, and Bagatella et al. (2025) developed the same idea for visual-motor tasks.
#### 5.2.2 TTT for Novel Instances: Better Generalization
As models become larger today, their competence is often limited not by their capacity, but by the amount of available training data, especially when they need to generalize to novel test instances that are “out-of-distribution”. In this case, it is even harder to prepare for all possible test instances in the future, especially the novel ones, with a static model. But once a specific test instance is given, we can use it to generate relevant data, which we can then use for training Sun et al. (2020) . In other words, the “neighbors” for TTT do not have to come from the training set; they can also be generated on-the-fly.
Since the test instance is unlabeled, one way to make it useful for training is through self-supervision, which generates new pairs of inputs and labels for an auxiliary task such as masked reconstruction (e.g., BERT Devlin et al. (2018) and MAE He et al. (2021) ). While the auxiliary task is different from the main prediction task, improving performance in one can help the other through their shared representations. This form of TTT can significantly improve generalization under distribution shifts Sun et al. (2020) ; Gandelsman et al. (2022) .
Recently, TTT has been an important part of AlphaProof Hubert et al. (2025) , which achieved IMO silver-medal standard in 2024. Given each test problem, their system first generates a targeted curriculum of easier problems by prompting a language model, and then performs reinforcement learning on the generated data. Another recent work, Akyurek et al. Akyürek et al. (2024) , found TTT effective for few-shot reasoning tasks such as ARC-AGI. Their system generates augmentations of the few-shot demonstrations in the test problem then performs supervised learning. In Li et al. (2025) , authors perform policy gradients at test time using the policy itself as an evaluator of solutions, similar to using LMs as a judge. Further, they optimize token representations with policy gradients, as opposed to optimizing the policy.
Three closest and concurrent works perform test-time training: MiGrATe Phan et al. (2025) , ThetaEvolve Wang et al. (2025a) , and EvoTune Surina et al. (2025) . All three combine per-instance RL updates with various replay/reuse mechanisms, and typically use PPO/GRPO/DPO-style updates for LMs Schulman et al. (2017) ; Guo et al. (2025) ; Rafailov et al. (2023) . Relative to Phan et al. (2025) ; Surina et al. (2025) , our contribution is to tailor both the learning objective and the reuse rule to the discovery goal, rather than largely standard RL or evolutionary baselines; also test in more realistic discovery tasks with human expert baselines. Compared to ThetaEvolve, TTT-Discover using the same model and compute budget still produces significant improvements (Table 2 ), which we attribute to our entropic objective and PUCT-based reuse instead of more complicated and brittle heuristics in evolutionary algorithms.
In an earlier work Bello et al. (2016) , the authors train a neural policy with policy gradients to directly output solutions to combinatorial problems like TSP, using (negative) tour length as reward, and they study both training across many instances and per-instance learning at test time.
### 5.3 RL on One Example
One Example RL Wang et al. (2025b) is relevant as they also train on a single problem. To be specific, they train on one example from a dataset, such as the MATH training set. They show that a policy trained with on one such problem with RL generalizes to other problems in the same dataset. In contrast, TTT-Discover trains on the test problem itself, where the goal is not to generalize but to solve this specific problem.
### 5.4 RL on the Test Set
TTRL Zuo et al. (2025) trains on an entire test set of problems using majority voting as pseudo-labels for reward estimation. In contrast, TTT-Discover trains on a single test problem with a continuous verifiable reward, where the goal is not to improve average performance across a set of problems but to find one exceptional solution.
## 6 Future Work
The current form of our method can only be applied to problems with continuous rewards, and the most important direction for future work is test-time training for problems with sparse or binary rewards, or problems in non-verifiable domains.
## Acknowledgments
We thank Matej Sirovatka, Davide Torlo, Eric Sun, Alex Zhang, Mark Saroufim, for reviewing our results and letting us cite their reviews in this paper. We would like to thank Amanda Moran and Nvidia for their support with the compute infrastructure; Charles Frye and Modal team, Clare Birch, John Schulman, Tianyi Zhang, Yangjun Ruan, and Thinking Machines Lab team for compute credits supporting this project; Anika Gupta, Zacharie Bugaud, and the broader Astera Institute for their support in various phases of the project; Matej Sirovatka, Alex Zhang, Mark Saroufim and the broader GPUMode community, and Simon Guo for their support in various phases of this project. We thank Mehmet Hamza Erol and Vipul Sharma for their short-term contributions. We thank Luke Bailey for feedback on this draft. Mert would like to thank Begum Ergun, Fatih Dinc, Omer Faruk Akgun, Ramiz Colak, Yigit Korkmaz for their support at every phase of this project.
## Appendix A Training details
Our hyperparameters are fixed throughout almost all experiment. For almost all applications we used a KL penalty coefficient of 0.1 0.1 . For algorithm engineering, we used a KL coefficient of 0.01 0.01 . We present details on our objective function and the reuse algorithm below.
### A.1 Entropic utility objective
We define the entropic utility objective explored also in the concurrent work [ 29 ] : J β ( θ ; s ) 𝐵 log E τ ∼ π θ ( ⋅ ∣ s ) [ e β r ( τ , s ) ] . J_{\beta}(\theta;s)\;\coloneqq\;\log\mathbb{E}_{\tau\sim\pi_{\theta}(\cdot\mid s)}\!\left[e^{\beta r(\tau;s)}\right].
The gradient of this objective yields ∇ θ J β ( θ ; s ) = E τ ∼ π θ ( ⋅ ∣ s ) [ ∇ θ log π θ ( τ ∣ s ) w β ( τ ∣ s ) ] , w β ( τ ∣ s ) = e β r ( τ , s ) E π θ [ e β r ( τ , s ) ] , A β ( τ ∣ s ) = w β ( τ ∣ s ) − 1 , \nabla_{\theta}J_{\beta}(\theta;s)=\mathbb{E}_{\tau\sim\pi_{\theta}(\cdot\mid s)}\!\left[\nabla_{\theta}\log\pi_{\theta}(\tau\mid s)\;w_{\beta}(\tau\mid s)\right],\quad w_{\beta}(\tau\mid s)\;=\;\frac{e^{\beta r(\tau;s)}}{\mathbb{E}_{\pi_{\theta}}[e^{\beta r(\tau;s)}]},\quad A_{\beta}(\tau\mid s)\;=\;w_{\beta}(\tau\mid s)-1, since E π θ [ w β ( τ ∣ s ) ] = 1 \mathbb{E}_{\pi_{\theta}}[w_{\beta}(\tau\mid s)]=1 , we get A β A_{\beta} as the mean baselined advantage. The remaining question is how to set β \beta . [ 29 ] recommends value β = 2 \beta=2 , yet we found it tricky to set it. Later in the training, improvements become harder, and unless β \beta is adjusted carefully advantages can become very small. Early in the training, a large β \beta can cause instabilities.
Adaptive β \beta . Define the auxiliary tilted distribution induced by the entropic weights, q β ( τ ∣ s ) = π θ ( τ ∣ s ) exp ( β r ( τ , s ) ) E π θ [ exp ( β r ( τ , s ) ) ] , w β ( τ ∣ s ) = q β ( τ ∣ s ) π θ ( τ ∣ s ) . q_{\beta}(\tau\mid s)\;=\;\frac{\pi_{\theta}(\tau\mid s)\exp(\beta r(\tau;s))}{\mathbb{E}_{\pi_{\theta}}[\exp(\beta r(\tau;s))]},\qquad w_{\beta}(\tau\mid s)\;=\;\frac{q_{\beta}(\tau\mid s)}{\pi_{\theta}(\tau\mid s)}. Then w β w_{\beta} is exactly the density ratio that appears in the entropic policy-gradient update, so β \beta controls the effective step size induced by this reweighting. We choose β ( s ) \beta(s) by enforcing a KL budget on the auxiliary distribution, KL ( q β ( s ) ( ⋅ ∣ s ) ∥ π θ ( ⋅ ∣ s ) ) = γ , \mathrm{KL}\!\big(q_{\beta(s)}(\cdot\mid s)\,\|\,\pi_{\theta}(\cdot\mid s)\big)\;=\;\gamma, analogous to Relative Entropy Policy Search, where the temperature is set by an exponential tilt under a relative-entropy constraint [ 51 ] . In words, β ( s ) \beta(s) is increased only until the KL budget is exhausted, ensuring the induced reweighting, and hence the update, does not move too far from π θ ( ⋅ ∣ s ) \pi_{\theta}(\cdot\mid s) . We fix γ = ln 2 \gamma=\ln 2 throughout our experiments.
Batch estimator. Given N N rollouts from the same s s with rewards { r n } n = 1 N \{r_{n}\}_{n=1}^{N} , the empirical sampling distribution is uniform on the batch, u ( n ) = 1 / N u(n)=1/N . The induced reweighting on the batch is q β ( n ) = e β r n ∑ m = 1 N e β r m , q_{\beta}(n)=\frac{e^{\beta r_{n}}}{\sumop\displaylimits_{m=1}^{N}e^{\beta r_{m}}}, and we set β ( s ) \beta(s) by solving the weight-concentration constraint KL ( q β ∥ u ) = ∑ n = 1 N q β ( n ) log ( N q β ( n ) ) = γ \mathrm{KL}(q_{\beta}\|u)=\sumop\displaylimits_{n=1}^{N}q_{\beta}(n)\log\!\big(Nq_{\beta}(n)\big)=\gamma via simple bisection search over β ≥ 0 \beta\geq 0 . With β ^ ( s ) \hat{\beta}(s) , we compute LOO entropic advantages using r max = max n r n r_{\max}=\max_{n}r_{n} , and an ϵ \epsilon in the denominator for numerical stability: Z ^ − n = 1 N − 1 ∑ m , n exp ( β ^ ( s ) ( r m − r max ) ) , A n = exp ( β ^ ( s ) ( r n − r max ) ) Z ^ − n + ε − 1 . \qquad\hat{Z}_{-n}=\frac{1}{N-1}\sumop\displaylimits_{m\neq n}\exp(\hat{\beta}(s)(r_{m}-r_{\max})),\qquad A_{n}=\frac{\exp(\hat{\beta}(s)(r_{n}-r_{\max}))}{\hat{Z}_{-n}+\varepsilon}-1.
##### Discussion.
States where improvements are consistently small (e.g. high-value / near-goal states) tend to make the batch weights q β ( n ) q_{\beta}(n) less peaky for a given β \beta , so the constraint typically permits a larger β ( s ) \beta(s) . In contrast, states that occasionally yield a few very large improvements (often earlier in training or low-value states with large headroom) make q β q_{\beta} concentrate quickly as β \beta grows; the same KL budget then forces a smaller β ( s ) \beta(s) , preventing the update from being dominated by a handful of outlier trajectories while still preferring better-than-average rollouts. Finally, this estimator is invariant to shifting or scaling the reward by a constant, i.e., r ( τ ) r(\tau) and r ′ ( τ ) = w r ( τ ) + b r^{\prime}(\tau)=wr(\tau)+b yield the same advantage for w ∈ R + w\in\mathbb{R}^{+} and b ∈ R b\in\mathbb{R} .
### A.2 PUCT Prioritization