TTRL: Test-Time Reinforcement Learning — part 2/3

Cleaned-text code points [15570, 30718). Bibliography omitted here; source.md includes it.

To gain a deeper understanding of the underlying mechanisms of TTRL , we conduct an analysis of the model’s training dynamics by tracking the average (pass@1/avg@16) and majority (maj@16) scores throughout the training process. Given that majority voting serves as the basis for generating training signals, examining its performance trajectory is essential for understanding how it functions. Furthermore, we investigate whether TTRL improves pass@1 at the cost of a reduction in maj@16 performance. Figure 6 illustrates the TTRL training dynamics on AMC with Qwen2.5-Math-1.5B as the base model. It is notable that, as training progresses, both metrics demonstrate a consistent upward trend. This indicates that TTRL is not simply approaching the initial model’s majority voting performance. Due to its dynamic nature, TTRL can generate higher-quality supervision signals as its capabilities improve. Moreover, through TTRL ’s use of RL for TTT, by converting voting-based pseudo-labels into reward signals, it enhances the effective supervision quality (e.g., accuracy; see Q2 4.2 ), while decoupling learning from the limitations imposed by maj@n.

## 4 Analysis and Discussions

### 4.1 Q1: How Well Can TTRL Perform?

We analyze the potential performance of TTRL using two upper bounds. The first upper bound is the maj@n of the initial model. The second upper bound is direct training on benchmark data, which assumes access to ground-truth labels and thus leaks label information to the policy model.

TTRL is Supervised by maj@n Yet Surpasses It. Since TTRL utilizes the model’s own majority-voted outputs for RL, this voting-based performance of the initial model can intuitively be regarded as an upper bound of the final performance. This upper bound is also the performance limit of traditional self-training methods ( Huang et al., 2022 ) , which select self-generated CoT through majority voting for supervised fine-tuning (SFT). However, we observe a surprising phenomenon: after training, the model not only matches but also surpasses the expected upper bound, suggesting that it exceeds the performance limit of the original model, which also serves as its initial supervision signal. Figure 6 illustrates this remarkable result, where it can be observed that the final avg@16 score exceeds the initial maj@16 score by more than 20 points. Furthermore, we perform additional evaluations of TTRL on Qwen2.5-Math-7B across various benchmarks, using more samples per question to enable more reliable assessment. Figure 7 shows results. It can be observed that TTRL avg@64 consistently outperforms Qwen2.5-Math-7B maj@64 across all benchmarks, with a considerable margin. Through a self-reinforcing loop, the model “lifts itself up by its own bootstraps” , evolving beyond the anticipated performance ceiling. Moreover, the performance of TTRL further improves when majority voting is applied.

TTRL’s Performance Gains Approach Training on the Benchmark. The motivation of TTRL is to estimate labels using majority voting to obtain more accurate rewards, facilitating effective self-improvement through RL on the data without ground-truth labels. Therefore, a natural upper bound of TTRL is performing RL directly on the test data, denoted as RL (leakage). Although this setting is rarely adopted or studied due to the issue of information leakage, it represents the most efficient way to improve performance on the particular dataset, with efficiency that far exceeds traditional training-evaluation paradigms. We use Qwen2.5-Math-7B to perform both TTRL and RL (leakage) on MATH-500 and conduct evaluations. Figure 8 shows results. Surprisingly, we find that the performance curve of TTRL closely approaches that of RL (leakage). This suggests that:

1. TTRL can achieve a level of self-improvement comparable to that of supervised learning (even in the information leakage scenario) through RL in an unsupervised setting. This indicates its substantial efficiency and performance gains.

2. TTRL provides evidence that even small LLMs can now effectively self-improve on input-only challenging tasks through RL, enabling continual learning. Results on Qwen2.5-Math-1.5B further support this observation: starting from a subpar performance of 32.7 32.7 on MATH-500, the model improved by 123.2 % 123.2\% to reach 73.0 73.0 , demonstrating clear self-improvement through TTRL .

### 4.2 Q2: Why Does TTRL Work?

This section presents a progressive analysis of the factors enabling TTRL to achieve stable and effective RL under unsupervised conditions. Our analysis identifies three key factors: label estimation, reward calculation, and online learning.

#### Label Estimations.

A direct difference between TTRL and standard RL algorithms is that TTRL involves label estimation, which introduces reward inaccuracies. We believe that TTRL works despite these inaccuracies due to the following two reasons. (i) Existing studies have shown that RL can tolerate a certain degree of reward inaccuracy. Moreover, RL tends to generalize better than SFT, which often relies on memorizing training data ( Chu et al., 2025 ) . In RL, rewards are typically vague and serve primarily as directional signals for exploration, leading to RL’s robustness to reward noise ( Razin et al., 2025 ) . (ii) Prior work has also examined what constitutes a good reward model from an optimization perspective, revealing that more accurate reward models are not necessarily better teachers ( Wang et al., 2020 ) . Therefore, reward signals estimated by the policy model itself may offer more suitable guidance for learning.

#### Reward Calculations.

When the model is capable of estimating accurate labels via majority voting, the reward and subsequently training are generally reliable. However, a natural question arises: Why does TTRL remain effective even when the model fails to estimate accurate labels via majority voting on challenging benchmarks such as AIME 2024? The most fundamental reason lies in the mechanism by which the verifier computes rewards in RL. For tasks such as mathematics, the verifier works based on “comparison” to obtain rule-based rewards by checking whether the predicted answer matches the given “label.” This mechanism can lead to the phenomenon of “ Lucky Hit ”: for an incorrectly predicted answer, even if the estimated label does not match the ground truth label, as long as it differs from the predicted answer, the verifier will still output a negative reward, and this is exactly the correct reward that we expect, as illustrated in Figure 10 . In other words, it is sufficient that the estimated label differs from the predicted answer for the verifier to assign the correct negative reward. To provide a more detailed case study, we examine the performance of TTRL on the AIME 2024 using Qwen2.5-Math-7B . Figure 9 presents the variation curves of the three metrics, as described in Appendix B . We identify two main reasons why TTRL remains effective on AIME 2024:

1. Reward robustness enabled by multiple outputs within a rollout. First, rewards are denser than labels, allowing for more opportunities to recover useful reward signals even when the estimated label is inaccurate. For example, even when the predicted label is incorrect, alternative outputs within the same rollout can still yield correct or high-quality rewards, as shown in Figure 10 , whereas a rollout containing only a single output would not provide such flexibility. This makes the overall reward signal more robust to errors in pseudo-label estimation.

2. High reward accuracy due to scattered incorrect predictions. Second, counterintuitively, when the model has weaker capability, the majority voting rewards of TTRL may be more accurate. As shown in Figure 9 , although the initial label estimation through majority voting achieves an accuracy of only 37 % 37\% , the reward accuracy reaches an impressive 92 % 92\% . By examining the model outputs, we find that this is because the model’s responses are highly scattered and consistently incorrect, as shown in Figure 10 . A result consistent with this observation is that, for the base model, the most frequently predicted answer accounts for only 16.6 % 16.6\% of all predictions, indicating that the outputs are highly scattered. Therefore, even when the labels are not accurately estimated, due to “ Lucky Hit ”, most outputs can still receive correct rewards. Moreover, the poorer the model’s performance, the more mistakes it tends to make, which paradoxically leads to more accurate reward estimation. An empirical observation supporting this view is the comparison between the label accuracy and reward accuracy, as shown in Figure 9 . Although the label accuracy rarely exceeds 50 % 50\% , the reward accuracy remains consistently high, staying above 75 % 75\% . This high reward accuracy provides a reliable foundation for effective self-improvement on test data.

#### Online Learning.

TTRL is designed based on an online RL approach, whereas traditional self-training and test-time training methods operate in an offline manner. The online nature of TTRL enables the model to improve its capabilities during the application, which in turn leads to more accurate labels generated through voting. As a result, the quality of the supervision signal improves, allowing for truly sustainable self-evolution. As shown in Figure 6 , this dynamic learning process leads to a complementary improvement of performance in both pass@1 and maj@n.

### 4.3 Q3: When Might TTRL Fail?

At the algorithmic level, TTRL is not fundamentally different from existing RL algorithms and therefore inherits several of their characteristics, such as sensitivity to data difficulty, strong reliance on priors, and risk of collapse under certain conditions. At the implementation level, these issues are further amplified by the constraints of TTRL , which estimates labels via majority voting and operates exclusively on test data that is both sparse and previously unseen, potentially resulting in failures in certain scenarios. In our preliminary experiments, we identified two potential issues:

#### Inappropriate RL Hyperparameters.

Hyperparameter settings play a crucial role in RL training, varying across projects 2 2 2 https://github.com/TsinghuaC3I/Awesome-RL-Reasoning-Recipes and often leading to training failures. The influence of hyperparameters is further amplified in TTRL due to potential noise in reward estimation and the characteristics of the test data. Figure 11 presents a comparison of several unsuccessful attempts on AIME 2024. Both of these failed attempts exhibit persistently high entropy that does not diminish throughout training, consistent with findings of prior work ( He et al., 2025 ) . In our preliminary experiments, we identified two key hyperparameters that can critically affect training stability and success:

• Temperature: Setting the temperature to 1.0 1.0 , as opposed to 0.6 0.6 , increases the model’s output entropy. This promotes more extensive exploration and allows the model to make better use of its prior knowledge for self-improvement, which is particularly important when addressing challenging benchmarks.

• Episodes: Given the substantial variation in size and difficulty across datasets, smaller and more difficult datasets need more episodes to achieve sufficient exploration.

#### Lack of Prior Knowledge on Target Task.

Prior knowledge plays a crucial role in RL, often determining the success or failure of the TTRL learning process 3 3 3 https://ysymyth.github.io/The-Second-Half/ . This is mainly because the test data generally exhibits higher difficulty and introduces new features, but TTRL does not incorporate mechanisms such as data filtering to support curriculum learning.

Therefore, for the same backbone, TTRL fails if the model’s prior knowledge is insufficient to handle the complexity of the data. To further validate this hypothesis, we conduct an ablation study on MATH-500. We divide MATH-500 into five subsets according to its annotated difficulty levels, ranging from 1 1 to 5 5 , and apply TTRL to each subset independently, using Qwen2.5-Math-1.5B . We then compare the results to those of the backbone, as shown in Table 3 . We observe that as the question difficulty increases, both the performance improvement and length reduction ratios tend to decrease. This suggests that the available prior knowledge of the backbone is insufficient to support learning on more challenging questions.

## 5 Related Works

### 5.1 Test-Time Scaling

Test-Time Scaling (TTS) is designed to enhance the capabilities of Large Language Models (LLMs) in handling complex tasks by increasing computational resources at test time. Prior research ( Snell et al., 2024 ; Liu et al., 2025a ) indicates that TTS is more efficient than scaling during pre-training ( Kaplan et al., 2020 ) . Therefore, reallocating the same computational resources from pre-training to test-time could yield greater improvements in model performance. Current studies on TTS fall into two categories ( Welleck et al., 2024 ) : parallel generation and sequential generation. Parallel generation involves LLMs producing multiple candidate responses (self-consistency ( Wang et al., 2022 ; Chen et al., 2023 ) , best-of-N ( Stiennon et al., 2020 ; Nakano et al., 2021 ) ), decision steps (Monte Carlo Tree Search ( Zhou et al., 2023 ; Xie et al., 2024 ) ), or tokens (Reward-guided Search ( Deng & Raffel, 2023 ; Khanov et al., 2024 ) ) during inference. Subsequently, an aggregation strategy is applied to integrate these candidates, commonly using process reward models ( Lightman et al., 2023 ; Wang et al., 2023 ; Zhang et al., 2025a ) . Concurrently, sequential generation focuses on extending the LLMs’ output to include longer responses with reflective and chain-of-thought (CoT) processes ( Wei et al., 2022 ; Madaan et al., 2023 ) . Although prompting techniques are widely adopted, they are often constrained by the capabilities of the underlying models. Notably, DeepSeek-R1 ( Guo et al., 2025 ) is a representative advancement in this area, achieving extended reasoning capabilities in pre-trained language models through outcome-based reinforcement learning (RL), more specifically group relative policy optimization ( Shao et al., 2024 ) . Compared to the first approach, which requires intensive process-level supervision ( Yuan et al., 2024 ) , the second approach is more scalable due to its reliance on rule-based rewards.

Beyond the aforementioned methods that focus on scaling test-time inference computation, another approach to increasing test-time computing is Test-Time Training (TTT) . We introduce the relationship between these terminologies in Appendix C . While prior work has primarily focused on applications such as video generation and understanding ( Hardt & Sun, 2024 ; Dalal et al., 2025 ) , and to some extent on large language models ( Wang et al., 2025a ; Akyürek et al., 2024 ) , the integration of test-time scaling with reinforcement learning remains largely underexplored.

### 5.2 RL for Reasoning