TTRL: Test-Time Reinforcement Learning — part 3/3

Cleaned-text code points [30718, 41305). Bibliography omitted here; source.md includes it.

Reinforcement Learning (RL) ( Sutton et al., 1998 ) plays a critical role in enhancing the instruction-following capabilities of Large Language Models (LLMs), particularly through approaches like Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) . RLHF aligns base models with human preferences using algorithms such as Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , where preference modeling is essential. Recently, Large Reasoning Models (LRMs), such as DeepSeek-R1 ( Guo et al., 2025 ) , have demonstrated the significance of RL in improving reasoning abilities using rule-based rewards, as exemplified by GRPO ( Shao et al., 2024 ) . Unlike RLHF, which is tailored to open-domain instructions, GRPO is specifically designed to elicit long CoT ( Wei et al., 2022 ) reasoning in mathematical problem-solving. Recent studies have focused primarily on improving the training stability of rule-based RL methods like GRPO and PPO ( Cui et al., 2025 ; Yu et al., 2025 ; Liu et al., 2025b ) . However, these methods typically train LLMs only on supervised training data, while inference involves generating extended CoT reasoning on unseen test problems. Moreover, current RL approaches ( Hu et al., 2025a ; Wei et al., 2025 ) depend on verifiable outputs—such as solutions in mathematics or code—that can provide reliable reward signals.

Previous studies have explored self-rewarding ( Yuan et al., 2025 ; Prasad et al., 2024 ) and self-play training ( Chen et al., 2024 ) for unlabeled data. However, these works primarily focus on open-domain instruction following ( Yuan et al., 2025 ; Chen et al., 2024 ) rather than mathematical reasoning or employ preference-based optimization strategies ( Prasad et al., 2024 ) such as DPO ( Rafailov et al., 2023 ) instead of online reinforcement learning algorithms. In addition to these studies, we identified several concurrent works ( Xu et al., 2025 ; Zhang et al., 2025b ; Zhao et al., 2025 ) , that explore self-supervised and semi-supervised reasoning using reinforcement-like methods. The key distinction lies in reward estimation: we employ majority voting, which is derived from the model itself and mitigates reward hacking. Recently, Wang et al. (2025b) demonstrated that using a single training example to incentivize the mathematical reasoning capabilities of LLMs is effective, showing substantial improvements even under minimal supervision. We acknowledge that future research integrating the insights and strengths of these approaches could lead to more robust reasoning models in the era of experience ( Silver & Sutton, 2025 ) . TTRL offers a preliminary attempt at RL with self-labeled rewards, advancing toward learning from streams of experience.

## 6 Conclusion

In this paper, we propose Test-Time Reinforcement Learning ( TTRL ), a novel framework for training large language models with Reinforcement Learning (RL) on test data without access to ground-truth labels. A key component of TTRL is its majority voting reward function, which generates rule-based rewards based on consensus among model predictions. Our experiments demonstrate the strong potential of TTRL , achieving consistent improvements across a variety of models and tasks. We view TTRL as a preliminary step toward RL with self-labeled rewards, marking an important direction of learning from continuous streams of experience.

## 7 Limitations and Future Works

#### Limitations

This work represents an initial exploration of test-time reinforcement learning using self-labeled rewards. While our experimental results are promising, several aspects require further investigation. In particular, we plan to conduct a more in-depth analysis of the impact of prior knowledge and hyperparameter configurations, both of which play critical roles in reinforcement learning dynamics. We will provide comprehensive discussions and ablation studies in future revisions of this paper.

#### Future Works

Building on our findings, we identify several directions for future research:

• Theoretical Analysis: Developing a formal convergence analysis of TTRL , particularly focusing on its ability to optimize toward the two upper bounds in § \lx@sectionsign 4.1 .

• Online Learning with Streaming Data: Extending TTRL to real-time learning scenarios, where models interact with continuously arriving data and adapt dynamically, that is Test-Time Adaptation ( Liang et al., 2025 ) .

• Large-Scale Self-Supervised RL Training: Scaling up TTRL to massive datasets and models to explore its potential in self-supervised regimes without human-labeled data.

• Agentic Tasks and Scientific Discovery: Applying TTRL to more complex, open-ended domains such as agentic tasks and multi-step scientific reasoning.

## Appendix A Additional Results

Table 4 shows pass@1 results using greedy decoding. For the two base models, we further include comparisons with their instruct versions that have undergone large-scale post-training. In addition, we include for reference current leading “R1-Zero-Like” models with similar backbones, which are extensively trained using RL: DeepSeek-R1-Distill-1.5B&7B ( Guo et al., 2025 ) , SimpleRL-Zero-7B ( Zeng et al., 2025 ) , PRIME-Zero-7B ( Cui et al., 2025 ) , OpenReasoner-Zero-7B ( Hu et al., 2025b ) , Oat-Zero-1.5B&7B ( Liu et al., 2025b ) , and LIMR ( Li et al., 2025 ) . Note that TTRL has a different setup from the previous models, which makes the comparison seem unfair.

On the highly challenging mathematical reasoning benchmark AIME 2024, TTRL achieves a substantial improvement of 159.3 % 159.3\% , surpassing all models trained on large-scale datasets. Furthermore, when applied to Qwen2.5-Math-7B , TTRL yields an average improvement of 84.1 % 84.1\% across three benchmarks. Figure 12 shows two curves of TTRL on AIME 2024 with Qwen2.5-Math-7B as an example.

## Appendix B Training Metrics

Given the absence of ground-truth labels in the test data, evaluating the performance of TTRL throughout the training process presents a challenge. To mitigate this limitation, we introduce a set of training-time metrics specifically designed to monitor and assess the effectiveness of TTRL . These metrics inform the selection of the optimal checkpoint and provide valuable insights regarding training dynamics.

• Entropy : Measures the uncertainty of the model’s generation.

• Majority Voting Reward : Rule-based rewards computed from the majority-voted label.

• Majority Ratio : The frequency of the most common answer within a rollout.

Furthermore, we define several metrics that rely on access to ground-truth labels, which allow for a deeper analysis of the model’s behavior during training:

• Label Accuracy (maj@n) : Indicates whether the estimated label matches ground-truth.

• Reward Accuracy : Indicates the proportion of majority voting rewards (computed from the estimated label) that match rewards computed from the ground-truth label.

• Ground-Truth Ratio : The frequency of the ground-truth answer within a rollout.

## Appendix C Terminology

Test-time scaling refers to increasing computational resources during test time, which can be categorized into test-time training and test-time inference. These two approaches are complementary. We will provide an introduction below.

### C.1 Test-Time Training (TTT)

Test-Time Training (TTT) is a technique for adapting a pre-trained model at inference time to improve generalization under distribution shifts. Let f θ f_{\theta} denote a model trained on a source domain 𝒟 ​ s = { ( x i , y i ) } ​ i = 1 N \mathcal{D}s=\{(x_{i},y_{i})\}{i=1}^{N} , where x i ∈ 𝒳 x_{i}\in\mathcal{X} , y i ∈ 𝒴 y_{i}\in\mathcal{Y} , and θ \theta represents the learned parameters. During standard inference, the model is evaluated on test samples x t ∼ 𝒟 t x_{t}\sim\mathcal{D}_{t} with fixed parameters θ \theta , where 𝒟 t ≠ 𝒟 s \mathcal{D}_{t}\neq\mathcal{D}_{s} .

In contrast, TTT allows the model to adapt to each test sample x t x_{t} by minimizing an auxiliary self-supervised loss ℒ aux \mathcal{L}_{\text{aux}} , without access to labels y t y_{t} . The model parameters are updated online with the auxiliary task, which is typically designed to be label-free and consistent with the main task.

### C.2 Test-Time Inference (TTI)

Test-Time Inference (TTI) refers to the strategy of enhancing the performance of a large language model during inference by allocating additional computational resources. Formally, let f θ f_{\theta} denote a language model with parameters θ \theta , and let x x be an input prompt. The model generates an output y y by sampling from the conditional distribution p θ ​ ( y ∣ x ) p_{\theta}(y\mid x) . TTI techniques aim to improve the quality of y y by employing methods such as generating multiple candidate outputs and selecting the best one based on a scoring function, or by refining the output through iterative processes ( Welleck et al., 2024 ) .

One common approach involves generating N N candidate outputs { y 1 , y 2 , … , y N } \{y_{1},y_{2},\ldots,y_{N}\} and selecting the optimal output y ∗ y^{*} using a scoring function s ⁡ ( y , x ) s(y,x) :

y ∗ = arg ⁡ max y i ⁡ s ⁡ ( y i , x ) \displaystyle y^{*}=\arg\max_{y_{i}}s(y_{i},x) (4)

The scoring function s ⁡ ( y , x ) s(y,x) can be instantiated in various ways, such as: 1. Majority Voting (MV): Selecting the most frequent output among the candidates.

2. Best-of-N (BoN): Using reward models to score each candidate, then selecting the highest-scoring one.

3. Weighted BoN: Integrating MV and BoN strategies to leverage their respective strengths.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .