TTRL: Test-Time Reinforcement Learning — part 1/3
Cleaned-text code points [0, 15570). Bibliography omitted here; source.md includes it.
# TTRL: Test-Time Reinforcement Learning
###### Abstract
This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning ( TTRL ), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 𝟐𝟏𝟏 % \mathbf{211\%} on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the maj@n metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model maj@n, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL ’s potential for broader tasks and domains.
## 1 Introduction
Recent advances in Large Reasoning Models (LRMs), such as DeepSeek-R1 ( Guo et al., 2025 ) and OpenAI’s o1 ( Jaech et al., 2024 ) , have demonstrated that Reinforcement Learning (RL) is essential for enhancing long chain-of-thought (CoT) reasoning ( Wei et al., 2022 ) through training on expensive human-annotated data. These models achieve remarkable performance on a range of highly challenging tasks. For example, OpenAI’s o3 attains a 75.7 % 75.7\% success rate on ARC-AGI-1. However, complex and unlabeled questions continuously emerge, posing significant challenges. For instance, o3 solves only 4 % 4\% of problems on the recently released ARC-AGI-2 benchmark (2025) 1 1 1 https://arcprize.org/ . Addressing such tasks typically involves scaling up training with more data and computational resources, and it may still fail to yield strong performance on these tasks. Silver & Sutton (2025) has recently advocated for a transition to the “era of experience,” emphasizing the limitations of existing AI systems that rely heavily on human supervision, as well as the importance of enabling models to self-evolve through experience.
Further building upon the substantial progress of LRMs, it naturally motivates a promising direction in which AI systems autonomously improve via RL on unlabeled data by directly engaging in self-experience and learning, thereby pushing the boundaries of RL and further advancing the frontier of AI capabilities. Such self-evolvement can be broadly categorized into two modes: adaptation to test-time data, which enables models to tackle harder benchmarks such as ARC-AGI-2, and training on external unlabeled data, which unlocks more training data beyond labeled corpora. This work focuses on the adaptation to test-time data, which has been extensively studied under the paradigm of Test-Time Training (TTT) ( Sun et al., 2019 ; Sun et al., 2024 ; Behrouz et al., 2024 ; Akyürek et al., 2024 ) . TTT has received increasing attention recently. These approaches adapt model parameters at test time by exploiting the structure and distributional properties of incoming test data.
Therefore, we aim to fully advance AI evolution by updating models at test time using RL, thereby enhancing their generalization to previously unseen data. However, this introduces a critical challenge: How to obtain rewards for RL at test-time? This also highlights a broader limitation of current RL approaches. Despite their promise, most existing methods still rely heavily on labeled data, which significantly limits their scalability. As real-world tasks continue to increase in both complexity and volume, large-scale annotation for RL becomes increasingly impractical, posing a substantial barrier to the continual improvement of state-of-the-art models.
We introduce Test-Time Reinforcement Learning ( TTRL ), which performs test-time training through RL. TTRL employs repeated sampling strategies in the rollout phase to accurately estimate the label and compute rule-based rewards, thereby enabling RL on unlabeled data. By incorporating effective majority voting rewards, TTRL facilitates efficient and stable RL in the absence of ground truth labels. As previously highlighted, the emergence of more challenging tasks will inevitably lead to larger proportions of unlabeled data. TTRL directly addresses the problem of training models via RL without explicit supervision, investigating a model’s ability to explore and learn in this challenging yet critical setting. Essentially, TTRL enables the model to generate its own experiences, estimate rewards, and improve its performance over time.
In experiments, applying TTRL to Qwen2.5-Math-7B results in an improvement on AIME 2024 of 𝟐𝟏𝟏 % \mathbf{211\%} ( 12.9 12.9 to 40.2 40.2 ), with an average gain of 𝟕𝟔 % \mathbf{76\%} across AIME 2024, AMC, MATH-500, and GPQA. These improvements are achieved through self-evolution without any labeled training data and further generalize to other tasks. TTRL not only enhances performance on pass@1 but also improves TTS through majority voting. Moreover, our preliminary experiments suggest that TTRL is effective across models of different scales and types and that it can be integrated with existing RL algorithms. We also found that TTRL exhibits favorable characteristics such as a high-performance ceiling. These observations highlight its potential to substantially reduce reliance on human annotations, enabling continual learning and scaling RL to large-scale unsupervised training. Below are several key takeaways:
## 2 Test-Time Reinforcement Learning (TTRL)
Unlike traditional RL, where the agent learns from known reward signals, TTRL operates on unlabeled test data. In other words, the model must learn and adapt without access to explicit supervision. Our task is defined as follows:
### 2.1 Methodology
Figure 2 illustrates how our approach, TTRL , tackles this challenge. Given a state represented by the prompt x x , the model acts by producing an output y y sampled from a policy π θ ( y ∣ x ) \pi_{\theta}(y\mid x) parameterized by θ \theta . To construct a reward signal without ground-truth labels, we generate multiple candidate outputs { y 1 , y 2 , … , y N } \{y_{1},y_{2},\ldots,y_{N}\} from the model through repeated sampling. A consensus output y ∗ y^{*} is derived, for instance, by majority voting or another aggregation method, serving as a proxy for the optimal action. The environment then provides a reward r ( y , y ∗ ) r(y,y^{*}) based on the alignment between the sampled action y y and the consensus action y ∗ y^{*} . The RL objective is thus to maximize the expected reward: max θ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y , y ∗ ) ] , \displaystyle\max_{\theta}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[r(y,y^{*})], (1) and parameters θ \theta are updated through gradient ascent: θ ← θ + η ∇ θ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y , y ∗ ) ] , \displaystyle\theta\leftarrow\theta+\eta\nabla_{\theta}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[r(y,y^{*})], (2) where η \eta denotes the learning rate. This approach enables the model to adapt during inference, effectively improving its performance on distribution-shifted inputs without the need for labeled data.
The pseudo-code of the majority voting reward function.
### 2.2 Majority Voting Reward Function
The majority voting reward is determined by first estimating a label through majority voting. This estimated label is then used to calculate rule-based rewards, which serve as the final rewards. Given a question x x , we first input x x into the LLM to generate a set of outputs. An answer extractor then processes these outputs to obtain the corresponding predicted answers, denoted as P = { y ^ i } i = 1 N P=\{\hat{y}_{i}\}_{i=1}^{N} . We first follow Equation 4 over P P to estimate a label, with majority voting as the scoring function s ( y , x ) s(y,x) to get y y , the most frequently occurring prediction in P P . The majority-voted prediction y y is then used as the estimated label to compute rule-based rewards ( Guo et al., 2025 ) . The reward function is: R ( y ^ i , y ) = { 1 , if y ^ i = y , 0 , otherwise . \displaystyle R(\hat{y}_{i},y)=\begin{cases}1,&\text{if }\hat{y}_{i}=y,\\ 0,&\text{otherwise}.\end{cases} (3) Listing presents the pseudo-code of the reward function.
## 3 Experiments
### 3.1 Experimental Setup
#### Models
To evaluate the generalizability of TTRL across different backbone models, we conduct experiments using both base and instruct models of various scales. In addition, we carry out experiments on leading LRMs to demonstrate that TTRL can improve model performance even after costly post-training. The models we experiment with are as follows: • Qwen Family: Qwen2.5-Math-1.5B ( Yang et al., 2024a ) , Qwen2.5-Math-7B ( Yang et al., 2024a ) , Qwen2.5-7B ( Yang et al., 2024b ) , Qwen2.5-32B ( Yang et al., 2024b ) , Qwen3-8B (thinking mode & non-thinking mode) ( Yang et al., 2024b ) ;
• LLaMA Family: LLaMA-3.1-8B-Instruct ( Grattafiori et al., 2024 ) , LLaMA-3.2-3B-Instruct ( Grattafiori et al., 2024 ) , LLaMA-3.2-3B-Oat-Zero ( Liu et al., 2025b ) ;
• Mistral Family: Mistral-Nemo-Instruct-2407 ( MistralAI-NeMo, 2024 ) , Ministral-8B-Instruct-2410 ( Ministral-8B-Instruct, 2024 ) ;
• DeepSeek Family: DeepSeek-Math-7B-Instruct ( Shao et al., 2024 ) , DeepSeek-R1-LLaMA-8B ( Guo et al., 2025 ) ;
• Others: Skywork-OR1-Math-7B ( He et al., 2025 ) ;
#### Benchmarks
We evaluate TTRL on GPQA-Diamond ( Rein et al., 2024 ) , a challenging and high-quality subset of the Graduate-Level Google-Proof Question Answering benchmark, and 3 3 mathematical reasoning benchmarks: AIME 2024 ( Li et al., 2024 ) , AMC ( Li et al., 2024 ) , and MATH-500 ( Hendrycks et al., 2021 ) .
#### Evaluation Setup
We apply TTRL to each benchmark individually and then evaluate. We set the maximum generation length to 3072 3072 tokens, unless otherwise specified. For the main experiments , following DeepSeek-R1 ( Guo et al., 2025 ) , we adopt the pass@ k k evaluation protocol ( Chen et al., 2021 ) and report pass@1 using non-zero temperature sampling. Specifically, we generate 16 16 responses ( 4 4 for 32k context) per question using a temperature of 0.6 0.6 and a top- p p value of 0.95 0.95 . The pass@1 score is computed as: pass@1 = 1 k ∑ i = 1 k p i , \text{pass@1}=\frac{1}{k}\sum_{i=1}^{k}p_{i}, where p i p_{i} indicates whether the i i -th response is correct. For the analysis and additional experiments on Qwen2.5-MATH , we evaluate using greedy decoding to report pass@1, to ensure a fair comparison with previous works. Appendix B presents a set of training-time metrics we used to monitor the performance of TTRL and analyze its training dynamics in the absence of ground-truth labels.
#### Baselines
Since the use of TTT for reasoning has not been previously explored, we primarily compare it with the backbone model to validate whether TTRL can achieve effective improvements through self-evolution. Appendix A presents additional experimental results comparing TTRL with previous state-of-the-art RL approaches for reasoning.
#### Implementation Details
We independently apply GRPO ( Shao et al., 2024 ) on each benchmark to implement TTRL . For hyperparameters, we use a cosine learning rate schedule with a peak value of 5 × 10 − 7 5\times 10^{-7} and adopt the AdamW optimizer for the policy model. For rollout, we sample 64 64 responses using a temperature of 0.6 0.6 ( 1.0 1.0 for Qwen2.5-Math and LRMs) for voting-based label estimation and downsample 32 32 responses per prompt for training. Evidence shows that our vote-then-sample strategy effectively reduces computational costs while still achieving strong performance. The maximum generation length is set to 32,768 32{,}768 tokens for LRMs and 3,072 3{,}072 tokens for all other models. We set the number of episodes to 10 10 , 30 30 , and 80 80 for MATH-500, AMC, and AIME 2024, respectively, based on the dataset size. All experiments were conducted on 8 * NVIDIA A100 80GB GPUs.
### 3.2 Main Results
#### TTRL performs well on most tasks and models.
Table 1 presents the main results. We apply TTRL to 6 6 models spanning 4 4 model families, 2 2 model types, and 3 3 model sizes, consistently demonstrating substantial improvements across 4 4 highly challenging benchmarks. On the demanding mathematical reasoning benchmark AIME 2024, TTRL achieves a minimum improvement of 105 % 105\% across all 6 6 models. Moreover, applying TTRL to a 1.5B model leads to a significant gain of up to 40.3 40.3 points on the MATH-500. Recently, Shao et al. (2025) demonstrated the importance of evaluating different models for RL-based methods to validate experimental conclusions. Therefore, we additionally report results on a broader range of models from various model families, such as DeepSeek-R1-LLaMA-8B, an LRM from DeepSeek trained on the LLaMA model. Table 2 presents the results. As shown, TTRL continues to exhibit consistent effectiveness. Furthermore, as shown in Appendix A , despite relying solely on self-evolution using unlabeled test data, TTRL achieves performance comparable to existing RL-based models that are trained on large-scale labeled datasets.
#### TTRL performs well on LRMs.
With the rapid progress in RL and TTS, LRMs are becoming increasingly central. To further examine whether TTRL remains effective on LRMs that have undergone expensive post-training, especially on highly challenging tasks, we evaluate two other powerful LRMs. Figure 3 presents the results of applying TTRL to additional reasoning models. Qwen3-8B is evaluated in thinking mode. Despite the extensive post-training these models have undergone, TTRL still achieves substantial performance gains, yielding improvements of approximately 10 points on both backbones.
#### TTRL naturally scales.
Another noteworthy observation is that as the model size increases (1.5B → \rightarrow 7B and 7B → \rightarrow 32B), performance consistently improves, highlighting the natural scaling behavior of TTRL : larger models can produce more accurate majority voting rewards during self-improvement, which leads to more effective learning on new data.
#### TTRL generalizes well beyond the target task.
We perform TTRL on each benchmark and further evaluate pass@1 using greedy decoding on others, with Qwen2.5-Math-7B as the backbone. Figure 4 shows the results. Despite the out-of-distribution nature of this setting, TTRL achieves substantial improvements across all benchmarks. This suggests that TTRL does not rely on overfitting, which would lead to trade-offs on other tasks, but instead acquires generalizable gains during self-improvement.
#### TTRL is compatible with different RL algorithms.
We further apply TTRL using two RL algorithms on MATH-500 to assess its compatibility, which are PPO ( Schulman et al., 2017 ) , a value mode based method, and PRIME ( Cui et al., 2025 ) , a process-level RL algorithm. Figure 5 presents the results. The performance trajectories of GRPO, PPO, and PRIME are closely aligned.
#### TTRL achieves sustainable self-evolution through “online” and “RL”.