# TTRL: Test-Time Reinforcement Learning

###### Abstract

This paper investigates Reinforcement Learning (RL) on data without explicit labels for reasoning tasks in Large Language Models (LLMs). The core challenge of the problem is reward estimation during inference while not having access to ground-truth information. While this setting appears elusive, we find that common practices in Test-Time Scaling (TTS), such as majority voting, yield surprisingly effective rewards suitable for driving RL training. In this work, we introduce Test-Time Reinforcement Learning ( TTRL ), a novel method for training LLMs using RL on unlabeled data. TTRL enables self-evolution of LLMs by utilizing the priors in the pre-trained models. Our experiments demonstrate that TTRL consistently improves performance across a variety of tasks and models. Notably, TTRL boosts the pass@1 performance of Qwen-2.5-Math-7B by approximately 𝟐𝟏𝟏 % \mathbf{211\%} on the AIME 2024 with only unlabeled test data. Furthermore, although TTRL is only supervised by the maj@n metric, TTRL has demonstrated performance to consistently surpass the upper limit of the initial model maj@n, and approach the performance of models trained directly on test data with ground-truth labels. Our experimental findings validate the general effectiveness of TTRL across various tasks and highlight TTRL ’s potential for broader tasks and domains.

## 1 Introduction

Recent advances in Large Reasoning Models (LRMs), such as DeepSeek-R1 ( Guo et al., 2025 ) and OpenAI’s o1 ( Jaech et al., 2024 ) , have demonstrated that Reinforcement Learning (RL) is essential for enhancing long chain-of-thought (CoT) reasoning ( Wei et al., 2022 ) through training on expensive human-annotated data. These models achieve remarkable performance on a range of highly challenging tasks. For example, OpenAI’s o3 attains a 75.7 % 75.7\% success rate on ARC-AGI-1. However, complex and unlabeled questions continuously emerge, posing significant challenges. For instance, o3 solves only 4 % 4\% of problems on the recently released ARC-AGI-2 benchmark (2025) 1 1 1 https://arcprize.org/ . Addressing such tasks typically involves scaling up training with more data and computational resources, and it may still fail to yield strong performance on these tasks. Silver & Sutton (2025) has recently advocated for a transition to the “era of experience,” emphasizing the limitations of existing AI systems that rely heavily on human supervision, as well as the importance of enabling models to self-evolve through experience.

Further building upon the substantial progress of LRMs, it naturally motivates a promising direction in which AI systems autonomously improve via RL on unlabeled data by directly engaging in self-experience and learning, thereby pushing the boundaries of RL and further advancing the frontier of AI capabilities. Such self-evolvement can be broadly categorized into two modes: adaptation to test-time data, which enables models to tackle harder benchmarks such as ARC-AGI-2, and training on external unlabeled data, which unlocks more training data beyond labeled corpora. This work focuses on the adaptation to test-time data, which has been extensively studied under the paradigm of Test-Time Training (TTT) ( Sun et al., 2019 ; Sun et al., 2024 ; Behrouz et al., 2024 ; Akyürek et al., 2024 ) . TTT has received increasing attention recently. These approaches adapt model parameters at test time by exploiting the structure and distributional properties of incoming test data.

Therefore, we aim to fully advance AI evolution by updating models at test time using RL, thereby enhancing their generalization to previously unseen data. However, this introduces a critical challenge: How to obtain rewards for RL at test-time? This also highlights a broader limitation of current RL approaches. Despite their promise, most existing methods still rely heavily on labeled data, which significantly limits their scalability. As real-world tasks continue to increase in both complexity and volume, large-scale annotation for RL becomes increasingly impractical, posing a substantial barrier to the continual improvement of state-of-the-art models.

We introduce Test-Time Reinforcement Learning ( TTRL ), which performs test-time training through RL. TTRL employs repeated sampling strategies in the rollout phase to accurately estimate the label and compute rule-based rewards, thereby enabling RL on unlabeled data. By incorporating effective majority voting rewards, TTRL facilitates efficient and stable RL in the absence of ground truth labels. As previously highlighted, the emergence of more challenging tasks will inevitably lead to larger proportions of unlabeled data. TTRL directly addresses the problem of training models via RL without explicit supervision, investigating a model’s ability to explore and learn in this challenging yet critical setting. Essentially, TTRL enables the model to generate its own experiences, estimate rewards, and improve its performance over time.

In experiments, applying TTRL to Qwen2.5-Math-7B results in an improvement on AIME 2024 of 𝟐𝟏𝟏 % \mathbf{211\%} ( 12.9 12.9 to 40.2 40.2 ), with an average gain of 𝟕𝟔 % \mathbf{76\%} across AIME 2024, AMC, MATH-500, and GPQA. These improvements are achieved through self-evolution without any labeled training data and further generalize to other tasks. TTRL not only enhances performance on pass@1 but also improves TTS through majority voting. Moreover, our preliminary experiments suggest that TTRL is effective across models of different scales and types and that it can be integrated with existing RL algorithms. We also found that TTRL exhibits favorable characteristics such as a high-performance ceiling. These observations highlight its potential to substantially reduce reliance on human annotations, enabling continual learning and scaling RL to large-scale unsupervised training. Below are several key takeaways:

## 2 Test-Time Reinforcement Learning (TTRL)

Unlike traditional RL, where the agent learns from known reward signals, TTRL operates on unlabeled test data. In other words, the model must learn and adapt without access to explicit supervision. Our task is defined as follows:

### 2.1 Methodology

Figure 2 illustrates how our approach, TTRL , tackles this challenge. Given a state represented by the prompt x x , the model acts by producing an output y y sampled from a policy π θ ​ ( y ∣ x ) \pi_{\theta}(y\mid x) parameterized by θ \theta . To construct a reward signal without ground-truth labels, we generate multiple candidate outputs { y 1 , y 2 , … , y N } \{y_{1},y_{2},\ldots,y_{N}\} from the model through repeated sampling. A consensus output y ∗ y^{*} is derived, for instance, by majority voting or another aggregation method, serving as a proxy for the optimal action. The environment then provides a reward r ⁡ ( y , y ∗ ) r(y,y^{*}) based on the alignment between the sampled action y y and the consensus action y ∗ y^{*} . The RL objective is thus to maximize the expected reward: max θ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y , y ∗ ) ] , \displaystyle\max_{\theta}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[r(y,y^{*})], (1) and parameters θ \theta are updated through gradient ascent: θ ← θ + η ∇ θ 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ r ( y , y ∗ ) ] , \displaystyle\theta\leftarrow\theta+\eta\nabla_{\theta}\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}[r(y,y^{*})], (2) where η \eta denotes the learning rate. This approach enables the model to adapt during inference, effectively improving its performance on distribution-shifted inputs without the need for labeled data.

The pseudo-code of the majority voting reward function.

### 2.2 Majority Voting Reward Function

The majority voting reward is determined by first estimating a label through majority voting. This estimated label is then used to calculate rule-based rewards, which serve as the final rewards. Given a question x x , we first input x x into the LLM to generate a set of outputs. An answer extractor then processes these outputs to obtain the corresponding predicted answers, denoted as P = { y ^ i } i = 1 N P=\{\hat{y}_{i}\}_{i=1}^{N} . We first follow Equation 4 over P P to estimate a label, with majority voting as the scoring function s ⁡ ( y , x ) s(y,x) to get y y , the most frequently occurring prediction in P P . The majority-voted prediction y y is then used as the estimated label to compute rule-based rewards ( Guo et al., 2025 ) . The reward function is: R ⁡ ( y ^ i , y ) = { 1 , if ​ y ^ i = y , 0 , otherwise . \displaystyle R(\hat{y}_{i},y)=\begin{cases}1,&\text{if }\hat{y}_{i}=y,\\ 0,&\text{otherwise}.\end{cases} (3) Listing presents the pseudo-code of the reward function.

## 3 Experiments

### 3.1 Experimental Setup

#### Models

To evaluate the generalizability of TTRL across different backbone models, we conduct experiments using both base and instruct models of various scales. In addition, we carry out experiments on leading LRMs to demonstrate that TTRL can improve model performance even after costly post-training. The models we experiment with are as follows: • Qwen Family: Qwen2.5-Math-1.5B ( Yang et al., 2024a ) , Qwen2.5-Math-7B ( Yang et al., 2024a ) , Qwen2.5-7B ( Yang et al., 2024b ) , Qwen2.5-32B ( Yang et al., 2024b ) , Qwen3-8B (thinking mode & non-thinking mode) ( Yang et al., 2024b ) ;

• LLaMA Family: LLaMA-3.1-8B-Instruct ( Grattafiori et al., 2024 ) , LLaMA-3.2-3B-Instruct ( Grattafiori et al., 2024 ) , LLaMA-3.2-3B-Oat-Zero ( Liu et al., 2025b ) ;

• Mistral Family: Mistral-Nemo-Instruct-2407 ( MistralAI-NeMo, 2024 ) , Ministral-8B-Instruct-2410 ( Ministral-8B-Instruct, 2024 ) ;

• DeepSeek Family: DeepSeek-Math-7B-Instruct ( Shao et al., 2024 ) , DeepSeek-R1-LLaMA-8B ( Guo et al., 2025 ) ;

• Others: Skywork-OR1-Math-7B ( He et al., 2025 ) ;

#### Benchmarks

We evaluate TTRL on GPQA-Diamond ( Rein et al., 2024 ) , a challenging and high-quality subset of the Graduate-Level Google-Proof Question Answering benchmark, and 3 3 mathematical reasoning benchmarks: AIME 2024 ( Li et al., 2024 ) , AMC ( Li et al., 2024 ) , and MATH-500 ( Hendrycks et al., 2021 ) .

#### Evaluation Setup

We apply TTRL to each benchmark individually and then evaluate. We set the maximum generation length to 3072 3072 tokens, unless otherwise specified. For the main experiments , following DeepSeek-R1 ( Guo et al., 2025 ) , we adopt the pass@ k k evaluation protocol ( Chen et al., 2021 ) and report pass@1 using non-zero temperature sampling. Specifically, we generate 16 16 responses ( 4 4 for 32k context) per question using a temperature of 0.6 0.6 and a top- p p value of 0.95 0.95 . The pass@1 score is computed as: pass@1 = 1 k ​ ∑ i = 1 k p i , \text{pass@1}=\frac{1}{k}\sum_{i=1}^{k}p_{i}, where p i p_{i} indicates whether the i i -th response is correct. For the analysis and additional experiments on Qwen2.5-MATH , we evaluate using greedy decoding to report pass@1, to ensure a fair comparison with previous works. Appendix B presents a set of training-time metrics we used to monitor the performance of TTRL and analyze its training dynamics in the absence of ground-truth labels.

#### Baselines

Since the use of TTT for reasoning has not been previously explored, we primarily compare it with the backbone model to validate whether TTRL can achieve effective improvements through self-evolution. Appendix A presents additional experimental results comparing TTRL with previous state-of-the-art RL approaches for reasoning.

#### Implementation Details

We independently apply GRPO ( Shao et al., 2024 ) on each benchmark to implement TTRL . For hyperparameters, we use a cosine learning rate schedule with a peak value of 5 × 10 − 7 5\times 10^{-7} and adopt the AdamW optimizer for the policy model. For rollout, we sample 64 64 responses using a temperature of 0.6 0.6 ( 1.0 1.0 for Qwen2.5-Math and LRMs) for voting-based label estimation and downsample 32 32 responses per prompt for training. Evidence shows that our vote-then-sample strategy effectively reduces computational costs while still achieving strong performance. The maximum generation length is set to 32,768 32{,}768 tokens for LRMs and 3,072 3{,}072 tokens for all other models. We set the number of episodes to 10 10 , 30 30 , and 80 80 for MATH-500, AMC, and AIME 2024, respectively, based on the dataset size. All experiments were conducted on 8 * NVIDIA A100 80GB GPUs.

### 3.2 Main Results

#### TTRL performs well on most tasks and models.

Table 1 presents the main results. We apply TTRL to 6 6 models spanning 4 4 model families, 2 2 model types, and 3 3 model sizes, consistently demonstrating substantial improvements across 4 4 highly challenging benchmarks. On the demanding mathematical reasoning benchmark AIME 2024, TTRL achieves a minimum improvement of 105 % 105\% across all 6 6 models. Moreover, applying TTRL to a 1.5B model leads to a significant gain of up to 40.3 40.3 points on the MATH-500. Recently, Shao et al. (2025) demonstrated the importance of evaluating different models for RL-based methods to validate experimental conclusions. Therefore, we additionally report results on a broader range of models from various model families, such as DeepSeek-R1-LLaMA-8B, an LRM from DeepSeek trained on the LLaMA model. Table 2 presents the results. As shown, TTRL continues to exhibit consistent effectiveness. Furthermore, as shown in Appendix A , despite relying solely on self-evolution using unlabeled test data, TTRL achieves performance comparable to existing RL-based models that are trained on large-scale labeled datasets.

#### TTRL performs well on LRMs.

With the rapid progress in RL and TTS, LRMs are becoming increasingly central. To further examine whether TTRL remains effective on LRMs that have undergone expensive post-training, especially on highly challenging tasks, we evaluate two other powerful LRMs. Figure 3 presents the results of applying TTRL to additional reasoning models. Qwen3-8B is evaluated in thinking mode. Despite the extensive post-training these models have undergone, TTRL still achieves substantial performance gains, yielding improvements of approximately 10 points on both backbones.

#### TTRL naturally scales.

Another noteworthy observation is that as the model size increases (1.5B → \rightarrow 7B and 7B → \rightarrow 32B), performance consistently improves, highlighting the natural scaling behavior of TTRL : larger models can produce more accurate majority voting rewards during self-improvement, which leads to more effective learning on new data.

#### TTRL generalizes well beyond the target task.

We perform TTRL on each benchmark and further evaluate pass@1 using greedy decoding on others, with Qwen2.5-Math-7B as the backbone. Figure 4 shows the results. Despite the out-of-distribution nature of this setting, TTRL achieves substantial improvements across all benchmarks. This suggests that TTRL does not rely on overfitting, which would lead to trade-offs on other tasks, but instead acquires generalizable gains during self-improvement.

#### TTRL is compatible with different RL algorithms.

We further apply TTRL using two RL algorithms on MATH-500 to assess its compatibility, which are PPO ( Schulman et al., 2017 ) , a value mode based method, and PRIME ( Cui et al., 2025 ) , a process-level RL algorithm. Figure 5 presents the results. The performance trajectories of GRPO, PPO, and PRIME are closely aligned.

#### TTRL achieves sustainable self-evolution through “online” and “RL”.

To gain a deeper understanding of the underlying mechanisms of TTRL , we conduct an analysis of the model’s training dynamics by tracking the average (pass@1/avg@16) and majority (maj@16) scores throughout the training process. Given that majority voting serves as the basis for generating training signals, examining its performance trajectory is essential for understanding how it functions. Furthermore, we investigate whether TTRL improves pass@1 at the cost of a reduction in maj@16 performance. Figure 6 illustrates the TTRL training dynamics on AMC with Qwen2.5-Math-1.5B as the base model. It is notable that, as training progresses, both metrics demonstrate a consistent upward trend. This indicates that TTRL is not simply approaching the initial model’s majority voting performance. Due to its dynamic nature, TTRL can generate higher-quality supervision signals as its capabilities improve. Moreover, through TTRL ’s use of RL for TTT, by converting voting-based pseudo-labels into reward signals, it enhances the effective supervision quality (e.g., accuracy; see Q2 4.2 ), while decoupling learning from the limitations imposed by maj@n.

## 4 Analysis and Discussions

### 4.1 Q1: How Well Can TTRL Perform?

We analyze the potential performance of TTRL using two upper bounds. The first upper bound is the maj@n of the initial model. The second upper bound is direct training on benchmark data, which assumes access to ground-truth labels and thus leaks label information to the policy model.

TTRL is Supervised by maj@n Yet Surpasses It. Since TTRL utilizes the model’s own majority-voted outputs for RL, this voting-based performance of the initial model can intuitively be regarded as an upper bound of the final performance. This upper bound is also the performance limit of traditional self-training methods ( Huang et al., 2022 ) , which select self-generated CoT through majority voting for supervised fine-tuning (SFT). However, we observe a surprising phenomenon: after training, the model not only matches but also surpasses the expected upper bound, suggesting that it exceeds the performance limit of the original model, which also serves as its initial supervision signal. Figure 6 illustrates this remarkable result, where it can be observed that the final avg@16 score exceeds the initial maj@16 score by more than 20 points. Furthermore, we perform additional evaluations of TTRL on Qwen2.5-Math-7B across various benchmarks, using more samples per question to enable more reliable assessment. Figure 7 shows results. It can be observed that TTRL avg@64 consistently outperforms Qwen2.5-Math-7B maj@64 across all benchmarks, with a considerable margin. Through a self-reinforcing loop, the model “lifts itself up by its own bootstraps” , evolving beyond the anticipated performance ceiling. Moreover, the performance of TTRL further improves when majority voting is applied.

TTRL’s Performance Gains Approach Training on the Benchmark. The motivation of TTRL is to estimate labels using majority voting to obtain more accurate rewards, facilitating effective self-improvement through RL on the data without ground-truth labels. Therefore, a natural upper bound of TTRL is performing RL directly on the test data, denoted as RL (leakage). Although this setting is rarely adopted or studied due to the issue of information leakage, it represents the most efficient way to improve performance on the particular dataset, with efficiency that far exceeds traditional training-evaluation paradigms. We use Qwen2.5-Math-7B to perform both TTRL and RL (leakage) on MATH-500 and conduct evaluations. Figure 8 shows results. Surprisingly, we find that the performance curve of TTRL closely approaches that of RL (leakage). This suggests that:

1. TTRL can achieve a level of self-improvement comparable to that of supervised learning (even in the information leakage scenario) through RL in an unsupervised setting. This indicates its substantial efficiency and performance gains.

2. TTRL provides evidence that even small LLMs can now effectively self-improve on input-only challenging tasks through RL, enabling continual learning. Results on Qwen2.5-Math-1.5B further support this observation: starting from a subpar performance of 32.7 32.7 on MATH-500, the model improved by 123.2 % 123.2\% to reach 73.0 73.0 , demonstrating clear self-improvement through TTRL .

### 4.2 Q2: Why Does TTRL Work?

This section presents a progressive analysis of the factors enabling TTRL to achieve stable and effective RL under unsupervised conditions. Our analysis identifies three key factors: label estimation, reward calculation, and online learning.

#### Label Estimations.

A direct difference between TTRL and standard RL algorithms is that TTRL involves label estimation, which introduces reward inaccuracies. We believe that TTRL works despite these inaccuracies due to the following two reasons. (i) Existing studies have shown that RL can tolerate a certain degree of reward inaccuracy. Moreover, RL tends to generalize better than SFT, which often relies on memorizing training data ( Chu et al., 2025 ) . In RL, rewards are typically vague and serve primarily as directional signals for exploration, leading to RL’s robustness to reward noise ( Razin et al., 2025 ) . (ii) Prior work has also examined what constitutes a good reward model from an optimization perspective, revealing that more accurate reward models are not necessarily better teachers ( Wang et al., 2020 ) . Therefore, reward signals estimated by the policy model itself may offer more suitable guidance for learning.

#### Reward Calculations.

When the model is capable of estimating accurate labels via majority voting, the reward and subsequently training are generally reliable. However, a natural question arises: Why does TTRL remain effective even when the model fails to estimate accurate labels via majority voting on challenging benchmarks such as AIME 2024? The most fundamental reason lies in the mechanism by which the verifier computes rewards in RL. For tasks such as mathematics, the verifier works based on “comparison” to obtain rule-based rewards by checking whether the predicted answer matches the given “label.” This mechanism can lead to the phenomenon of “ Lucky Hit ”: for an incorrectly predicted answer, even if the estimated label does not match the ground truth label, as long as it differs from the predicted answer, the verifier will still output a negative reward, and this is exactly the correct reward that we expect, as illustrated in Figure 10 . In other words, it is sufficient that the estimated label differs from the predicted answer for the verifier to assign the correct negative reward. To provide a more detailed case study, we examine the performance of TTRL on the AIME 2024 using Qwen2.5-Math-7B . Figure 9 presents the variation curves of the three metrics, as described in Appendix B . We identify two main reasons why TTRL remains effective on AIME 2024:

1. Reward robustness enabled by multiple outputs within a rollout. First, rewards are denser than labels, allowing for more opportunities to recover useful reward signals even when the estimated label is inaccurate. For example, even when the predicted label is incorrect, alternative outputs within the same rollout can still yield correct or high-quality rewards, as shown in Figure 10 , whereas a rollout containing only a single output would not provide such flexibility. This makes the overall reward signal more robust to errors in pseudo-label estimation.

2. High reward accuracy due to scattered incorrect predictions. Second, counterintuitively, when the model has weaker capability, the majority voting rewards of TTRL may be more accurate. As shown in Figure 9 , although the initial label estimation through majority voting achieves an accuracy of only 37 % 37\% , the reward accuracy reaches an impressive 92 % 92\% . By examining the model outputs, we find that this is because the model’s responses are highly scattered and consistently incorrect, as shown in Figure 10 . A result consistent with this observation is that, for the base model, the most frequently predicted answer accounts for only 16.6 % 16.6\% of all predictions, indicating that the outputs are highly scattered. Therefore, even when the labels are not accurately estimated, due to “ Lucky Hit ”, most outputs can still receive correct rewards. Moreover, the poorer the model’s performance, the more mistakes it tends to make, which paradoxically leads to more accurate reward estimation. An empirical observation supporting this view is the comparison between the label accuracy and reward accuracy, as shown in Figure 9 . Although the label accuracy rarely exceeds 50 % 50\% , the reward accuracy remains consistently high, staying above 75 % 75\% . This high reward accuracy provides a reliable foundation for effective self-improvement on test data.

#### Online Learning.

TTRL is designed based on an online RL approach, whereas traditional self-training and test-time training methods operate in an offline manner. The online nature of TTRL enables the model to improve its capabilities during the application, which in turn leads to more accurate labels generated through voting. As a result, the quality of the supervision signal improves, allowing for truly sustainable self-evolution. As shown in Figure 6 , this dynamic learning process leads to a complementary improvement of performance in both pass@1 and maj@n.

### 4.3 Q3: When Might TTRL Fail?

At the algorithmic level, TTRL is not fundamentally different from existing RL algorithms and therefore inherits several of their characteristics, such as sensitivity to data difficulty, strong reliance on priors, and risk of collapse under certain conditions. At the implementation level, these issues are further amplified by the constraints of TTRL , which estimates labels via majority voting and operates exclusively on test data that is both sparse and previously unseen, potentially resulting in failures in certain scenarios. In our preliminary experiments, we identified two potential issues:

#### Inappropriate RL Hyperparameters.

Hyperparameter settings play a crucial role in RL training, varying across projects 2 2 2 https://github.com/TsinghuaC3I/Awesome-RL-Reasoning-Recipes and often leading to training failures. The influence of hyperparameters is further amplified in TTRL due to potential noise in reward estimation and the characteristics of the test data. Figure 11 presents a comparison of several unsuccessful attempts on AIME 2024. Both of these failed attempts exhibit persistently high entropy that does not diminish throughout training, consistent with findings of prior work ( He et al., 2025 ) . In our preliminary experiments, we identified two key hyperparameters that can critically affect training stability and success:

• Temperature: Setting the temperature to 1.0 1.0 , as opposed to 0.6 0.6 , increases the model’s output entropy. This promotes more extensive exploration and allows the model to make better use of its prior knowledge for self-improvement, which is particularly important when addressing challenging benchmarks.

• Episodes: Given the substantial variation in size and difficulty across datasets, smaller and more difficult datasets need more episodes to achieve sufficient exploration.

#### Lack of Prior Knowledge on Target Task.

Prior knowledge plays a crucial role in RL, often determining the success or failure of the TTRL learning process 3 3 3 https://ysymyth.github.io/The-Second-Half/ . This is mainly because the test data generally exhibits higher difficulty and introduces new features, but TTRL does not incorporate mechanisms such as data filtering to support curriculum learning.

Therefore, for the same backbone, TTRL fails if the model’s prior knowledge is insufficient to handle the complexity of the data. To further validate this hypothesis, we conduct an ablation study on MATH-500. We divide MATH-500 into five subsets according to its annotated difficulty levels, ranging from 1 1 to 5 5 , and apply TTRL to each subset independently, using Qwen2.5-Math-1.5B . We then compare the results to those of the backbone, as shown in Table 3 . We observe that as the question difficulty increases, both the performance improvement and length reduction ratios tend to decrease. This suggests that the available prior knowledge of the backbone is insufficient to support learning on more challenging questions.

## 5 Related Works

### 5.1 Test-Time Scaling

Test-Time Scaling (TTS) is designed to enhance the capabilities of Large Language Models (LLMs) in handling complex tasks by increasing computational resources at test time. Prior research ( Snell et al., 2024 ; Liu et al., 2025a ) indicates that TTS is more efficient than scaling during pre-training ( Kaplan et al., 2020 ) . Therefore, reallocating the same computational resources from pre-training to test-time could yield greater improvements in model performance. Current studies on TTS fall into two categories ( Welleck et al., 2024 ) : parallel generation and sequential generation. Parallel generation involves LLMs producing multiple candidate responses (self-consistency ( Wang et al., 2022 ; Chen et al., 2023 ) , best-of-N ( Stiennon et al., 2020 ; Nakano et al., 2021 ) ), decision steps (Monte Carlo Tree Search ( Zhou et al., 2023 ; Xie et al., 2024 ) ), or tokens (Reward-guided Search ( Deng & Raffel, 2023 ; Khanov et al., 2024 ) ) during inference. Subsequently, an aggregation strategy is applied to integrate these candidates, commonly using process reward models ( Lightman et al., 2023 ; Wang et al., 2023 ; Zhang et al., 2025a ) . Concurrently, sequential generation focuses on extending the LLMs’ output to include longer responses with reflective and chain-of-thought (CoT) processes ( Wei et al., 2022 ; Madaan et al., 2023 ) . Although prompting techniques are widely adopted, they are often constrained by the capabilities of the underlying models. Notably, DeepSeek-R1 ( Guo et al., 2025 ) is a representative advancement in this area, achieving extended reasoning capabilities in pre-trained language models through outcome-based reinforcement learning (RL), more specifically group relative policy optimization ( Shao et al., 2024 ) . Compared to the first approach, which requires intensive process-level supervision ( Yuan et al., 2024 ) , the second approach is more scalable due to its reliance on rule-based rewards.

Beyond the aforementioned methods that focus on scaling test-time inference computation, another approach to increasing test-time computing is Test-Time Training (TTT) . We introduce the relationship between these terminologies in Appendix C . While prior work has primarily focused on applications such as video generation and understanding ( Hardt & Sun, 2024 ; Dalal et al., 2025 ) , and to some extent on large language models ( Wang et al., 2025a ; Akyürek et al., 2024 ) , the integration of test-time scaling with reinforcement learning remains largely underexplored.

### 5.2 RL for Reasoning

Reinforcement Learning (RL) ( Sutton et al., 1998 ) plays a critical role in enhancing the instruction-following capabilities of Large Language Models (LLMs), particularly through approaches like Reinforcement Learning from Human Feedback (RLHF) ( Ouyang et al., 2022 ) . RLHF aligns base models with human preferences using algorithms such as Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , where preference modeling is essential. Recently, Large Reasoning Models (LRMs), such as DeepSeek-R1 ( Guo et al., 2025 ) , have demonstrated the significance of RL in improving reasoning abilities using rule-based rewards, as exemplified by GRPO ( Shao et al., 2024 ) . Unlike RLHF, which is tailored to open-domain instructions, GRPO is specifically designed to elicit long CoT ( Wei et al., 2022 ) reasoning in mathematical problem-solving. Recent studies have focused primarily on improving the training stability of rule-based RL methods like GRPO and PPO ( Cui et al., 2025 ; Yu et al., 2025 ; Liu et al., 2025b ) . However, these methods typically train LLMs only on supervised training data, while inference involves generating extended CoT reasoning on unseen test problems. Moreover, current RL approaches ( Hu et al., 2025a ; Wei et al., 2025 ) depend on verifiable outputs—such as solutions in mathematics or code—that can provide reliable reward signals.

Previous studies have explored self-rewarding ( Yuan et al., 2025 ; Prasad et al., 2024 ) and self-play training ( Chen et al., 2024 ) for unlabeled data. However, these works primarily focus on open-domain instruction following ( Yuan et al., 2025 ; Chen et al., 2024 ) rather than mathematical reasoning or employ preference-based optimization strategies ( Prasad et al., 2024 ) such as DPO ( Rafailov et al., 2023 ) instead of online reinforcement learning algorithms. In addition to these studies, we identified several concurrent works ( Xu et al., 2025 ; Zhang et al., 2025b ; Zhao et al., 2025 ) , that explore self-supervised and semi-supervised reasoning using reinforcement-like methods. The key distinction lies in reward estimation: we employ majority voting, which is derived from the model itself and mitigates reward hacking. Recently, Wang et al. (2025b) demonstrated that using a single training example to incentivize the mathematical reasoning capabilities of LLMs is effective, showing substantial improvements even under minimal supervision. We acknowledge that future research integrating the insights and strengths of these approaches could lead to more robust reasoning models in the era of experience ( Silver & Sutton, 2025 ) . TTRL offers a preliminary attempt at RL with self-labeled rewards, advancing toward learning from streams of experience.

## 6 Conclusion

In this paper, we propose Test-Time Reinforcement Learning ( TTRL ), a novel framework for training large language models with Reinforcement Learning (RL) on test data without access to ground-truth labels. A key component of TTRL is its majority voting reward function, which generates rule-based rewards based on consensus among model predictions. Our experiments demonstrate the strong potential of TTRL , achieving consistent improvements across a variety of models and tasks. We view TTRL as a preliminary step toward RL with self-labeled rewards, marking an important direction of learning from continuous streams of experience.

## 7 Limitations and Future Works

#### Limitations

This work represents an initial exploration of test-time reinforcement learning using self-labeled rewards. While our experimental results are promising, several aspects require further investigation. In particular, we plan to conduct a more in-depth analysis of the impact of prior knowledge and hyperparameter configurations, both of which play critical roles in reinforcement learning dynamics. We will provide comprehensive discussions and ablation studies in future revisions of this paper.

#### Future Works

Building on our findings, we identify several directions for future research:

• Theoretical Analysis: Developing a formal convergence analysis of TTRL , particularly focusing on its ability to optimize toward the two upper bounds in § \lx@sectionsign 4.1 .

• Online Learning with Streaming Data: Extending TTRL to real-time learning scenarios, where models interact with continuously arriving data and adapt dynamically, that is Test-Time Adaptation ( Liang et al., 2025 ) .

• Large-Scale Self-Supervised RL Training: Scaling up TTRL to massive datasets and models to explore its potential in self-supervised regimes without human-labeled data.

• Agentic Tasks and Scientific Discovery: Applying TTRL to more complex, open-ended domains such as agentic tasks and multi-step scientific reasoning.

## Appendix A Additional Results

Table 4 shows pass@1 results using greedy decoding. For the two base models, we further include comparisons with their instruct versions that have undergone large-scale post-training. In addition, we include for reference current leading “R1-Zero-Like” models with similar backbones, which are extensively trained using RL: DeepSeek-R1-Distill-1.5B&7B ( Guo et al., 2025 ) , SimpleRL-Zero-7B ( Zeng et al., 2025 ) , PRIME-Zero-7B ( Cui et al., 2025 ) , OpenReasoner-Zero-7B ( Hu et al., 2025b ) , Oat-Zero-1.5B&7B ( Liu et al., 2025b ) , and LIMR ( Li et al., 2025 ) . Note that TTRL has a different setup from the previous models, which makes the comparison seem unfair.

On the highly challenging mathematical reasoning benchmark AIME 2024, TTRL achieves a substantial improvement of 159.3 % 159.3\% , surpassing all models trained on large-scale datasets. Furthermore, when applied to Qwen2.5-Math-7B , TTRL yields an average improvement of 84.1 % 84.1\% across three benchmarks. Figure 12 shows two curves of TTRL on AIME 2024 with Qwen2.5-Math-7B as an example.

## Appendix B Training Metrics

Given the absence of ground-truth labels in the test data, evaluating the performance of TTRL throughout the training process presents a challenge. To mitigate this limitation, we introduce a set of training-time metrics specifically designed to monitor and assess the effectiveness of TTRL . These metrics inform the selection of the optimal checkpoint and provide valuable insights regarding training dynamics.

• Entropy : Measures the uncertainty of the model’s generation.

• Majority Voting Reward : Rule-based rewards computed from the majority-voted label.

• Majority Ratio : The frequency of the most common answer within a rollout.

Furthermore, we define several metrics that rely on access to ground-truth labels, which allow for a deeper analysis of the model’s behavior during training:

• Label Accuracy (maj@n) : Indicates whether the estimated label matches ground-truth.

• Reward Accuracy : Indicates the proportion of majority voting rewards (computed from the estimated label) that match rewards computed from the ground-truth label.

• Ground-Truth Ratio : The frequency of the ground-truth answer within a rollout.

## Appendix C Terminology

Test-time scaling refers to increasing computational resources during test time, which can be categorized into test-time training and test-time inference. These two approaches are complementary. We will provide an introduction below.

### C.1 Test-Time Training (TTT)

Test-Time Training (TTT) is a technique for adapting a pre-trained model at inference time to improve generalization under distribution shifts. Let f θ f_{\theta} denote a model trained on a source domain 𝒟 ​ s = { ( x i , y i ) } ​ i = 1 N \mathcal{D}s=\{(x_{i},y_{i})\}{i=1}^{N} , where x i ∈ 𝒳 x_{i}\in\mathcal{X} , y i ∈ 𝒴 y_{i}\in\mathcal{Y} , and θ \theta represents the learned parameters. During standard inference, the model is evaluated on test samples x t ∼ 𝒟 t x_{t}\sim\mathcal{D}_{t} with fixed parameters θ \theta , where 𝒟 t ≠ 𝒟 s \mathcal{D}_{t}\neq\mathcal{D}_{s} .

In contrast, TTT allows the model to adapt to each test sample x t x_{t} by minimizing an auxiliary self-supervised loss ℒ aux \mathcal{L}_{\text{aux}} , without access to labels y t y_{t} . The model parameters are updated online with the auxiliary task, which is typically designed to be label-free and consistent with the main task.

### C.2 Test-Time Inference (TTI)

Test-Time Inference (TTI) refers to the strategy of enhancing the performance of a large language model during inference by allocating additional computational resources. Formally, let f θ f_{\theta} denote a language model with parameters θ \theta , and let x x be an input prompt. The model generates an output y y by sampling from the conditional distribution p θ ​ ( y ∣ x ) p_{\theta}(y\mid x) . TTI techniques aim to improve the quality of y y by employing methods such as generating multiple candidate outputs and selecting the best one based on a scoring function, or by refining the output through iterative processes ( Welleck et al., 2024 ) .

One common approach involves generating N N candidate outputs { y 1 , y 2 , … , y N } \{y_{1},y_{2},\ldots,y_{N}\} and selecting the optimal output y ∗ y^{*} using a scoring function s ⁡ ( y , x ) s(y,x) :

y ∗ = arg ⁡ max y i ⁡ s ⁡ ( y i , x ) \displaystyle y^{*}=\arg\max_{y_{i}}s(y_{i},x) (4)

The scoring function s ⁡ ( y , x ) s(y,x) can be instantiated in various ways, such as: 1. Majority Voting (MV): Selecting the most frequent output among the candidates.

2. Best-of-N (BoN): Using reward models to score each candidate, then selecting the highest-scoring one.

3. Weighted BoN: Integrating MV and BoN strategies to leverage their respective strengths.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .