Can Large Reasoning Models Self-Train? — part 4/5

Cleaned-text code points [46755, 61982). Bibliography omitted here; source.md includes it.

For all three methods (SFT, DPO ( Rafailov et al., 2024 ) , ScPO ( Prasad et al., 2024 ) ), we sweep over three learning rates ( 10 − 5 10^{-5} , 10 − 6 10^{-6} and 10 − 7 10^{-7} ) and pick the checkpoint with the highest validation score. The best checkpoint with highest validation in the SFT stage has been used to initialize the DPO/ScPO training. We also train DPO with the above mentioned learning rates and picked the best score. For DPO and ScPO ( Prasad et al., 2024 ) , we used β = 0.1 \beta=0.1 , which we also found through sweep over (0.1, 0.3 and 0.5). Moreover, we add a negative log-likelihood loss with weight 1.0 1.0 to the DPO and ScPO losses to stabilize them, similar to RPO ( Pang et al., 2024b ) . We do not train for more than 1 epoch to prevent overfitting/unintentional unalignment ( Tajwar et al., 2024 ; Razin et al., 2025 ) and fair comparison with SRT . ScPO was also performed for one round.

#### Dataset Curation

For DPO and ScPO we labeled the most consistent response as the positive example and the least consistent response as the negative example for each question. Moreover, we only kept the instances where w ⁡ ( x ) w(x) , the variance based weighing parameter ( Prasad et al., 2024 ) , was greater than 2.

## Appendix G Detailed Experiment Results on Different Training Settings

### G.1 GRPO vs RLOO

Figure 18 shows our experiment comparing how SRT behaves with different RL algorithms. In particular, we test two algorithms: GRPO and RLOO. While GRPO seems to achieve higher performance, both GRPO and RLOO training with SRT -reward leads to model collapse at similar number of steps — showing that the choice of the RL algo does not influence model collapse.

### G.2 Different KL Penalty Coefficients

The most straightforward way of preventing reward hacking is to add a strong KL penalty to the training objective. In Figure 19 , we explore this idea: to our surprise, we don’t find a higher KL penalty coefficient to delay or prevent model collapse. We attribute this to the reward hacking training signal being too strong to be overcome by the KL regularization.

### G.3 Learning Rate

Another common hyperparameter to tune is the learning rate. To investigate the effect learning rate has on SRT , we finetune a Qwen2.5-Math-7B model with different learning rates using SRT on the DAPO dataset, with all other hyperparameters kept fixed at their default values described in Appendix B.3 . Figure 20 shows our empirical findings: lowering learning rate seems to prevent model collapse within our training budget. However, we notice performance degradation on the harder AIME 2024 and 2025 datasets within our training budget, and hypothesize that prolonged training with SRT , even with a considerably lower learning rate, would still lead to model collapse. We could not study this in detail due to computational constraints, and leave this for future work.

### G.4 Different Number of generations per prompt

We observe the most surprising result among our hyperparameter tuning experiments when we vary the number of rollouts per prompt during training. Similarly as before, we finetune a Qwen2.5-Math-7B model with SRT on the DAPO dataset, with all hyperparameters (except number of rollouts per prompt) kept fixed at the their default values described in Appendix B.3 . Figure 21 records our results: model collapse happens progressively later in the training run as we lower the number of rollouts per prompt. We hypothesize that generating fewer rollouts makes estimating the “true" majority voting label for a single prompt more noisy. This noisy estimation then makes hacking the reward signal harder as well, thereby delaying model collapse. We have not been able to study this phenomenon in more detail due to computational constraints, and leave studying this for future work.

### G.5 Entropy Coefficient

The final hyperparameter with which we experiment is adding an entropy loss to our regular RL objective. The modified RL objective is listed below:

ℒ entropy-augmented ​ ( π θ ) = ℒ RL ​ ( π θ ) − α ​ ℋ ​ ( π θ ) \mathcal{L}_{\text{entropy-augmented}}(\pi_{\theta})=\mathcal{L}_{\text{RL}}(\pi_{\theta})-\alpha\mathcal{H}(\pi_{\theta}) (7)

where ℒ RL ​ ( π θ ) \mathcal{L}_{\text{RL}}(\pi_{\theta}) is the regular RL loss objective, α \alpha is the entropy coefficient, and ℋ ⁡ ( π θ ) \mathcal{H}(\pi_{\theta}) is the per-token entropy averaged across all tokens in all rollouts. A reasonable hypothesis is that adding entropy to the loss objective will prevent model collapse ( Cheng et al., 2025 ) by discouraging the model to converge to one solution for every prompt. However, Figure 22 shows results in the contrary: adding an entropy term to the loss function and increasing the corresponding coefficient α \alpha accelerates model collapse. Upon inspection of the rollouts, we see that increasing α \alpha incentivizes the model to generate more random tokens in the rollouts, which maximizes entropy, followed by the same template final answer, which maximizes training (pseudo-)reward. This suggests that a better mechanism to prevent model diversity collapse is needed ( Song et al., 2025a ; Zhou et al., 2025 ) , which we leave to future work to study.

## Appendix H Test-Time Self-Improvement

### H.1 SRT can be used for test-time training

An appealing application of self-training is improving model accuracy via test-time training ( Sun et al., 2020 ; Wang et al., 2021 ) , a direction also explored by the concurrent work of Zuo et al. (2025) . Test-time training refers to the procedure of further adapting or fine-tuning a pre-trained model on the actual test set itself, typically without access to labels or ground truth annotations. Applying SRT as a test-time training technique is remarkably straightforward: the unlabeled test set is treated precisely as if it were a training dataset, and SRT is directly applied.

We compare the test-time performance of majority voting after SRT test-time training as well as without any test-time training. Empirically, we observe (Fig 23 ) that test-time training via SRT provides relatively limited, yet noticeable, performance gains when measured under the maj@32 metric, compared to the popular majority voting baseline applied directly to outputs generated by the base model.

### H.2 Why Doesn’t the Performance Collapse during Test-Time-Training?

Interestingly, upon completion of test-time training, a visual inspection of model’s outputs reveals that the model’s predictions still degenerate to a single response for nearly every test prompt—precisely the behavior identified as optimal solution to the SRT objective; however, the test-time accuracy remains high.

We conjecture that test-time self-training is inherently more stable due to crucial differences in dataset size. For example, consider the AIME24 test dataset, which contains only 30 samples for self-improvement. With such a limited sample size, the model quickly converges to a stable majority vote answer on these examples by reinforcing the particular chain-of-thought reasoning that leads to such solutions. After reaching this convergence, SRT ceases to receive meaningful gradient signals for further parameter updates, naturally stabilizing test-time performance (see Figure 24 for test-time training dynamics).

In contrast, during regular training on large-scale datasets, the iterative supply of many fresh samples continually pushes the model to optimize heavily for consistency. In such conditions, the model is incentivized to adopt an overly simplistic generalization strategy (producing same \boxed{} answer)—eventually collapsing by producing a uniform, prompt-independent prediction.

## Appendix I More on Model Collapse Resulting from Self-Training

As discussed before, the optimization objective in SRT can lead to significant initial improvements followed by eventual model collapse. Here, we explore complementary strategies to address model collapse and further enhance the performance achievable via self-training: (1) An early stopping strategy leveraging a small labeled validation dataset to detect and prevent model collapse.

(2) An algorithmic strategy that delays model collapse by using pseudo-labels generated from a stable base model rather than from the continuously updated model.

(3) A data-driven curriculum-based strategy to enhance model performance beyond simple early stopping.

### I.1 Early Stopping

In our algorithm, even a small labeled validation dataset can effectively identify the peak performance point during self-training, thereby mitigating model collapse. Figure 25 shows the progression of model performance, measured throughout training on the DAPO dataset and evaluated across several test sets. Crucially, we find that the peak performance consistently occurs around the same training step across different held-out datasets. The vertical line in Figure 25 marks early stopping using only 1% of DAPO as validation, with performance on other datasets remaining near-optimal.

### I.2 Self-Training with Offline-Generated Labels

The tendency toward model collapse arises because SRT prioritizes consistency over correctness, increasingly rewarding model agreement even if incorrect. A straightforward yet effective approach that delays (but not completely prevent) the model collapse is to get the pseudo-labels from a stable, previously fixed checkpoint, rather than leveraging labels from the evolving policy (but with the downside that the generated labels will not benefit from the improvement in majority voting as training progressed, observed in Figure 2 ). Here we generate pseudo-labels via majority voting rollouts from the Qwen2.5-Math-7B base model, store these offline-generated labels, and subsequently perform RL training against them. Figure 26 demonstrates that training with these offline-generated labels significantly stabilizes training while achieving performance comparable to SRT . Importantly, this indicates that the dynamic updating of pseudo-labels during training (online labeling) may not always confer substantial benefits, and instead can contribute to training instability. However, we noticed training collapse even with offline labeled datasets when trained for longer than one epoch for the MATH-12k dataset.

### I.3 Self-Training with Curriculum Learning

Our third approach, curriculum learning, is motivated by the observation that the model experiences earlier collapse when training on the difficult DAPO dataset compared to the simpler MATH-12K dataset. The intuition is that, on a more challenging dataset, the model finds it easier to abandon its pretrained knowledge in favor of optimizing self-consistency rather than genuinely learning to solve the underlying task.

We leverage this hypothesis to implement a curriculum learning strategy ( Bengio et al., 2009 ; Andrychowicz et al., 2017 ; Portelas et al., 2020 ; Florensa et al., 2017 ; Song et al., 2025b ; Lee et al., 2025 ; Tajwar et al., 2025 ) by identifying the ‘easiest’ subset of the DAPO dataset. To be precise, we retain 1/3-rd of the easiest DAPO prompts selected according to two distinct metrics: (1) Pass rate of the base model , which utilizes ground-truth labels.

(2) Frequency of the majority vote , which does not require ground-truth labels.

Figure 27 shows that training on these easier subsets significantly delays the onset of reward hacking, allowing for continuous improvement even across multiple epochs. Remarkably, performance on these curriculum subsets reaches levels comparable to standard RL training with ground-truth labels on the entire DAPO dataset. More importantly, we did not observe training collapse even after 3 epochs of training on 1/3rd of "easy" DAPO dataset. These promising results suggest that curriculum strategies may further extend the benefits of SRT , which we leave as a future research direction.

### I.4 Training Dynamics of SRT (Qwen2.5-Math-7B) on the Easy DAPO Subset

Figure 7 showed the common signs of reward hacking during SRT -training: namely, sudden drop in accuracy on a held-out dataset, sudden increase in KL penalty, etc. However, we found a simple yet effective way of mitigating reward hacking — simply train on the easiest subset of the training data seems to retain the performance improvement obtained by training on the entire dataset, while preventing reward hacking within the same compute budget. Here we attempt to analyze this phenomenon further, from the lense of the same metrics we recorded in Figure 7 .

Figure 28 shows our results on Qwen2.5-Math-7B: SRT -training on the easiest subset does not show the same behavior as training on the full dataset: accuracy on the heldout set does not drop, and KL penalty, while being slightly higher than that of training with ground truth, is still significantly lower than SRT -training on the full dataset. We also see that model entropy does not explode, so the model keeps outputting reasonable responses instead of the degenerate ones resulting from full dataset training. The most intriguing observation is that regarding pseudo-reward (Figure 28 , second from left): it very quickly gets very close to 1 and stabilizes around 0.9. This tends to suggest the model gets very little learning signal as the mean of the pseudo reward is already approximately 1, which is probably the reason it does not learn to reward hack within the same compute budget. We leave investigating this further for future work.

### I.5 Training Qwen3-14B-Base on the Easy DAPO Subset

One of our most interesting observations is that simply using the easiest 1/3-rd of the DAPO dataset eliminates the performance collapse within our training budget (it can still happen if one trains more, though we do not observe it). We want to test whether this is still true for a different base model. To do so, we take the same easiest subset used in Figure 28 (so the subset is determined using either the ground truth pass rate or the frequency of the majority answer of a Qwen2.5-Math-7B model) and train a Qwen3-14B-Base model with SRT on this subset. Figure 29 shows the result of our experiments: similar to Qwen2.5-Math-7B model, the Qwen3-14B-Base model also does not exhibit performance collapse within the same training budget.

### I.6 Generating Easy DAPO Subset using Qwen2.5-Math-1.5B

Next, we want to test if the easy subset generation process of our curriculum algorithm itself is reproducible using different base models. To do so, we generate the easy 1/3-rd subset using a Qwen2.5-Math-1.5B model by both majority voting frequency and pass rate. This is in contrast with our earlier sections, especially Figure 27 and Figure 29 , where we used a Qwen2.5-Math-7B model for the subset generation. We also train the Qwen2.5-Math-1.5B model on the resulting easy subsets. Figure 30 shows our results — we see the same trend as our earlier result, i.e., training on the easy subsets, even for 3 epochs, does not lead to any performance collapse. Moreover, surprisingly, up to our training budget, SRT on the easy subset matches the performance of RL training with ground truth labels.

### I.7 SRT training with online curriculum