Can Large Reasoning Models Self-Train? — part 2/5

Cleaned-text code points [14938, 30775). Bibliography omitted here; source.md includes it.

Complex reasoning tasks like math require domain-specific pre-training/midtraining for online RL to be effective ( Wu et al., 2025a ; Gandhi et al., 2025 ) ; it is also more difficult to control the difficulty of the individual tasks that the model sees during training. Therefore, we first study this question using synthetic reasoning tasks from REASONING GYM ( Stojanovski et al., 2025 ) , a collection of over 100 reasoning environments for reinforcement learning with verifiable rewards. These tasks can also be generated with adjustable difficulty, making it a suitable test-bed for our work. Concretely, we use 3 tasks from Reasoning Gym: (1) Family Relationships , a logic puzzle involving a group of individuals connected via different relationships, and the model has to reason about the relationship between two individuals within this group, (2) Bitwise Arithmetic , a task for testing models’ understanding of Bitwise Arithmetic operations, and (3) Knights & Knaves , a logic puzzle involving characters who always either tell the truth (knights) or always lie (knaves), and the challenge is to deduce who is who based on their statements. Appendix K shows example prompts from each of these tasks. We use a Qwen-3-4B-Base ( Yang et al., 2025 ) model for all our reasoning gym experiments.

Since our goal is to study whether SRT can improve performance on top of a reasonably strong base model, following the setting of Lee et al. (2025) , we first train the base model with GRPO using ground truth labels on the easiest difficulty setting. This is used to teach the base model proper formatting rules and how to solve the basic task before we train using SRT on the next level of difficulty without ground-truth labels. The detailed training settings can be found in Appendix B.3 .

#### Results.

Figure 2 shows our main results: SRT using the current policy as the label generator improves both avg@16 and majority vote@16 accuracy on all 3 reasoning gym tasks. Since we derive our training signal from the majority votes of the current policy evolving with every RL step — this demonstrates that self-improvement using SRT can progressively improve the quality of the pseudo-labels as well. Note that this would not be possible in prior works ( Prasad et al., 2024 ; Huang et al., 2023 ) which distills the majority votes of a fixed teacher policy throughout one round of training. We expect the improvement in the evolving teacher policy to result in further performance gain. To validate this, we compare SRT with a variant of the same algorithm where we use the majority votes of the fixed starting policy instead of the evolving current policy as pseudo-labels. Figure 2 shows their comparison: in Family Relationships and Bitwise Arithmetic, we see larger gains in majority voting performance, and likewise SRT outperforms its fixed teacher variant substantially, by 10% for Bitwise Arighmatics, 8% in Family Relationship, and 6% in Knights and Knaves. On Knights & Knaves, the starting policy already has >90% majority voting accuracy, and we see the difference between the evolving teacher and fixed teacher variants of SRT to be smaller here. Furthermore, these performance gains cannot be explained by learning to format properly: since the model was trained on an easier difficulty level, it can already format its responses correctly in most cases (Appendix E ), and we see performance climbing after saturation in model’s format correctness. Overall, on the synthetic reasoning tasks, SRT clearly pushes the model beyond its starting capabilities , showing the promise of self-improvement even from this basic recipe for self-supervision.

We show a summary of our empirical findings using different base models and training datasets in Figures 3 and 4 (for training curves using more combinations of (train, test) dataset pairs, refer to Appendix C.1 and C.2 ). On all instances, SRT improves both avg@16 and majority vote@16 accuracies on heldout MATH-500 prompts, and performs on par with regular RL training with ground truth verification. More impressively, the observations hold for base models like Llama-3.1-8B-Instruct, which is known to be particularly difficult for RL training on reasoning tasks ( Gandhi et al., 2025 ) , improving its average accuracy from 52.6% to nearly 60%.

We also compare SRT with its offline variants: SFT on the majority vote ( Huang et al., 2023 ) , DPO and ScPO ( Prasad et al., 2024 ) employing contrastive learning between the majority vote and non-majority vote answer in Table 1 . We observe that SRT retains better performance compared to its offline variants distilling the majority vote decisions of the base policy, showing the benefit of self-improvement in the label-generating policy . For more details about the baselines, including choice of hyperparameters and performance on additional test datasets, refer to Appendix F .

### 4.2 Can Self-Improvement from SRT be Sustained Indefinitely?

Given the strong performance of SRT in various reasoning tasks and model architectures, an important question is whether self-training can be maintained over extended iterations. Similar to the prior section, we first test SRT on synthetic tasks with controllable difficulty to rigorously study its properties, and then test the resulting insights on real-world reasoning domains.

#### Multi-level self-improvement on synthetic tasks using curriculum.

Since Reasoning Gym provides a built-in way to control the difficulty of generated tasks, we first investigate whether self-training on an easier set of tasks can produce a model capable of self-improvement on progressively harder levels of difficulty. To do so, we choose 2 Reasoning Gym tasks: Bitwise Arithmetic and Knights & Knaves. Similar to our previous setting, we first train using RL on ground truth labels on the easiest level of difficulty, then progressively train on harder levels without ground truth labels (i.e., SRT on level 5 starts with the checkpoint obtained from SRT on level 4, and so on). For more details, refer to Appendix B.3 .

Figure 5 shows our primary results: in this controlled setting, SRT is able to maintain self-improvement on progressively harder difficulties. In particular, SRT can show reasonable improvement in Bitwise Arithmetic Level 4 after being initialized on Level 2 with ground-truth training, and also progressively climb to near 100% accuracy on Knights & Knave Level 9 after being trained with ground-truth on Level 2 only (intermediate levels are trained with SRT ). In synthetic reasoning tasks where we have a knob for controlling the difficulty of training problems, even the simple curriculum of progressively training on more difficult problems with self-supervision can maintain improvement across multiple difficulty levels.

#### Extended SRT -training on math problems.

Next, we test whether our insights from synthetic tasks also extend to real-world math problems. Specifically, we take take the same 4 base models as in the previous section, and train them on a difficult math dataset through an extended number of iterations using SRT . Figure 6 demonstrates our surprising finding: while SRT initially increases the base model’s performance at a comparable rate with ground-truth RL training, extended training using SRT leads to sudden performance collapse. We observe performance collapse or degradation from extended SRT -training across all models and training datasets, and record these in detail in Appendix C.1 and C.2 . Given that this surprising phenomenon deserves more investigation, we study it in more detail next.

#### What happens after SRT peaks in performance?

To develop a clearer understanding of the underlying reasons for this phenomenon, in this section we investigate this SRT -induced model collapse closely.

We plot the training statistics of the SRT objective (Eqn. 6 ) in Figure 7 . The observed performance collapse closely coincides with a sudden increase in the SRT self-reward objective, implying that the optimization procedure has, in fact, maximally optimized the training objective (self-consistency majority voting), despite a decline in actual output correctness. On the same figure, we further report the token-level average Kullback–Leibler divergence between the model under SRT training and the base model. We observe a sharp increase in KL divergence at the exact point when performance begins to deteriorate, indicating that the generative distribution of the model has substantially diverged from the original model.

These findings strongly suggest the occurrence of reward hacking —the model has learned to produce consistent responses in order to optimize its self-assigned reward, irrespective of their true correctness. Indeed, manual analysis of the model outputs (examples provided in Appnedix L ) confirms this hypothesis: after collapse, the model outputs a very high entropy, essentially random, set of tokens followed by the same “template” final answer (for example, \boxed{1} ) that is nearly independent of the input prompt). In other words, the initially strong correlation between the SRT objective and correctness is ultimately compromised, becoming no longer a reliable proxy signal. This behavior is also related to the well-known simplicity bias in neural networks ( Palma et al., 2019 ; Valle-Pérez et al., 2019 ; Mingard et al., 2020 ; Mingard et al., 2025 ) , as well as the Occam’s razor, where neural networks tend to find the simplest solution that generalizes to the observed signal — in this case this leads to the same final template answer for all prompts.

#### Additional experiments and ablations.

We have run additional experiments and ablations to study the behavior of self-training under different training scenarios. Figure 8 shows our main findings: (1) choice of RL algorithm (GRPO vs RLOO) does not affect the final outcome of SRT -training, (2) increasing KL coefficient to incentivize the model to stay close to the base policy also does not mitigate reward hacking, as the training signal from the reward hacked solution is too strong, (3) decreasing learning rate seems to delay model collapse but not eliminate it, and we hypothesize that prolonged training with lower learning rates would still result in complete model collapse, and (4) surprisingly, reducing the number of generations per prompt injects noise into the training signal, which delays quick model collapse by exploiting the majority voting answer. Additional experiments related to SRT training, including training curves on all test datasets and effect of tuning additional hyperparameters like entropy coefficient, can be found in Appendix C and G . We also investigated various mitigation strategies for SRT -induced model collapse, including early stopping, curriculum learning, and employing a fixed teacher to generate labels – refer to Appendix I for our findings.

## 5 Related Works

#### Self Improving LLM.

Previous works ( Zelikman et al., 2022 ; Wang et al., 2023b ; Huang et al., 2023 ; Madaan et al., 2023 ; Chen et al., 2024 ; Gulcehre et al., 2023 ; Singh et al., 2024 ; Ni et al., 2023 ; Hwang et al., 2024 ; Havrilla et al., 2024 ; Pang et al., 2024a ) have demonstrated the feasibility of LLMs’ self-improvement over their previous iteration by training on data distilled by the previous instances of the model. Most of these approaches usually have data filtering/reranking step in the pipeline, which is often performed by the model itself ( Wu et al., 2025b ) or by training another ( Hosseini et al., 2024 ) verifier model. Particularly, ( Huang et al., 2023 ; Wang et al., 2023a ; Prasad et al., 2024 ) demonstrated the feasibility of using majority voting and self-consistency to filter chain-of-thought traces that, when used as SFT training data, improve the LLM performance on downsteaming tasks. A concurrent work, ( Zhao et al., 2025a ) , proposes a self-evolution, self-play pipeline where an LLM generates coding problems of appropriate difficulty, solves and trains on them using RLVR. The model-generated solutions are validated by a code executor in the loop. Previous works have studied self-improvement by generating labels through majority voting ( Prasad et al., 2024 ; Huang et al., 2023 ) , but these works are typically confined to one or a few rounds of SFT or DPO ( Rafailov et al., 2024 ) . This essentially involves distilling majority voting labels from a fixed policy into the current policy over each round of training. In contrast, we explore online RLVR’s potential in self-improving LLMs where the label generating policy evolves after every gradient step. A few concurrent works ( Chen et al., 2025 ; Prabhudesai et al., 2025 ; Zhao et al., 2025b ; Shao et al., 2025 ) have also explored various forms of self-rewarding mechanisms through majority voting or a similar metric of self-consistency (e.g., token/sequence level entropy) in Online RLVR. Finally, in a recent work, ( Song et al., 2025b ) formalized a generation verification gap as central to the model’s ability to self-improve. Similarly, ( Huang et al., 2025 ) proposed a “sharpening" mechanism as the key to self-improvement. Our SRT pipeline builds on top of both of these intuitions. We refer the interested reader to Gao et al. (2025) for a survey of other works associated with self-evolving LLM agents.

#### Online RLVR and Easy to Hard Generalization.

Online reinforcement learning with verifiable reward (RLVR) ( Lambert et al., 2025 ) has emerged as a new paradigm of LLM post-training especially for enhancing math, coding and reasoning performances ( OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ; Team et al., 2025 ; Lambert et al., 2025 ) . Despite the success of the reasoning models, it is still unclear to what extent they can generalize beyond the difficulty of their training data distribution, a problem termed easy to hard generalization ( Sun et al., 2024 ) . ( Sun et al., 2024 ) shows that models can be trained to solve level 4-5 MATH ( Hendrycks et al., 2021 ) problems after training using a process reward model trained on MATH level 1-3 dataset. Another work ( Lee et al., 2025 ) explores this question and finds that transformers are capable of easy-to-hard generalization by utilizing transcendence phenomenon ( Zhang et al., 2024 ) in the context of simple addition, string copying, and maze solving using small language models.

#### Model Collapse and Reward Hacking.

Model collapse is a well-known phenomenon in training on self-generated training data ( Alemohammad et al., 2024 ; Shumailov et al., 2024b ; Shumailov et al., 2024a ; Bertrand et al., 2024 ; Briesch et al., 2025 ) , and multiple approaches related to data mixing, reliable verification, training using contrastive loss using negative samples and curriculum learning have been proposed ( Gerstgrasser et al., 2024 ; Feng et al., 2025 ; Briesch et al., 2025 ; Song et al., 2025b ; Gillman et al., 2024 ; Setlur et al., 2024 ) to prevent models from collapsing, which previous work on LLM’s easy to hard generalization ( Lee et al., 2025 ) also utilize. However, in RL paradigm, we do not directly do supervised fine-tuning on model-generated data, and it remains an open question to what extent the previous findings of model collapse apply to our RLVR setting. In our work, we show that models trained using RL on self-labeled data often suffer from actor collapsing due to reward hacking ( Amodei et al., 2016 ; Denison et al., 2024 ) and propose a few strategies to mitigate it. Finally, a concurrent and complementary work, Zhang et al. (2025b) , has also demonstrated the failure of various self-reward mechanisms to sustain self-improvement under prolonged training, which further validates the observations in this work.

#### Data Efficient RLVR.