Can Large Reasoning Models Self-Train? — part 3/5
Cleaned-text code points [30775, 46755). Bibliography omitted here; source.md includes it.
Several concurrent works look into the data efficiency of the RLVR pipeline. Notably, ( Wang et al., 2025 ) shows that by just training on one example, the model can achieve performance equivalent to training on 1.2k examples from the DeepScaleR dataset ( Luo et al., 2025 ) . Similarly, TTRL ( Zuo et al., 2025 ) , also proposes a label-free online RLVR paradigm in a test-time setting, similar to SRT . Our work also explores this paradigm in Section H.1 as an extension of SRT.
## 6 Limitations and Conclusion
In this work, we examine a simple strategy of leveraging an LLM’s self-consistency to train it via an RL framework. Our experiments on synthetic reasoning tasks with controllable difficulty levels demonstrate the promise of such a framework: not only can LLMs improve their performance on these tasks, but they can also improve the quality of their self-supervision for the next training step. Our analysis, however, also reveals the limitation of leveraging self-consistency as training reward: prolonged training under such a framework can lead to reward hacking and complete model collapse. Future investigations could explore how to develop more robust forms of verification, extend self-verification to other domains like coding, and curriculum learning strategies that expose the model to problems of only the right difficulty. Additionally, leveraging LLM-as-judges or generative verifiers to improve the training signal, or employing additional consistency regularization between the chain-of-thought and the final answer to mitigate reward hacking can be promising future research. We leave these and other promising directions for sustained self-improvement for future work.
## Acknowledgements
This work has greatly benefited from the use of Delta’s advanced computing and data resource supported by the National Science Foundation (OAC 2005572) and the State of Illinois, as part of ACCESS-approved compute grants ( Boerner et al., 2023 ) . We also appreciate the computing resources of Bridges-2 ( Brown et al., 2021 ) at Pittsburgh Supercomputing Center through ACCESS allocation CIS240901 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. Overall, this project used ACCESS grants CIS240901, CIS230366, CIS250216, and CIS250428 for its compute resources. Moreover, FT was partially supported by the U.S. Army Futures Command under Contract No. W519TC-23-C-0030 during the project. Andrea Zanette was partially supported by the National Science Foundation under Grants CCF-2106778 and DMS-2134080.
The authors thank Brandon Pusateri, Jillian Lehosky, and Greg Bauer from ACCESS Support Staff for their incredible help at approving supplements and renewals for ACCESS compute grants throughout this project. Moreover, the work would not have finished so quickly without the help of Brett Bode from NCSA Delta Support Staff, who provided the authors with critical help in properly utilizing the Delta cluster. FT gratefully acknowledges Daman Arora, Yuda Song, Yiding Jiang, Ruiqi Zhang, Zhaoyi Zhou, Guanning Zeng, Yutong He and other members of the Zanette, Russ, Auton, and AIRe lab for feedback and suggestions received on earlier versions of this work.
## Appendix A Definition of Generation-Verification Gap
The single-generation accuracy is defined as: Acc gen ( θ ) = 𝔼 x ∼ 𝒳 , y ∼ π θ ( ⋅ | x ) [ 𝟏 ( y = y ∗ ) ] , \text{Acc}_{\text{gen}}(\theta)=\mathbb{E}_{x\sim\mathcal{X},y\sim\pi_{\theta}(\cdot|x)}[\mathbf{1}(y=y^{*})], where y ∗ y^{*} is the correct solution. A verifier function f f selects one candidate from multiple generations: f ( x , { y ( 1 ) , … , y ( n ) } ) ∈ { y ( 1 ) , … , y ( n ) } . f\left(x,\{y^{(1)},\dots,y^{(n)}\}\right)\in\{y^{(1)},\dots,y^{(n)}\}. We define verification accuracy as: Acc ver ( θ , n ) = 𝔼 x ∼ 𝒳 [ 𝟏 ( f ( x , { y ( 1 ) , … , y ( n ) } ) = y ∗ ) ] . \text{Acc}_{\text{ver}}(\theta,n)=\mathbb{E}_{x\sim\mathcal{X}}\left[\mathbf{1}\left(f\left(x,\{y^{(1)},\dots,y^{(n)}\}\right)=y^{*}\right)\right]. We say that a positive generation-verification gap occurs whenever Acc ver ( θ , n ) > Acc gen ( θ ) . \text{Acc}_{\text{ver}}(\theta,n)>\text{Acc}_{\text{gen}}(\theta). Such a gap indicates the verifier’s greater proficiency in recognizing correct solutions within a set of candidates compared to the generator independently generating correct answers.
### A.1 Generation Verification Gap Through Majority Voting
## Appendix B Details on Implementation and Training
### B.1 RL Algorithms
We use two RL algorithms in this work: RLOO ( Ahmadian et al., 2024 ; Kool et al., 2019 ) and GRPO ( Shao et al., 2024 ) . Recent work ( Oertell et al., 2024 ) has shown that heuristic policy gradient algorithms like GRPO can produce unexpected results by increasing or decreasing reasoning performance even under random rewards, where policy gradient should be zero in expectation, and that RLOO does not have this problem. Since SRT is compatible with both RL algorithms, we experiment with both and observe no noticeable difference in the resulting behavior.
In our implementation ( verl ), we use the following RL objective for both RLOO and GRPO:
𝒥 GRPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\ \{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\Big( w i , t ( θ ) A ^ i , t , \displaystyle w_{i,t}(\theta)\hat{A}_{i,t}, clip ( w i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) ] \displaystyle\text{clip}(w_{i,t}(\theta),\ 1-\varepsilon,\ 1+\varepsilon)\hat{A}_{i,t}\Big)\Bigg]
where w i , t ( θ ) w_{i,t}(\theta) is the importance ratio, defined as:
w i , t ( θ ) = π θ ( y i , t ∣ x , y i , < t ) π θ old ( y i , t ∣ x , y i , < t ) w_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}\mid x,\ y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}\mid x,\ y_{i,<t})}
Since we operate fully on-policy, i.e., one RL step per one batch of generated rollouts, this is always one in our experiments. The same advantage defined at a sequence level is applied to each token in the sequence, so henceforth we will drop the t t from the notation as well.
The main difference between GRPO and RLOO then stems from their use of different advantage functions. RLOO objective uses the following advantage function:
1 G ∑ i = 1 G [ R ( y ( i ) , x ) − 1 G − 1 ∑ j ≠ k R ( y ( j ) , x ) ] \frac{1}{G}\sum_{i=1}^{G}[R(y_{(i)},x)-\frac{1}{G-1}\sum_{j\neq k}R(y_{(j)},x)]
whereas GRPO uses the following advantage function:
A ^ i = r ( x , y i ) − mean ( { r ( x , y i ) } i = 1 G ) std ( { r ( x , y i ) } i = 1 G ) \hat{A}_{i}=\frac{r(x,y_{i})-\text{mean}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}{\text{std}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}
Here G G is the number of online samples generated. Both RLOO and GRPO create a dynamic baseline for each sample without needing a separate value function, effectively estimating the expected return on-the-fly during training. Not using a separately trained value network (as opposed to PPO) makes the training much simpler for both algorithms.
In our implementation, we did not add the KL penalty to the loss function, but rather to the reward itself while running RLOO, following recent work such as Tang & Munos (2025) . In verl framework, this can be configured using algorithm.use_kl_in_reward=True and actor_rollout_ref.actor.use_kl_loss=False . However, this does not work for GRPO due to advantage normalization by the standard deviation, and so for GRPO we add KL penalty to the loss function directly. To estimate KL penalty, we use the low variance KL estimator proposed by Schulman (2020) :
𝔻 KL ( π θ | | π ref ) ≈ π ref ( y | x ) π θ ( y | x ) − 1 − log ( π ref ( y | x ) π θ ( y | x ) ) \mathbb{D}_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})\approx\frac{\pi_{\text{ref}}(y|x)}{\pi_{\theta}(y|x)}-1-\log\left(\frac{\pi_{\text{ref}}(y|x)}{\pi_{\theta}(y|x)}\right)
#### Sampling.
For all experiments, we kept the generation temperature to 1.0, top_k to -1, and top_p to 1 for rollouts generated during RL rollouts. Decoding temperature used for validation varies in different settings, see Appendix B.3 and D for more discussion. We cut off maximum prompt length at 1024 and maximum response length to 3072 (note: Qwen2.5-Math-7B models support a maximum context window of 4096).
### B.2 GPU Infrastructure
All experiments in this work were conducted using either a single node consisting of 8 NVIDIA H200 GPUs (141 GB of GPU memory per GPU) or a single node consisting of 4 NVIDIA GH200 GPUs (96 GB of GPU memory per GPU). All experiments can be replicated in single-node training, and we did not, in fact, utilize multinode training. In total, this work consumed ∼ \sim 15000 GPU hours (including preliminary studies and failed runs). All the final results listed in this paper can be replicated within 2000 H200 GPU hours.
### B.3 Details on Training Settings
We choose Family Relationships, Bitwise Arithmetic, and Knights & Knaves ( Xie et al., 2024 ) tasks from Reasoning Gym ( Stojanovski et al., 2025 ) for our experiments. Examples for each task is shown in Appendix K .
For the Bitwise Arithmetic task, Level refers to the difficulty parameter. The model was first trained with level 2 data for 950 steps, reaching 97% accuracy. We then used this initialized model to train with SRT on levels 3 and 4.
For the Family Relationships task, we trained a model to 99% accuracy on the level 4 dataset. Here, Level corresponds to the parameters min_family_size and max_family_size , both set to 4. We then applied SRT on level 5.
For the Knights & Knaves task, we varied only the n_people parameter as the difficulty control. We first trained a model with difficulty level 2 to 99% accuracy, and then used that checkpoint to further experiment with SRT , climbing from level 2 to 3, 3 to 5, 5 to 7 and 7 to 9. We only report levels 7 and 9 in the paper since they are the highest level difficulty among our experiments .
Across all multi-level experiments, we applied SRT progressively. For example, in Bitwise Arithmetic we trained on level 2 with ground truth supervision; then, starting from the level 2 checkpoint, we applied SRT on level 3; finally, we repeated SRT again on level 4 using the checkpoint from level 3. For comparing against SRT with fixed teacher , we use the same starting policy (Qwen3-4B-Base trained with ground-truth on the easiest difficulty level on each task) and generate the same number of rollouts per prompt using temperature 1.0 and perform majority voting among these rollouts to generate our pseudo-labels.
#### Default training hyperparameters for Reasoning Gym tasks.
For all Reasoning Gym experiments, we used the Qwen3-4B-Base model, with GRPO as the main algorithm. The learning rate was set to 1e-6 and the KL penalty to 0.0001 . For all experiments, we used 32 rollouts per prompt for training and 16 rollouts for evaluation.
#### Default training hyperparameters for Math Datasets.
• Qwen2.5-Math-7B : Learning rate 10 − 6 10^{-6} , KL penalty coefficient 0.001 0.001 , decoding temperature for training and evaluation rollouts 1.0 1.0 , top-p 1.0 1.0 , and no top-k sampling.
• Qwen3-14B-Base : Same default hyperparameter setting as Qwen2.5-Math-7B.
• Llama-3.1-8B-Instruct : Learning rate 10 − 7 10^{-7} , KL penalty coefficient 0.001 0.001 , decoding temperature for training is 1.0 1.0 and evaluation is 0.0 0.0 , top-p 1.0 1.0 , and no top-k sampling. We subsample the Big-Math-RL-Verified dataset to only keep prompts that has average Llama-3.1-8B-Instruct pass rate between 0.3 and 0.7, since the model is unable to improve during training otherwise.
• Deepseek-Math-7B-Instruct : Learning rate 10 − 7 10^{-7} , KL penalty coefficient 0.001 0.001 , decoding temperature for training is 1.0 1.0 and evaluation is 0.7 0.7 (following official protocol), top-p 1.0 1.0 , and no top-k sampling.
### B.4 Code and Dataset Release
For the ease of reproducing the results in this paper, we release the code and datasets used in this work. Specifically, we release four datasets (the others being publicly available):
1. Deduplicated version of the DAPO dataset, containing only unique prompts: https://huggingface.co/datasets/ftajwar/deduplicated_dapo_dataset
2. The test datasets for math tasks compiled together: https://huggingface.co/datasets/ftajwar/srt_test_dataset
3. Individual training and evaluation datasets for various levels of the reasoning gym tasks (bitwise arithmetic, knights and knaves, family relationships) can be found here: https://huggingface.co/collections/ftajwar/self-rewarding-llm-training-6835218091832c3664176553 .
4. The easy subset of DAPO, chosen based on per prompt pass rate calculated using 32 responses for each prompt from a Qwen2.5-Math-7B model (at temperature 1.0 and no top_p or top_k sampling): ftajwar/dapo_easy_one_third_sorted_by_pass_rate
5. The easy subset of DAPO, chosen based on per prompt frequency of the majority answer from a Qwen2.5-Math-7B model using the same generation setting as above: https://huggingface.co/datasets/ftajwar/dapo_easy_one_third_sorted_by_pass_rate
Finally, our code is available at: https://github.com/tajwarfahim/srt . For a summary of project related assets, visit our project website at https://self-rewarding-llm-training.github.io/ .
## Appendix C Additional Experimental Results
### C.1 Qwen2.5-Math-7B
Here we compare the performance of training Qwen2.5-Math-7B with SRT and RL with ground truth on each individual test set for the sake of completeness. Figures 10 , 11 , and 12 show the detailed results when we train on DAPO, MATH-12K, and AIME (1983-2023) respectively. Additionally, when training on MATH-12K and DAPO, we also evaluate the intermediate checkpoints on the heldout set MATH-500, which is reported in Figure 11 . Since MATH-500 contains 500 examples, calculating average@32 accuracy becomes expensive, and hence we could not use it as a test set for all our training setups .
### C.2 Qwen3-14B-Base
In addition to Qwen2.5-Math-7B, we apply our algorithm on another LLM — namely Qwen3-14B-Base ( Yang et al., 2025 ) . We choose the base model since it has not gone through additional post-training on reasoning tasks, unlike the Qwen3-14B model. Additionally, this is a significantly larger model with a different pre-training, making it suitable for testing our algorithm’s effectiveness.
Figures 13 , 14 , and 15 shows our results with DAPO, MATH-12K, and AIME (1983-2023) used as training dataset respectively. Our experiments with Qwen3-14B-Base mostly follows similar patterns as Qwen2.5-Math-7B: SRT maintains stable performance on MATH-12K, mixed results on AIME (1983-2023), and performance collapse on DAPO.
## Appendix D Effect of Decoding Temperature (Qwen2.5-Math-7B)
## Appendix E Additional Self-Training Metrics
We are interested to know if SRT teaches actual reasoning strategies beyond just proper formatting rules necessary for extracting the final answer. To do so, we track two additional metrics throughout SRT -training: fraction of generations among all generations where the final answer is parseable and fraction of generations among those that are parseable where the final answer is correct. Figure 17 summarizes our findings on Reasoning Gym: We see that due to training with RLVR on an easier level of difficulty, the starting policy can already format most generations correctly, and in the case of Knights & Knaves, fraction of correct responses keeps increasing even after fraction of properly formatted (and thus parseable) responses have saturated.
## Appendix F More Details on Baselines
#### Baseline Implementation.