Can Large Reasoning Models Self-Train? — part 5/5

Cleaned-text code points [61982, 69102). Bibliography omitted here; source.md includes it.

Based on the insights we gained from the previous sections, the final strategy we experiments with to mitigate model collapse with SRT is dynamic oversampling then filtering the prompts, similar to the one introduced by DAPO ( Yu et al., 2025 ) . Concretely, let the prompt batch size be B B . Then we employ the following routine: (1) Sample B B prompts from the training dataset, (2) Generate K K responses per prompt, calculate per prompt majority votes and the fraction of generations per prompt where the final answer is the same as the majority vote (i.e., self-consistency level), (3) Discard prompts where the majority voting fraction is smaller than a threshold, and repeat the prior steps until we have B B prompts and their corresponding generations, (4) Take one RL step on the generated dataset.

We investigate this idea with a Qwen2.5-Math-7B model trained on the DAPO dataset. We set the filtering threshold to be 0.6 0.6 (so any prompt where the majority vote appears in less than 60% of all generations is discarded), and all other hyperparameters are kept fixed at the default ones from Appendix B.3 . Figure 31 shows our empirical findings: we see that our intuition holds, and this dynamic oversampling and filtering procedure prevents model collapse within our training budget. We hypothesize that extended SRT -training would still lead to model collapse since the reward hacking solution is still an optimal solution under our training objective and it is still possible for the model to discover it during training. Despite this, our procedure of controlling the task distribution is effectively filtering away the harder prompts dynamically during training and employing an online curriculum, which results in delaying model collapse, just as we expect from our experiments using only offline curriculum for SRT -training in Appendix I.3 .

## Appendix J Detailed Experiment Results using non-Qwen models

To validate the efficacy of SRT on LLMs with different pre-training/post-training routine, we run additional experiments on two more models: Deepseek-Math-7B-Instruct ( Shao et al., 2024 ) and Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 ) . Our results are described below.

### J.1 Deepseek-Math-7B-Instruct

We train Deepseek-Math-7B-Instruct on the MATH-12K dataset, and test on AIME 24, AIME 2025, AMC and MATH-500. For training, we use the same hyperparameters use used to train Qwen2.5-Math-7B-Instruct due to lack of compute for running a sweep over possible hyperparameters. We note that we did not find the recommended temperature or other sampling parameters for the Instruct model in ( Shao et al., 2024 ) , but their base models were evaluated with temperature 0.7, so we choose temperature 0.7, top-p 1.0 and no top-k sampling for our evaluations. Figure 32 shows our results: we see similar trends as our earlier experiments, where SRT initially matches performance gain from RL with ground truth, but then leads to performance collapse after prolonged training.

### J.2 Llama-3.1-8B-Instruct

Llama-3.1-8B-Instruct showed no gains while being trained on DAPO or MATH-12K. This can be due to insufficient hyperparameter tuning or the model’s starting performance on these datasets not suitable for learning. So we chose the Big-Math dataset ( Albalak et al., 2025 ) , a dataset with over 250,000 math questions with verifiable answers. Moreover, this dataset has been constructed by filtering common evaluation datasets like MATH-500, making it suitable for our purposes. The primary benefit of using this dataset is that it comes with Llama-3.1-8B-Instruct pass rate , so we can easily ascertain the difficulty of each datapoint and aggregate a subset that can be suitable for training the Llama model. Specifically, we take the subset of Big-Math where Llama-3.1-8B-Instruct has pass rate between 0.3 and 0.7, to filter away too easy or too difficult questions. Next, we train the model on this subset using the same hyperparameters as we used for training the Qwen2.5-Math-7B-Instruct, except we lower the learning rate to 10 − 7 10^{-7} (from 10 − 6 10^{-6} ), as we see that leads to more stable learning curves. For evaluation, we use the same prompt template but temperature 0 (greedy decoding), to match the starting model’s performance reported on its model card, and report pass@1 accuracy.

Figure 33 shows our results: on MATH-500, Llama-3.1-8B-Instruct shows the same performance growth when trained via SRT or RL with ground truth, up to our training budget of 2000 steps. Performance growth is also significant, and training improves pass@1 accuracy from 52.6% to around 60%. Note that we do note report performance on the harder datasets like AIME, because the Llama model’s performance remain close to 0 throughtout training (with both objectives) on these datasets, signalling that they might be too hard for this model. We also ran an additional experiment using a higher learning rate of 3 × 10 − 7 3\times 10^{-7} . Figure 34 shows our empirical findings: with the higher learning rate, Llama-3.1-8B-Instruct also start to show model collapse within our training budget.

## Appendix K Example Tasks from Reasoning Gym

Above of we see one example from each of the three Reasoning Gym tasks used in our work. The examples shown are of the lowest difficulties that we first train the model on using ground truth RL. We do self training on more difficult variants on the tasks. Difficulty can be changed by the modifying the either number of person or digits. In this work, we abstract it away by calling it "level".

## Appendix L Behavior of a Model after Performance Collapse

We provide example generations from a Qwen2.5-Math-7B model before and after prolonged training with SRT on DAPO in Table 3 . After SRT training for 1200 steps, the model exhibits performance collapse and generates \boxed{1} together with some other incoherent set of tokens for all prompts. This also shows that the model has learned to reward hack the SRT objective, since outputting the same answer for all queries will invariably lead to maximum consistency regardless of the accuracy of the answer.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .