Can Large Reasoning Models Self-Train?

arxiv:2505.21444

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-05-27

Cites (12) · Cited by (5)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Can Large Reasoning Models Self-Train?
• Abstract
• 1 Introduction
• 2 Preliminaries
• 3 Self-Rewarded Training
• Self-supervision via majority voting.
• Self-Rewarded Training (SRT).
• 4 Experiments and Analysis
• 4.1 Can SRT go beyond the base model’s capabilities?
• Experiments on synthetic reasoning tasks.
• Results.
• 4.2 Can Self-Improvement from SRT be Sustained Indefinitely?
• Multi-level self-improvement on synthetic tasks using curriculum.
• Extended SRT -training on math problems.
• What happens after SRT peaks in performance?
• Additional experiments and ablations.
• 5 Related Works
• Self Improving LLM.
• Online RLVR and Easy to Hard Generalization.
• Model Collapse and Reward Hacking.
• Data Efficient RLVR.
• 6 Limitations and Conclusion
• Acknowledgements
• Appendix A Definition of Generation-Verification Gap
• A.1 Generation Verification Gap Through Majority Voting
• Appendix B Details on Implementation and Training
• B.1 RL Algorithms
• Sampling.
• B.2 GPU Infrastructure
• B.3 Details on Training Settings
• Default training hyperparameters for Reasoning Gym tasks.
• Default training hyperparameters for Math Datasets.
• B.4 Code and Dataset Release
• Appendix C Additional Experimental Results
• C.1 Qwen2.5-Math-7B
• C.2 Qwen3-14B-Base
• Appendix D Effect of Decoding Temperature (Qwen2.5-Math-7B)
• Appendix E Additional Self-Training Metrics
• Appendix F More Details on Baselines
• Baseline Implementation.
• Dataset Curation
• Appendix G Detailed Experiment Results on Different Training Settings
• G.1 GRPO vs RLOO
• G.2 Different KL Penalty Coefficients
• G.3 Learning Rate
• G.4 Different Number of generations per prompt
• G.5 Entropy Coefficient
• Appendix H Test-Time Self-Improvement
• H.1 SRT can be used for test-time training
• H.2 Why Doesn’t the Performance Collapse during Test-Time-Training?
• Appendix I More on Model Collapse Resulting from Self-Training
• I.1 Early Stopping
• I.2 Self-Training with Offline-Generated Labels
• I.3 Self-Training with Curriculum Learning
• I.4 Training Dynamics of SRT (Qwen2.5-Math-7B) on the Easy DAPO Subset
• I.5 Training Qwen3-14B-Base on the Easy DAPO Subset
• I.6 Generating Easy DAPO Subset using Qwen2.5-Math-1.5B
• I.7 SRT training with online curriculum
• Appendix J Detailed Experiment Results using non-Qwen models
• J.1 Deepseek-Math-7B-Instruct
• J.2 Llama-3.1-8B-Instruct
• Appendix K Example Tasks from Reasoning Gym
• Appendix L Behavior of a Model after Performance Collapse
• Instructions for reporting errors