Recursive Self-Improvement via On-Policy Distillation for Reasoning

arxiv:2609.30652

Original source · Paper JSON

Post-training · 2026-09-25

Cites (5) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Recursive Self-Improvement via On-Policy Distillation for Reasoning
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 Reflection and Verification in Reasoning Post-Training
• 2.2 On-Policy and Privileged Self-Distillation
• 2.3 Self-Refinement and Efficient Reasoning
• 3 Method
• 3.1 Dynamic Co-Evolution
• 3.2 Self-Refined Concise Learning
• 3.3 Joint Optimization
• 4 Experimental Setup
• 4.1 Models, Data, and Metrics
• 4.2 Baselines and Controls
• 4.3 Training Protocol
• 5 Results
• 5.1 Main Experimental Results
• 5.2 Does Recursive Improvement Transfer Beyond Qwen3?
• 5.3 What Happens During Recursive Self-Improvement?
• 5.4 How Do Different Loss Weights Affect Performance?
• 5.5 How Does the Guidance Objective Influence Performance?
• 5.6 Does the Privileged Teacher Need to Co-Evolve?
• 5.7 Which Representation of the Verified Solution Works Better?
• 5.8 How Does the Method Perform under Tight Reasoning Budgets?
• 5.9 How Sensitive Is Performance to Decoding Hyperparameters?
• 6 Conclusion
• Appendix A Does Recursive Improvement Scale to a Larger Model?
• Appendix B Gemma-4-12B-IT Training Details and Additional Results
• Appendix C Complete Qwen3-8B and Qwen3-4B Trajectories
• Appendix D Teacher-Update Schedules
• Appendix E Effect of Privileged-Context Placement
• Evolution of the privileged signal.
• Appendix F Qwen3-1.7B Training Stability
• Appendix G Fixed-Trace Probe Values
• Appendix H EOS-Penalty Ablation
• Objective.
• Results.
• Appendix I Cross-Scale Evaluation of OPSD-TTS
• Appendix J Reproducibility Details
• Appendix K Additional SRCL-Weight Sensitivity
• Appendix L Results on Additional Benchmarks
• Appendix M SRCL Rewrite Filtering and Acceptance
• Filtering pipeline.
• Endpoint verification.
• Appendix N Prompt Templates
• Instructions for reporting errors