Reinforcement Learning via Self-Distillation

arxiv:2601.20802

Original source · Paper JSON

Post-training · 2026-01-28

Cites (20) · Cited by (11)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Reinforcement Learning via Self-Distillation
• Abstract
• 1 Introduction
• Summary of evaluation results.
• 2 SDPO: Self-Distillation Policy Optimization
• Proposition 2.1 .
• 2.1 Comparison to RLVR
• 2.2 Compute time & memory
• 2.3 Stability improvements
• 3 Learning without Rich Environment Feedback
• 3.1 Experimental setting
• Baselines.
• 3.2 Results
• 3.3 Self-distillation learns to reason concisely
• 4 Learning with Rich Environment Feedback
• Results.
• 4.1 Self-distillation benefits from stronger models
• 4.2 Self-distillation performs dense credit assignment
• 4.3 The self-teacher improves during training
• 4.4 On-policy self-distillation avoids catastrophic forgetting
• Off-policy self-distillation baseline.
• 4.5 Can GRPO and SDPO be combined?
• 4.6 Which feedback is most informative?
• Sample solutions.
• Environment output.
• Student’s original attempt.
• 5 Solving Hard Questions via Test-Time Self-Distillation
• Definition 5.1 (Discovery time) .
• 5.1 Experimental setting
• 5.2 Results
• Question 3 is only solved by SDPO.
• The initial self-teacher does not solve hard questions.
• 6 Related Work
• 6.1 Reinforcement Learning with LLMs
• 6.2 Learning from Rich Feedback and through Retrospection
• 6.3 Distillation
• 6.4 Self-Distillation
• 7 Conclusion, Limitations, and Future Work
• Limitations.
• Future Work.
• Author Contributions
• Acknowledgments
• Appendix A Implementation of SDPO
• A.1 Gradient Estimators
• Per-token estimator.
• Sequence-level estimator.
• A.2 Regularized teacher
• A.3 Approximate Logit Distillation
• A.4 Off-Policy Training: Generalization to Logit-Level Losses
• Appendix B Theoretical Analysis
• B.1 Proof of Proposition 2.1 .
• Proof.
• B.2 Trust-region Teacher
• Proposition B.1 .
• Proof.
• Appendix C Additional Related Work
• Value networks and Monte Carlo advantage estimation.
• Dense credit assignment with a reward model.
• Partial observability.
• Relation to test-time training.
• C.1 SDPO as Maximum Entropy RL
• Maximum Entropy RL
• SDPO optimizes an implicit reward defined by the teacher
• Appendix D Additional Results & Ablations
• D.1 Learning without rich environment feedback
• D.2 Learning with rich environment feedback
• D.2.1 Additional Results
• D.2.2 Training Stability
• D.2.3 Baselines
• D.3 Test-time self-distillation
• Appendix E Experiment Details
• E.1 Technical setup
• E.2 Hyperparameters
• E.2.1 Details on Hyperparameter Selection ( Section 3 )
• E.3 User Templates
• Appendix F Qualitative Examples
• F.1 Visualization of Advantages
• F.2 Examples
• F.3 Environment Feedback
• F.4 Illustrative Example
• Instructions for reporting errors