Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models

arxiv:2601.18734

Original source · Paper JSON

Post-training · 2026-01-26

Cites (9) · Cited by (13)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
• Abstract
• Keywords:
• 1 Introduction
• 2 Background
• 2.1 Knowledge Distillation for Autoregressive Large Language Models
• 2.2 Reinforcement Learning with Verifiable Rewards
• 3 Methods
• 3.1 Learning from Verifiable Reasoning Dataset
• 3.2 On-Policy Self-Distillation
• Motivation: Learning by understanding solutions.
• Teacher and student policies.
• On-policy sampling from the student.
• Training objective: Full-vocabulary logit distillation.
• Per-Token Pointwise Divergence Clipping.
• Alternative objective: Sampled-token distillation through policy gradient.
• OPSD as dense-reward policy gradient and comparison to STaR.
• 4 Experiments
• 4.1 Experimental Setup
• 4.2 Main Results
• 4.3 Ablation Studies & Discussions
• 4.3.1 Effect of Divergence Objective
• 4.3.2 Effect of Generation Styles and per-token KL Clipping
• 4.3.3 Effect of Per-Token Pointwise Clipping
• 4.3.4 Effect of Generation Length
• 4.3.5 Learning Objective Comparison: Full Vocabulary Logits Distillation vs. Sampled-Token Distillation
• 5 Related Work
• LLM Self-Training.
• 6 Conclusion
• 7 Impact Statement
• Appendix A Limitations and Future Directions
• Appendix B Experimental Details
• Appendix C Token Category Definitions
• Style Tokens.
• Math Tokens.
• Appendix D Policy-Gradient Interpretation of OPSD and Comparison to STaR
• D.1 STaR as Sequence-Level Policy-Gradient
• D.2 OPSD as Dense-Reward Policy Gradient
• Comparison.
• Instructions for reporting errors