Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe

arxiv:2604.13016

Original source · Paper JSON

Post-training · 2026-04-14

Cites (8) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
• Abstract
• 1 Introduction
• 2 Preliminaries
• 2.1 Notation
• 2.2 On-Policy Distillation
• Sampled-Token OPD.
• Full-Vocabulary OPD.
• Top- k k OPD.
• 2.3 Dynamic Metrics
• Overlap Ratio.
• Overlap-Token Advantage.
• Entropy and Entropy Gap.
• 3 Phenomenology of On-Policy Distillation
• 3.1 Thinking-Pattern Consistency
• Setup.
• Results.
• 3.2 New Knowledge, Not Just Scale
• Setup.
• Results.
• 3.3 Validation via Reverse Distillation
• Setup.
• Results.
• 4 Mechanism of On-Policy Distillation
• 4.1 Progressive Alignment of High-Probability Tokens
• Setup.
• Results.
• 4.2 Optimizing Shared Tokens Alone Suffices
• Results.
• Overlap optimization is self-reinforcing.
• 5 Practical Recipe
• 5.1 Off-Policy Distillation from Teacher Rollouts as Cold Start
• Setup.
• Results.
• 5.2 Leveraging Teacher Post-Training Prompts
• Setup.
• Results.
• 6 Discussion
• 6.1 Reward Quality Degrades with Trajectory Depth
• Response length exhibits a sweet spot.
• Instability originates at later tokens.
• Teacher continuation degrades with prefix depth.
• 6.2 Globally Informative Reward Does Not Guarantee Local Exploitability
• Setup.
• Global reward structure is preserved in both settings.
• A hypothesis on local optimization geometry.
• 6.3 Sampled-Token Reward Is Already Sufficient
• Setup.
• Results.
• 7 Related Work
• Knowledge Distillation.
• On-Policy Distillation.
• Capacity Gap and Distillability.
• 8 Conclusion and Future Work
• Appendix A Details for Section 3
• A.1 GRPO Training Details
• Base Model.
• Training Dataset.
• Training and Evaluation Settings.
• A.2 Experimental Setup
• A.3 Benchmark-wise breakdown of thinking-pattern compatibility
• Appendix B Details for Section 4
• B.1 Additional Analysis of Token Overlap Mass
• B.2 Auxiliary Optimization Dynamics
• Diagnostics.
• Results.
• B.3 Cross-Model Validation of High-Probability-Token Alignment
• Results.
• Appendix C Details for Section 5
• C.1 Cold-Start Distillation Details
• Offline teacher rollout.
• Student SFT.
• C.2 Additional Analysis of Overlap Mass
• C.3 Deduplication Details for the DeepMath Subset
• Question extraction.
• Stage 1: Exact-match deduplication.
• Stage 2: Semantic deduplication.
• Final retained subset.
• C.4 Benchmark-wise breakdown of prompt-template alignment
• Appendix D Details for Section 6
• D.1 Teacher entropy by output position
• Instructions for reporting errors