Self-Distillation Enables Continual Learning

arxiv:2601.19897

Original source · Paper JSON

Post-training · 2026-01-27

Cites (7) · Cited by (9)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Self-Distillation Enables Continual Learning
• Abstract
• 1 Introduction
• 2 Related Work
• Off-policy versus On-policy Learning.
• Inverse Reinforcement Learning.
• Context Distillation.
• Self-Distillation.
• 3 Self-Distillation Fine-Tuning
• 3.1 Self-Distillation as Inverse RL
• 3.2 Validating the ICL Assumption
• 4 Experiments
• 4.1 Experimental Setting
• Evaluation.
• Baselines.
• 4.2 On-policy learning leads to better generalization
• Results.
• 4.3 Learning without forgetting
• Results.
• 4.4 Effect of model size
• 4.5 Training reasoning models without reasoning data
• Results.
• 4.6 What drives the improvement in performance?
• Results.
• 5 Discussion and Limitations
• Relationship to on-policy RL.
• Computational Costs.
• Learned Artifacts.
• Requirements for Model Capability.
• Future Work.
• Author Contributions
• Acknowledgment
• Appendix A Additional Ablations
• A.1 Estimating the KL gradient
• Token-level (partial) estimator
• Full analytic per-token estimator.
• Rao-Blackwellized estimator.
• A.2 The Importance of Demonstration-Conditioned Context
• Results.
• A.3 Choice of teacher model
• Appendix B Training and Evaluation details
• B.1 Training Details
• B.2 Evaluation Details
• B.3 Dataset Details
• Science Q&A.
• Tool Use.
• Medical.
• Knowledge Acquisition.
• Instructions for reporting errors