Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

arxiv:2603.24472

Original source · Paper JSON

Post-training · 2026-03-25

Cites (4) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
• Abstract
• 1 Introduction
• 2 Preliminaries
• Self-Distillation
• Key Characteristics of Math Reasoning
• 3 LLM Reasoning Behavior Under Richer Information
• Prompts
• Epistemic Tokens
• Results
• 4 Supervised Finetuning with Self-Distillation
• 5 On-Policy Self-Distillation
• 5.1 DeepSeek-R1-Distill-Qwen-7B
• Training Performance
• OOD Evaluation - AIME24, AMC23
• Reasoning Pattern
• 5.2 Qwen3-8B (Thinking Mode: ON)
• Training Performance
• OOD Evaluation - AIME24, AMC23
• Reasoning Pattern
• 5.3 Qwen3-8B (Thinking Mode: OFF)
• 5.4 Causal Study: Epistemic Token Suppression during GRPO
• 5.5 Ablation Study: Fixed vs. Moving Target Teacher
• 6 Relationship Between Task Coverage, Epistemic Verbalization and Generalization Ability
• 6.1 Comparison of Task Coverage
• 6.2 Relationship Between Task Coverage and Learning Performance
• Training Logs
• OOD Evaluation - AIME24, MATH500
• 7 Conclusion
• Acknowledgments
• Ethics Statement
• Reproducibility Statement
• Appendix A Related Works
• Self-Distillation for LLM Reasoning.
• Uncertainty Expression in LLM Reasoning.
• Reasoning Compression.
• Appendix B Basis for Selecting Epistemic Tokens
• Appendix C Experimental Details
• Training
• Evaluation
• Chat Templates for Different Model Series
• Appendix D More Ablation Study
• D.1 Effect of Maximum Response Length
• D.2 Top- k k Distillation and Learning Rate
• Appendix E Additional Analysis of Epistemic Verbalization
• E.1 Per-Token Analysis of Epistemic Verbalization
• E.2 Comparison of Epistemic Token Usage Across Models
• E.3 Analysis of GRPO and SDPO Reasoning Patterns by Task Coverage
• E.4 Comparative Analysis of Epistemic and Overall Word Usage
• E.5 Uncertainty Annotation via LLM-as-Judge
• Appendix F Student–Teacher KL Divergence on Epistemic Tokens
• Appendix G Analysis of Changes in Model Internal Uncertainty under Self-Distillation
• Appendix H Can Uncertainty-Encouraging Prompts Mitigate the Suppression?
• Appendix I Comparison with OPSD from Zhao et al. (2026)
• Appendix J More On-Policy Self-Distillation Results
• J.1 Qwen3-8B (Thinking Mode: OFF)
• J.2 Qwen3-1.7B (Thinking Mode: ON)
• J.3 Olmo-3-7B-Instruct
• J.4 Pass@16 Score
• Appendix K Qualitative Analysis
• K.1 Unguided vs. Solution-Guided Generation
• K.2 GRPO vs. SDPO Generation
• Problem (AIME24).
• Appendix L Disclosure of LLM Usage
• Instructions for reporting errors