Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing

arxiv:2604.02288

Original source · Paper JSON

Post-training · 2026-04-02

Cites (10) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
• Abstract
• 1 Introduction
• 2 Preliminaries
• 2.1 Group Relative Policy Optimization
• 2.2 Self-Distillation Policy Optimization
• 3 The SRPO
• 3.1 Sample-Level Routing
• 3.2 Dynamic-Weighted SDPO
• 3.3 Training Objective
• 4 Experiments
• 4.1 Experimental Setup
• 4.2 Main Results
• 4.3 Ablation Study
• 4.4 Response Length and Compute Time
• 5 Conclusion
• Ethics Statement
• Appendix A Related Work
• A.1 Reinforcement Learning with Verifiable Rewards
• A.2 On-Policy Distillation and Self-Distillation
• Appendix B Experimental Details
• B.1 Technical Setup
• B.2 Hyperparameters
• B.3 Prompt Templates
• B.4 Benchmark Details
• B.5 Teacher Information Construction
• Construction procedure.
• Illustrative example.
• Fallback to GRPO when no teacher information is available.
• Appendix C Routing Statistics Over Training
• Instructions for reporting errors