Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards

arxiv:2606.18810

Original source · Paper JSON

Post-training · 2026-06-17

Cites (8) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 On-Policy Distillation
• 2.2 Token-Level Credit Assignment in RL
• 3 Preliminaries
• Notation
• 3.1 GRPO
• 3.2 On-Policy Distillation
• 4 Why Self-Distillation Fails in RLVR
• 4.1 Available Supervision in RLVR
• 4.2 Why OPSD Fails in RLVR
• 5 SC-GRPO
• Overview
• 5.1 Self-Conditioned GRPO
• Teacher construction
• KL weighting
• Group routing
• 5.2 Computational Overhead
• 6 Experiments
• 6.1 Setup
• Tasks
• Metrics
• 6.2 Baselines
• Reinforcement learning
• OPSD with external demonstrations
• 6.3 Main Results
• REINFORCE++ exhibits unstable performance across tasks.
• 7 Analysis
• 7.1 RQ1: Additive Loss
• 7.2 RQ2: Ablation Study
• A1 Group Routing
• A2 Normalization threshold
• A3 Diversity coefficient
• 7.3 RQ3: Out-of-Domain Performance
• 8 Conclusion
• Limitations
• Ethics Statement
• Appendix A Derivation for Direct OPSD in RLVR
• A.1 Setup
• A.2 Forward-KL Direct OPSD
• Notation.
• A.3 Reverse-KL Variant
• Interpretation
• A.4 Summary: Why Direct OPSD Fails in RLVR
• Pointwise minimizers under different divergences.
• Three structural problems.
• Implication.
• Appendix B Experiment Details
• B.1 Training Details
• B.2 OPSD Details
• Demonstration Acquisition
• Training Hyperparameters
• OPSD Prompt
• B.3 OPD Details
• B.4 Analysis of Reinforce++
• Appendix C Training Dynamics
• C.1 Training Dynamics on AIME 24 & 25
• AIME 2024 (Top Row)
• AIME 2025 (Bottom Row)
• Stability and convergence
• C.2 Policy Entropy
• C.3 KL Weighting Function
• C.4 KL Distribution Evolution
• Appendix D Case Study: Token-Level KL Heatmap
• What tokens are selected
• Appendix E Prompt Templates for SC-GRPO
• E.1 Prompt Construction
• E.2 LiveCodeBench Templates Example
• Instructions for reporting errors