From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning

arxiv:2605.22074

Original source · Paper JSON

Post-training · 2026-05-21

Cites (5) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• 1 Introduction
• 2 Related Work
• Reinforcement Learning with Verifiable Rewards (RLVR)
• Curriculum Learning for Reasoning
• 3 Method
• 3.1 Preliminaries: GRPO
• 3.2 SCRL Framework
• Build subproblems.
• Curriculum prompt.
• Response format.
• 3.2.1 Progress-Aware Subproblem Rewards
• Curriculum progress.
• Progress-aware correction.
• 3.2.2 SCRL Training Algorithm
• Subproblem-level normalization.
• Token-level credit assignment.
• Mixed-group training.
• 4 Theoretical Analysis
• Definition 4.1 (Effective and Lifted Gradient Information Matrices) .
• Theorem 4.2 (Gradient Dead Zone) .
• Theorem 4.3 (Metric Recovery via Subproblem Decomposition) .
• 5 Experiment
• 5.1 Experimental Setup
• Models.
• Training Setup.
• Benchmark.
• Baseline Settings.
• 5.2 Main Results and Further Analysis
• Superior Performance Across All Benchmarks.
• Curriculum progress transfers to hard-problem solving.
• SCRL does not rely on highly curated subproblems.
• Subproblem-level normalization enables better credit assignment.
• 6 Conclusion
• Appendix A More Ablation Study
• Number of subproblems.
• Training data construction.
• Appendix B Proofs for Section 4
• B.1 Proof of Theorem 4.2 (Bound on λ min ​ ( 𝑭 x ​ ( θ ) ) \lambda_{\min}(\bm{F}_{x}(\theta)) )
• B.2 Proof of Theorem 4.3
• First claim.
• Second claim.
• B.3 Bound on C A ^ C_{\hat{A}}
• Appendix C SCRL Training Algorithm
• Appendix D OOD Task Performance
• SCRL generalizes to out-of-distribution tasks.
• Appendix E Detailed Experimental Results
• Appendix F Implementation Details
• F.1 Hyperparameters
• F.2 Low-Variance pass@ k k Estimation
• F.3 Baseline Implementation Details
• SFT.
• GRPO.
• DAPO.
• QuestA.
• NuRL.
• Appendix G Limitations and Future Work
• Appendix H Hardware Setup
• Appendix I Prompt for Subproblem Generation
• Appendix J Chat Template
• J.1 Chat Template of Curriculum Learning
• J.2 Chat Template of Original Problem
• Appendix K Case Study
• Instructions for reporting errors