Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

arxiv:2506.06632

Original source · Paper JSON

Post-training · 2025-06-07

Cites (6) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
• Abstract
• 1 Introduction
• 2 Background and Related Work
• 3 Method
• 3.1 Task decomposition for RL post-training
• 3.2 Training schedulers for LLM reasoning ability
• 3.3 Theoretical Analysis
• 3.3.1 Theoretical Setup
• 3.3.2 Convergence Guarantee
• Theorem 3.1 (CRL Performance Guarantee) .
• 3.3.3 Finite Sample Approximation Error Analyses
• Theorem 3.2 (Sample Complexity) .
• 4 Experiments
• 4.1 Experimental Setup
• 4.2 Baselines
• 4.3 Experimental Results
• 5 Conclusion
• Appendix
• Appendix A Proofs
• A.1 Proof of Theorem 3.1
• Proof.
• A.1.1 Theorem A.1 and Proof
• Theorem A.1 (Finite-Sample Guarantee) .
• Proof.
• A.1.2 Proof of Theorem 3.2
• Proof.
• Appendix B Sample Efficiency Gains with CRL
• Appendix C Dataset Details
• C.1 Datasets with Human Annotated Difficulties
• C.2 Datasets without Human Annotated Difficulties
• Appendix D Model License Information
• Appendix E Implementation Details
• E.1 Hardware Details
• E.2 Training Details
• E.3 Difficulty-Based Training Split Creation for GSM8K and AQuA
• E.4 Inference Details
• Appendix F Comparisons with SFT
• Appendix G Additional Analysis and Ablations
• G.1 Results on More LLMs
• G.2 Results Across Gaussian Parameters
• G.3 Choice of Number of Difficulty Splits
• G.4 DAPO + E2H
• G.5 Qualitative Analysis of Scheduling Techniques
• Appendix H LLM Usage
• Appendix I Limitations
• Appendix J Societal Impacts
• Instructions for reporting errors