Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

arxiv:2504.13837

Original source · Paper JSON

Post-training · 2025-04-18

Cites (6) · Cited by (21)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• 1 Introduction
• 2 Preliminaries
• 2.1 Reinforcement Learning with Verifiable Rewards
• 2.2 Metrics for LLM Reasoning Capacity Boundary
• 3 RLVR’s Effect on Reasoning Capacity Boundary
• 3.1 RLVR for Mathematical Reasoning
• 3.2 RLVR for Code Generation
• 3.3 RLVR for Visual Reasoning
• 4 Deep Analysis
• 4.1 Reasoning Paths Already Present in Base Models
• 4.2 Distillation Expands the Reasoning Boundary
• 4.3 Effects of Different RL Algorithms
• 4.4 Effects of RL Training
• 4.5 Effects of Entropy
• 4.6 Effects of Model Size Scaling
• 5 Discussion
• 6 Related Work
• 7 Conclusion and Limitations
• Author Contributions
• Acknowledgements
• Appendix
• Appendix Contents
• Appendix A Implementation Details
• A.1 RLVR Algorithms
• A.2 Low-Variance pass@ k k Estimation
• Appendix B More Related Works
• Appendix C Detailed Experimental Results
• C.1 More Results on Mathematics and Coding
• C.2 Validity of Chain-of-Thought on AIME24
• C.3 Accuracy Distribution Visulization
• C.4 Perplexity Analysis
• C.5 Different RLVR Algorithms
• C.6 Effects of KL and Rollout Number
• C.7 Solvable Problem Coverage Analysis
• C.8 Temperature and Entropy Analysis
• C.9 Training Dynamics
• C.10 CoT Case Analysis
• Appendix D Prompt Templates
• Appendix E Broader Impacts
• Instructions for reporting errors