Reinforcement Learning for Reasoning in Large Language Models with One Training Example

arxiv:2504.20571

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-04-29

Cites (7) · Cited by (14)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Reinforcement Learning for Reasoning in Large Language Models with One Training Example
• Abstract
• 1 Introduction
• 2 Preliminary
• RL Loss Function.
• Data Selection: Historical Variance Score.
• 3 Experiments
• 3.1 Setup
• 3.2 Observation of 1/Few-Shot RLVR
• 3.2.1 Dissection of π 1 \pi_{1} : A Not-So-Difficult Problem
• 3.2.2 Post-saturation Generalization: Generalization After Training Accuracy Saturation
• 3.2.3 1-shot RLVR is Effective for Many Examples & Brings Improvements across Categories
• 3.2.4 More Frequent Self-Reflection on Test Data
• 3.3 1/Few-shot RLVR on Other Models/Algorithms
• 4 Analysis
• 4.1 Ablation Study: Policy Gradient Loss is the Main Contributor, and Entropy Loss Further Improve Post-Saturation Generalization
• 4.2 Entropy-Loss-Only Training & Label Correctness
• 5 Conclusion
• 6 Acknoledgements
• Appendix A Related Work
• Appendix B Experiment Setup
• B.1 Details of Loss Function
• B.2 Training Dataset
• DeepScaleR-sub.
• MATH.
• B.3 Evaulation Dataset
• MATH500.
• AIME 2024/2025.
• AMC 2023.
• Minerva Math.
• OlympiadBench.
• ARC-Challenge/Easy.
• B.4 More Training Details
• B.5 More Evaluation Details
• B.6 Performance Difference on Initial Model
• Appendix C Evaluation Result
• C.1 Main Experiments
• C.1.1 Detailed performance on Qwen2.5-Math-1.5B.
• C.1.2 Detailed Performance on More Models and Training Examples.
• C.1.3 Detailed performance with best per-benchmark results
• C.1.4 Detailed Test curves on MATH500 for 1-shot RLVR on Qwen2.5-Math-1.5B.
• C.1.5 Detailed RLVR results on eacn benchmark over training process.
• C.1.6 More Evaluation on DeepSeek-R1-Distill-Qwen-1.5B
• C.2 Analysis
• C.2.1 Test Curves for Ablation Study
• C.2.2 Entropy loss
• Detailed results of entropy-loss-only training.
• Discussion of entropy loss and its function in 1-shot RLVR.
• C.2.3 (Only) Format Correction?
• (a). Applying format reward in RLVR.
• (b) Observe the change of format in 1-shot RLVR.
• C.2.4 Influence of Random Wrong Labels
• C.2.5 Change the Prompt of π 1 \pi_{1}
• C.3 Response Length
• C.4 Pass@8 Results
• Appendix D Discussions
• D.1 Limitations of Our Work
• D.2 Reasoning Capability of Base Models
• D.3 Why Model Continues Improving After the Training Accuracy Reaches Near 100%?
• D.4 Future Works
• Data Selection and Curation.
• Understanding 1-shot RLVR and Post-saturation Generalization
• Importance of Exploration.
• Other Applications.
• Appendix E Example Details
• Instructions for reporting errors