DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
arxiv:2402.03300
Post-training · 2024-02-05
Complete cleaned Markdown · Source Markdown including bibliography
Read in parts
Contents
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models
- Abstract
- 1 Introduction
- 1.1 Contributions
- 1.2 Summary of Evaluations and Metrics
- 2 Math Pre-Training
- 2.1 Data Collection and Decontamination
- 2.2 Validating the Quality of the DeepSeekMath Corpus
- 2.2.1 Training Setting
- 2.2.2 Evaluation Results
- 2.3 Training and Evaluating DeepSeekMath-Base 7B
- Mathematical Problem Solving with Step-by-Step Reasoning
- Mathematical Problem Solving with Tool Use
- Formal Mathematics
- Natural Language Understanding, Reasoning, and Code
- 3 Supervised Fine-Tuning
- 3.1 SFT Data Curation
- 3.2 Training and Evaluating DeepSeekMath-Instruct 7B
- 4 Reinforcement Learning
- 4.1 Group Relative Policy Optimization
- 4.1.1 From PPO to GRPO
- 4.1.2 Outcome Supervision RL with GRPO
- 4.1.3 Process Supervision RL with GRPO
- 4.1.4 Iterative RL with GRPO
- 4.2 Training and Evaluating DeepSeekMath-RL
- 5 Discussion
- 5.1 Lessons Learnt in Pre-Training
- 5.1.1 Code Training Benefits Mathematical Reasoning
- Results
- 5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning
- 5.2 Insights of Reinforcement Learning
- 5.2.1 Towards to a Unified Paradigm
- Observation about Data Source
- Observation about Gradient Coefficient
- 5.2.2 Why RL Works?
- 5.2.3 How to Achieve More Effective RL?
- Data Source
- Algorithms
- Reward Function
- 6 Conclusion, Limitation, and Future Work
- Appendix A Appendix
- A.1 Analysis of Reinforcement Learning
- A.1.1 Supervised Fine-tuning
- A.1.2 Rejection Sampling Fine-tuning
- A.1.3 Online Rejection Sampling Fine-tuning
- A.1.4 Direct Preference Optimization (DPO)
- A.1.5 Proximal Policy Optimization (PPO)
- A.1.6 Group Relative Policy Optimization (GRPO)
- Instructions for reporting errors