DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — part 4/5
Cleaned-text code points [45630, 61403). Bibliography omitted here; source.md includes it.
ArXiv papers are commonly included as a component of math pre-training data ( Lewkowycz et al., 2022a ; Polu and Sutskever, 2020 ; Azerbayev et al., 2023 ; Wang et al., 2023c ) . However, detailed analysis regarding their impact on mathematical reasoning has not been extensively conducted. Perhaps counter-intuitively, according to our experiments, arXiv papers seem ineffective in improving mathematical reasoning. We experiment with models of different sizes, including DeepSeek-LLM 1.3B and DeepSeek-Coder-Base-v1.5 7B ( Guo et al., 2024 ) , using arXiv corpora that underwent varied processing pipelines: • MathPile ( Wang et al., 2023c ) : an 8.9B-token corpus developed with cleaning and filtering heuristic rules, over 85% of which are scientific arXiv papers;
• ArXiv-RedPajama ( Computer, 2023 ) : the entirety of arXiv LaTeX files with preambles, comments, macros, and bibliographies removed, totaling 28.0B tokens.
In our experiments, we separately train DeepSeek-LLM 1.3B for 150B tokens and DeepSeek-Coder-Base-v1.5 7B for 40B tokens on each arXiv corpus. It seems that arXiv papers are ineffective in improving mathematical reasoning. When trained on a arXiv-only corpus, both models display no notable improvements or even deterioration across various mathematical benchmarks of different complexities employed in this study. These benchmarks include quantitative reasoning datasets like GSM8K and MATH (Table 8 ), multiple-choice challenges like MMLU-STEM (Table 8 ), and formal mathematics like miniF2F (Table 9 ).
However, this conclusion has its limitations and should be taken with a grain of salt. We have not yet studied: • The impact of arXiv tokens on specific math-related tasks not included in this research, such as informalization of theorems which is to convert formal statements or proofs to their informal versions;
• The effect of arXiv tokens when combined with other types of data;
• Whether the benefits of arXiv papers would manifest themselves at a larger model scale.
Thus, further exploration is required, which we leave for future studies.
### 5.2 Insights of Reinforcement Learning
#### 5.2.1 Towards to a Unified Paradigm
In this section, we provide a unified paradigm to analyze different training methods, such as SFT, RFT, DPO, PPO, GRPO, and further conduct experiments to explore the factors of the unified paradigm. Generally, the gradient with respect to the parameter θ \theta of a training method can be written as: ∇ θ 𝒥 𝒜 ( θ ) = 𝔼 [ ( q , o ) ∼ 𝒟 ⏟ D a t a S o u r c e ] ( 1 | o | ∑ t = 1 | o | G C 𝒜 ( q , o , t , π r f ) ⏟ G r a d i e n t C o e f f i c i e n t ∇ θ log π θ ( o t | q , o < t ) ) . \nabla_{\theta}\mathcal{J}_{{\color[rgb]{1,0,0}\mathcal{A}}}(\theta)=\mathbb{E}[\underbrace{(q,o)\sim{\color[rgb]{1,0,0}\mathcal{D}}}_{Data\ Source}]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}\underbrace{GC_{{\mathcal{A}}}(q,o,t,{\color[rgb]{1,0,0}\pi_{{rf}}})}_{Gradient\ Coefficient}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (5) There exist three key components: 1) Data Source 𝒟 \mathcal{D} , which determines the training data; 2) Reward Function π r f \pi_{{rf}} , which is the source of the training reward signal; 3) Algorithm 𝒜 \mathcal{A} : which processes the training data and the reward signal to the gradient coefficient G C GC that determines the magnitude of the penalty or reinforcement for the data. We analyze several representative methods based on such a unified paradigm:
• Supervised Fine-tuning (SFT) : SFT fine-tunes pretrained model on human selected SFT data.
• Rejection Sampling Fine-tuning (RFT) : RFT further fine-tunes the SFT model on the filtered outputs sampled from the SFT model based on SFT questions. RFT filters the outputs based on the correctness of their answers.
• Direct Preference Optimization (DPO) : DPO further refines the SFT model by fine-tuning it on augmented outputs sampled from the SFT model, using pair-wise DPO loss.
• Online Rejection Sampling Fine-tuning (Online RFT) : Different from RFT, Online RFT initiates the policy model using the SFT model and refines it by fine-tuning with the augmented outputs sampled from the real-time policy model.
• PPO/GRPO : PPO/GRPO initializes the policy model using the SFT model and reinforces it with the outputs sampled from the real-time policy model.
We summarize the components of these methods in Table 10 . Please refer to Appendix A.1 for a more detailed derivation process.
##### Observation about Data Source
We divide the data source into two categories, online sampling, and offline sampling. Online sampling denotes that the training data is from the exploration results of the real-time training policy model, while offline sampling denotes that the training data is from the sampling results of the initial SFT model. RFT and DPO follow the offline style, while Online RFT and GRPO follow the online style.
As shown in Figure 5 , we find that the Online RFT significantly outperforms RFT on two benchmarks. Specifically, Online RFT is comparable to RFT in the early stage of training but gains an absolute advantage in the later stage, demonstrating the superiority of online training. This is intuitive, as in the initial stage, the actor and the SFT model exhibit close resemblance, with the sampled data revealing only minor differences. In the later stage, however, the data sampled from the actor will exhibit more significant differences, and real-time data sampling will offer greater advantages.
##### Observation about Gradient Coefficient
The algorithm processes the reward signal to the gradient coefficient to update the model parameter. We divide the reward function as ‘Rule’ and ‘Model’ in our experiments. Rule refers to judging the quality of a response based on the correctness of the answer, and Model denotes that we train a reward model to score each response. The training data of the reward model is based on the rule judgment. Equations 10 and 21 highlight a key difference between GRPO and Online RFT: GRPO uniquely adjusts its gradient coefficient based on the reward value provided by the reward model. This allows for differential reinforcement and penalization of responses according to their varying magnitudes. In contrast, Online RFT lacks this feature; it does not penalize incorrect responses and uniformly reinforces all responses with correct answers at the same level of intensity.
As demonstrated in Figure 5 , GRPO surpasses online RFT, thereby highlighting the efficiency of altering positive and negative gradient coefficients. In addition, GRPO+PS shows superior performance compared to GRPO+OS, indicating the benefits of using fine-grained, step-aware gradient coefficients. Furthermore, we explore the iterative RL, in our experiments, we conduct two rounds of iteration. As shown in Figure 6 , we notice that the iterative RL significantly improves the performance, especially at the first iteration.
#### 5.2.2 Why RL Works?
In this paper, we conduct reinforcement learning based on a subset of instruction tuning data, and it achieves significant performance enhancement upon the instruction tuning model. To further explain why reinforcement learning works. We evaluate the Pass@K and Maj@K accuracy of the Instruct and RL models on two benchmarks. As shown in Figure 7 , RL enhances Maj@K’s performance but not Pass@K. These findings indicate that RL enhances the model’s overall performance by rendering the output distribution more robust, in other words, it seems that the improvement is attributed to boosting the correct response from TopK rather than the enhancement of fundamental capabilities. Similarly, ( Wang et al., 2023a ) identified a misalignment problem in reasoning tasks within the SFT model, showing that the reasoning performance of SFT models can be improved through a series of preference alignment strategies ( Yuan et al., 2023b ; Song et al., 2023 ; Wang et al., 2023a ) .
#### 5.2.3 How to Achieve More Effective RL?
We demonstrate RL works pretty well in mathematical reasoning tasks. We also provide a unified paradigm to understand different representative training methods. Within this paradigm, all methods are conceptualized as either direct or simplified RL techniques. As summarized in Equation 5 , there exist three key components: Data Source, Algorithm, and Reward Function. We provide some potential future directions about the three components.
##### Data Source
Data source is the raw material of all training methods. In the context of RL, we specifically refer to the data source as the unlabeled questions with the outputs sampled from the policy model. In this paper, we only use the questions from the instruction tuning stage and a naive nucleus sampling to sample outputs. We think this is a potential reason that our RL pipeline only improves the Maj@K performance. In the future, we will explore our RL pipeline on out-of-distribution question prompts, in conjunction with advanced sampling (decoding) strategies , like those based on tree-search methods ( Yao et al., 2023 ) . Also, the efficient inference techniques ( Xia et al., 2023 ; Leviathan et al., 2023 ; Kwon et al., 2023 ; Xia et al., 2024 ) , which determines the exploration efficiency of policy models, also play an exceedingly important role.
##### Algorithms
Algorithms process the data and reward signal to the gradient coefficient to update the model parameter. Based on Equation 5 , to some extent, all methods now fully TRUST the signal of the reward function to increase or decrease the conditional probability of a certain token. However, it is impossible to ensure the reward signal is always reliable, especially in extremely complex tasks. For example, even the PRM800K datasets ( Lightman et al., 2023 ) , which have been carefully annotated by well-trained annotators, still contain approximately 20% of incorrectly annotations 7 7 7 https://github.com/openai/prm800k/issues/12#issuecomment-1728491852 . To this end, we will explore the reinforcement learning algorithm that is robust against noisy reward signals. We believe such WEAK-TO-STRONG ( Burns et al., 2023 ) alignment methods will bring a fundamental change to the learning algorithms.
##### Reward Function
Reward function is the source of the training signal. In RL, the reward function is usually the neural reward model. We think there exist three important directions for reward models: 1) How to enhance the generalization ability of the reward model. The reward model must be effectively generalized to handle out-of-distribution questions and advanced decoding outputs; otherwise, reinforcement learning may merely stabilize the distribution of LLMs rather than improve their fundamental capabilities; 2) How to reflect the uncertainty of reward model. The uncertainty could potentially act as a linking bridge between the weak reward model and the weak-to-strong learning algorithms; 3) How to efficiently build high-quality process reward models that can provide fine-grained training signals for the reasoning process ( Lightman et al., 2023 ; Wang et al., 2023b ) .
## 6 Conclusion, Limitation, and Future Work
We present DeepSeekMath, which outperforms all open-source models on the competition-level MATH benchmark and approaches the performance of closed models. DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B and undergoes continual training for 500B tokens, with a significant component of the training data being 120B math tokens sourced from Common Crawl. Our extensive ablation study shows web pages offer significant potential for high-quality mathematical data, while arXiv may not as beneficial as we expected. We introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), which can notably improve mathematical reasoning capabilities with less memory consumption. The experiment results show that GRPO is effective even if DeepSeekMath-Instruct 7B has reached a high score on benchmarks. We also provide a unified paradigm to understand a series of methods and summarize several potential directions for more effective reinforcement learning.
Although DeepSeekMath achieves impressive scores on quantitative reasoning benchmarks, its capability on geometry and theorem-proof are relatively weaker than closed models. For instance, in our dry run, the model cannot handle problems related to triangles and ellipses, which may indicate data selection bias in pre-training and fine-tuning. In addition, restricted by the model scale, DeepSeekMath is worse than GPT-4 on few-shot capability. GPT-4 could improve its performance with few-shot inputs, while DeepSeekMath shows similar performance in zero-shot and few-shot evaluation. In the future, we will further improve our engineered data selection pipeline to construct more high-quality pre-trained corpus. In addition, we will explore the potential directions (Section 5.2.3 ) for more effective reinforcement learning of LLMs.
## Appendix A Appendix
### A.1 Analysis of Reinforcement Learning
We provide the detailed derivation of the data source and gradient coefficient (algorithm and reward function) across various methods, including SFT, RFT, Online RFT, DPO, PPO, and GRPO.
#### A.1.1 Supervised Fine-tuning
The objective of Supervised Fine-tuning is maximizing the following objective: 𝒥 S F T ( θ ) = 𝔼 [ q , o ∼ P s f t ( Q , O ) ] ( 1 | o | ∑ t = 1 | o | log π θ ( o t | q , o < t ) ) . \mathcal{J}_{SFT}(\theta)=\mathbb{E}[q,o\sim P_{sft}(Q,O)]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (6) The gradient of 𝒥 S F T ( θ ) \mathcal{J}_{SFT}(\theta) is: ∇ θ 𝒥 S F T = 𝔼 [ q , o ∼ P s f t ( Q , O ) ] ( 1 | o | ∑ t = 1 | o | ∇ θ log π θ ( o t | q , o < t ) ) . \nabla_{\theta}\mathcal{J}_{SFT}=\mathbb{E}[q,o\sim P_{sft}(Q,O)]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (7) Data Source: The dataset employed for SFT. Reward Function: This can be regarded as human selection. Gradient Coefficient: always set to 1.
#### A.1.2 Rejection Sampling Fine-tuning
Rejection Sampling Fine-tuning first samples multiple outputs from the supervised fine-tuned LLMs for each question, and then trains LLMs on the sampled outputs with the correct answer. Formally, the objective of RFT is to maximize the following objectives: 𝒥 R F T ( θ ) = 𝔼 [ q ∼ P s f t ( Q ) , o ∼ π s f t ( O | q ) ] ( 1 | o | ∑ t = 1 | o | 𝕀 ( o ) log π θ ( o t | q , o < t ) ) . \mathcal{J}_{RFT}(\theta)=\mathbb{E}[q\sim P_{sft}(Q),o\sim\pi_{sft}(O|q)]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}\mathbb{I}(o)\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (8) The gradient of 𝒥 R F T ( θ ) \mathcal{J}_{RFT}(\theta) is: ∇ θ 𝒥 R F T ( θ ) = 𝔼 [ q ∼ P s f t ( Q ) , o ∼ π s f t ( O | q ) ] ( 1 | o | ∑ t = 1 | o | 𝕀 ( o ) ∇ θ log π θ ( o t | q , o < t ) ) . \nabla_{\theta}\mathcal{J}_{RFT}(\theta)=\mathbb{E}[{q\sim P_{sft}(Q),o\sim\pi_{sft}(O|q)}]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}{\mathbb{I}(o)}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (9) Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function: Rule (whether the answer is correct or not). Gradient Coefficient: G C R F T ( q , o , t ) = 𝕀 ( o ) = { 1 the answer of o is correct 0 the answer of o is incorrect GC_{RFT}(q,o,t)=\mathbb{I}(o)=\left\{\begin{aligned} 1&&{\rm the\ answer\ of\ o\ is\ correct}\\ 0&&{\rm the\ answer\ of\ o\ is\ incorrect}\\ \end{aligned}\right. (10)
#### A.1.3 Online Rejection Sampling Fine-tuning