DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models β€” part 3/5

Cleaned-text code points [30110, 45630). Bibliography omitted here; source.md includes it.

Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) is an actor-critic RL algorithm that is widely used in the RL fine-tuning stage of LLMs ( Ouyang et al., 2022 ) . In particular, it optimizes LLMs by maximizing the following surrogate objective: π’₯ P ​ P ​ O ​ ( ΞΈ ) = 𝔼 ⁑ [ q ∼ P ⁑ ( Q ) , o ∼ Ο€ ΞΈ o ​ l ​ d ​ ( O | q ) ] ​ 1 | o | ​ βˆ‘ t = 1 | o | min ⁑ [ Ο€ ΞΈ ​ ( o t | q , o < t ) Ο€ ΞΈ o ​ l ​ d ​ ( o t | q , o < t ) ​ A t , clip ​ ( Ο€ ΞΈ ​ ( o t | q , o < t ) Ο€ ΞΈ o ​ l ​ d ​ ( o t | q , o < t ) , 1 βˆ’ Ξ΅ , 1 + Ξ΅ ) ​ A t ] , \footnotesize\mathcal{J}_{PPO}(\theta)=\mathbb{E}{[q\sim P(Q),o\sim\pi_{\theta_{old}}(O|q)]}\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left[\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{old}}(o_{t}|q,o_{<t})}A_{t},\text{clip}\left(\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{old}}(o_{t}|q,o_{<t})},1-\varepsilon,1+\varepsilon\right)A_{t}\right], (1) where Ο€ ΞΈ \pi_{\theta} and Ο€ ΞΈ o ​ l ​ d \pi_{\theta_{old}} are the current and old policy models, and q , o q,o are questions and outputs sampled from the question dataset and the old policy Ο€ ΞΈ o ​ l ​ d \pi_{\theta_{old}} , respectively. Ξ΅ \varepsilon is a clipping-related hyper-parameter introduced in PPO for stabilizing training. A t A_{t} is the advantage, which is computed by applying Generalized Advantage Estimation (GAE) ( Schulman et al., 2015 ) , based on the rewards { r β‰₯ t } \{r_{\geq t}\} and a learned value function V ψ V_{\psi} . Thus, in PPO, a value function needs to be trained alongside the policy model and to mitigate over-optimization of the reward model, the standard approach is to add a per-token KL penalty from a reference model in the reward at each token ( Ouyang et al., 2022 ) , i.e., r t = r Ο† ​ ( q , o ≀ t ) βˆ’ Ξ² ​ log ⁑ Ο€ ΞΈ ​ ( o t | q , o < t ) Ο€ r ​ e ​ f ​ ( o t | q , o < t ) , r_{t}=r_{\varphi}(q,o_{\leq t})-\beta\log\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{ref}(o_{t}|q,o_{<t})}, (2) where r Ο† r_{\varphi} is the reward model, Ο€ r ​ e ​ f \pi_{ref} is the reference model, which is usually the initial SFT model, and Ξ² \beta is the coefficient of the KL penalty.

As the value function employed in PPO is typically another model of comparable size as the policy model, it brings a substantial memory and computational burden. Additionally, during RL training, the value function is treated as a baseline in the calculation of the advantage for variance reduction. While in the LLM context, usually only the last token is assigned a reward score by the reward model, which may complicate the training of a value function that is accurate at each token. To address this, as shown in Figure 4 , we propose Group Relative Policy Optimization (GRPO), which obviates the need for additional value function approximation as in PPO, and instead uses the average reward of multiple sampled outputs, produced in response to the same question, as the baseline. More specifically, for each question q q , GRPO samples a group of outputs { o 1 , o 2 , β‹― , o G } \{o_{1},o_{2},\cdots,o_{G}\} from the old policy Ο€ ΞΈ o ​ l ​ d \pi_{\theta_{old}} and then optimizes the policy model by maximizing the following objective: π’₯ G ​ R ​ P ​ O ​ ( ΞΈ ) = 𝔼 ⁑ [ q ∼ P ⁑ ( Q ) , { o i } i = 1 G ∼ Ο€ ΞΈ o ​ l ​ d ​ ( O | q ) ] 1 G βˆ‘ i = 1 G 1 | o i | βˆ‘ t = 1 | o i | { min [ Ο€ ΞΈ ​ ( o i , t | q , o i , < t ) Ο€ ΞΈ o ​ l ​ d ​ ( o i , t | q , o i , < t ) A ^ i , t , clip ( Ο€ ΞΈ ​ ( o i , t | q , o i , < t ) Ο€ ΞΈ o ​ l ​ d ​ ( o i , t | q , o i , < t ) , 1 βˆ’ Ξ΅ , 1 + Ξ΅ ) A ^ i , t ] βˆ’ Ξ² 𝔻 K ​ L [ Ο€ ΞΈ | | Ο€ r ​ e ​ f ] } , \footnotesize\begin{split}\mathcal{J}_{GRPO}(\theta)&=\mathbb{E}{[q\sim P(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\ &\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left\{\min\left[\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})}\hat{A}_{i,t},\text{clip}\left(\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})},1-\varepsilon,1+\varepsilon\right)\hat{A}_{i,t}\right]-\beta\mathbb{D}_{KL}\left[\pi_{\theta}||\pi_{ref}\right]\right\},\end{split} (3) where Ξ΅ \varepsilon and Ξ² \beta are hyper-parameters, and A ^ i , t \hat{A}_{i,t} is the advantage calculated based on relative rewards of the outputs inside each group only, which will be detailed in the following subsections. The group relative way that GRPO leverages to calculate the advantages, aligns well with the comparative nature of rewards models, as reward models are typically trained on datasets of comparisons between outputs on the same question. Also note that, instead of adding KL penalty in the reward, GRPO regularizes by directly adding the KL divergence between the trained policy and the reference policy to the loss, avoiding complicating the calculation of A ^ i , t \hat{A}_{i,t} . And different from the KL penalty term used in ( 2 ), we estimate the KL divergence with the following unbiased estimator ( Schulman, 2020 ) : 𝔻 K ​ L [ Ο€ ΞΈ | | Ο€ r ​ e ​ f ] = Ο€ r ​ e ​ f ​ ( o i , t | q , o i , < t ) Ο€ ΞΈ ​ ( o i , t | q , o i , < t ) βˆ’ log Ο€ r ​ e ​ f ​ ( o i , t | q , o i , < t ) Ο€ ΞΈ ​ ( o i , t | q , o i , < t ) βˆ’ 1 , \small\mathbb{D}_{KL}\left[\pi_{\theta}||\pi_{ref}\right]=\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-\log\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-1, (4) which is guaranteed to be positive.

Input initial policy model Ο€ ΞΈ init \pi_{\theta_{\text{init}}} ; reward models r Ο† r_{\varphi} ; task prompts π’Ÿ \mathcal{D} ; hyperparameters Ξ΅ \varepsilon , Ξ² \beta , ΞΌ \mu

Output Ο€ ΞΈ \pi_{\theta}

#### 4.1.2 Outcome Supervision RL with GRPO

Formally, for each question q q , a group of outputs { o 1 , o 2 , β‹― , o G } \{o_{1},o_{2},\cdots,o_{G}\} are sampled from the old policy model Ο€ ΞΈ o ​ l ​ d \pi_{\theta_{old}} . A reward model is then used to score the outputs, yielding G G rewards 𝐫 = { r 1 , r 2 , β‹― , r G } \mathbf{r}=\{r_{1},r_{2},\cdots,r_{G}\} correspondingly. Subsequently, these rewards are normalized by subtracting the group average and dividing by the group standard deviation. Outcome supervision provides the normalized reward at the end of each output o i o_{i} and sets the advantages A ^ i , t \hat{A}_{i,t} of all tokens in the output as the normalized reward, i.e., A ^ i , t = r ~ i = r i βˆ’ mean ⁑ ( 𝐫 ) std ⁑ ( 𝐫 ) \hat{A}_{i,t}=\widetilde{r}_{i}=\frac{r_{i}-{\rm mean}(\mathbf{r})}{{\rm std}(\mathbf{r})} , and then optimizes the policy by maximizing the objective defined in equation ( 3 ).

#### 4.1.3 Process Supervision RL with GRPO

Outcome supervision only provides a reward at the end of each output, which may not be sufficient and efficient to supervise the policy in complex mathematical tasks. Following Wang et al. (2023b) , we also explore process supervision, which provides a reward at the end of each reasoning step. Formally, given the question q q and G G sampled outputs { o 1 , o 2 , β‹― , o G } \{o_{1},o_{2},\cdots,o_{G}\} , a process reward model is used to score each step of the outputs, yielding corresponding rewards: 𝐑 = { { r 1 i ​ n ​ d ​ e ​ x ​ ( 1 ) , β‹― , r 1 i ​ n ​ d ​ e ​ x ​ ( K 1 ) } , β‹― , { r G i ​ n ​ d ​ e ​ x ​ ( 1 ) , β‹― , r G i ​ n ​ d ​ e ​ x ​ ( K G ) } } \mathbf{R}=\{\{r_{1}^{index(1)},\cdots,r_{1}^{index(K_{1})}\},\cdots,\{r_{G}^{index(1)},\cdots,r_{G}^{index(K_{G})}\}\} , where i ​ n ​ d ​ e ​ x ​ ( j ) index(j) is the end token index of the j j -th step, and K i K_{i} is the total number of steps in the i i -th output. We also normalize these rewards with the average and the standard deviation, i.e., r ~ i i ​ n ​ d ​ e ​ x ​ ( j ) = r i i ​ n ​ d ​ e ​ x ​ ( j ) βˆ’ mean ⁑ ( 𝐑 ) std ⁑ ( 𝐑 ) \widetilde{r}_{i}^{index(j)}=\frac{r_{i}^{index(j)}-{\rm mean(\mathbf{R})}}{{\rm std(\mathbf{R})}} . Subsequently, the process supervision calculates the advantage of each token as the sum of the normalized rewards from the following steps, i.e., A ^ i , t = βˆ‘ i ​ n ​ d ​ e ​ x ​ ( j ) β‰₯ t r ~ i i ​ n ​ d ​ e ​ x ​ ( j ) \hat{A}_{i,t}=\sum_{index(j)\geq t}\widetilde{r}_{i}^{index(j)} , and then optimizes the policy by maximizing the objective defined in equation ( 3 ).

#### 4.1.4 Iterative RL with GRPO

As the reinforcement learning training process progresses, the old reward model may not be sufficient to supervise the current policy model. Therefore, we also explore the iterative RL with GRPO. As shown in Algorithm 1 , in iterative GRPO, we generate new training sets for the reward model based on the sampling results from the policy model and continually train the old reward model using a replay mechanism that incorporates 10% of historical data. Then, we set the reference model as the policy model, and continually train the policy model with the new reward model.

### 4.2 Training and Evaluating DeepSeekMath-RL

We conduct RL based on DeepSeekMath-Instruct 7B. The training data of RL are chain-of-thought-format questions related to GSM8K and MATH from the SFT data, which consists of around 144K questions. We exclude other SFT questions to investigate the impact of RL on benchmarks that lack data throughout the RL phase. We construct the training set of reward models following ( Wang et al., 2023b ) . We train our initial reward model based on the DeepSeekMath-Base 7B with a learning rate of 2e-5. For GRPO, we set the learning rate of the policy model as 1e-6. The KL coefficient is 0.04. For each question, we sample 64 64 outputs. The max length is set to 1024, and the training batch size is 1024. The policy model only has a single update following each exploration stage. We evaluate DeepSeekMath-RL 7B on benchmarks following DeepSeekMath-Instruct 7B. For DeepSeekMath-RL 7B, GSM8K and MATH with chain-of-thought reasoning can be regarded as in-domain tasks and all the other benchmarks can be regarded as out-of-domain tasks.

Table 5 demonstrates the performance of open- and closed-source models with both chain-of-thought and tool-integrated reasoning on English and Chinese benchmarks. We find that: 1) DeepSeekMath-RL 7B attains accuracies of 88.2% and 51.7% on GSM8K and MATH, respectively, utilizing chain-of-thought reasoning. This performance surpasses that of all open-source models in the 7B to 70B range, as well as the majority of closed-source models. 2) Crucially, DeepSeekMath-RL 7B is only trained on chain-of-thought-format instruction tuning data of GSM8K and MATH, starting from DeepSeekMath-Instruct 7B. Despite the constrained scope of its training data, it outperforms DeepSeekMath-Instruct 7B across all evaluation metrics, showcasing the effectiveness of reinforcement learning.

## 5 Discussion

In this section, we will share our findings in pre-training and RL experiments.

### 5.1 Lessons Learnt in Pre-Training

We first share our experience in pre-training. Unless otherwise specified, we will adhere to the training settings outlined in Section 2.2.1 . It is worth noting that, when referring to the DeepSeekMath Corpus in this section, we use an 89B-token dataset from the second iteration of the data collection process.

#### 5.1.1 Code Training Benefits Mathematical Reasoning

A popular yet unverified hypothesis suggests that code training improves reasoning. We attempt to offer a partial response to this, particularly within the mathematical domain: code training improves models’ ability to do mathematical reasoning both with and without tool use.

To study how code training affects mathematical reasoning, we experimented with the following two-stage training and one-stage training settings:

Two-Stage Training β€’ Code Training for 400B Tokens β†’ \rightarrow Math Training for 150B Tokens : We train DeepSeek-LLM 1.3B for 400B code tokens followed by 150B math tokens;

β€’ General Training for 400B Tokens β†’ \rightarrow Math Training for 150B Tokens : As a control experiment, we also experiment with general tokens (sampled from a large-scale general corpus created by DeepSeek-AI) instead of code tokens in the first stage of training, in an attempt to investigate the advantages of code tokens over general tokens in improving mathematical reasoning.

One-Stage Training β€’ Math Training for 150B Tokens : We train DeepSeek-LLM 1.3B for 150B math tokens;

β€’ Training on a mixture of 400B Code Tokens and 150B Math Tokens : Math training following code training degrades coding performance. We investigate whether code tokens, when mixed with math tokens for one-stage training, would still improve mathematical reasoning and also alleviate the problem of catastrophic forgetting.

Training Setting Training Tokens w/o Tool Use w/ Tool Use General Code Math GSM8K MATH CMATH GSM8K+Python MATH+Python No Continual Training – – – 2.9% 3.0% 12.3% 2.7% 2.3% Two-Stage Training Stage 1: General Training 400B – – 2.9% 3.2% 14.8% 3.3% 2.3% Stage 2: Math Training – – 150B 19.1% 14.4% 37.2% 14.3% 6.7% Stage 1: Code Training – 400B – 5.9% 3.6% 19.9% 12.4% 10.0% Stage 2: Math Training – – 150B 21.9% 15.3% 39.7% 17.4% 9.4% One-Stage Training Math Training – – 150B 20.5% 13.1% 37.6% 11.4% 6.5% Code & Math Mixed Training – 400B 150B 17.6% 12.1% 36.3% 19.7% 13.5%

Training Setting Training Tokens MMLU BBH HumanEval (Pass@1) MBPP (Pass@1) General Code Math No Continual Training – – – 24.5% 28.1% 12.2% 13.0% Two-Stage Training Stage 1: General Training 400B – – 25.9% 27.7% 15.2% 13.6% Stage 2: Math Training – – 150B 33.1% 32.7% 12.8% 13.2% Stage 1: Code Training – 400B – 25.0% 31.5% 25.0% 40.0% Stage 2: Math Training – – 150B 36.2% 35.3% 12.2% 17.0% One-Stage Training Math Training – – 150B 32.3% 32.5% 11.6% 13.2% Code & Math Mixed Training – 400B 150B 33.5% 35.6% 29.3% 39.4%

##### Results

Table 6 and Table 7 demonstrate the downstream performance under different training settings.

Code training benefits program-aided mathematical reasoning, both under the two-stage training and one-stage training settings. As shown in Table 6 , under the two-stage training setting, code training alone already significantly enhances the ability to solve GSM8K and MATH problems using Python. Math training in the second stage yields further improvements. Interestingly, under the one-stage training setting, mixing code tokens and math tokens effectively mitigates the issue of catastrophic forgetting that arises from two-stage training, and also synergizes coding (Table 7 ) and program-aided mathematical reasoning (Table 6 ).

Code training also improves mathematical reasoning without tool use. Under the two-stage training setting, the initial stage of code training already results in moderate enhancements. It also boosts the efficiency of the subsequent math training, eventually leading to the best performance. However, combining code tokens and math tokens for one-stage training compromises mathematical reasoning without tool use. One conjecture is that DeepSeek-LLM 1.3B, due to its limited scale, lacks the capacity to fully assimilate both code and mathematical data simultaneously.

Model Size ArXiv Corpus English Benchmarks Chinese Benchmarks GSM8K MATH OCW SAT MMLU STEM CMATH Gaokao MathCloze Gaokao MathQA DeepSeek-LLM 1.3B No Math Training 2.9% 3.0% 2.9% 15.6% 19.5% 12.3% 0.8% 17.9% MathPile 2.7% 3.3% 2.2% 12.5% 15.7% 1.2% 0.0% 2.8% ArXiv-RedPajama 3.3% 3.4% 4.0% 9.4% 9.0% 7.4% 0.8% 2.3% DeepSeek-Coder-Base-v1.5 7B No Math Training 29.0% 12.5% 6.6% 40.6% 38.1% 45.9% 5.9% 21.1% MathPile 23.6% 11.5% 7.0% 46.9% 35.8% 37.9% 4.2% 25.6% ArXiv-RedPajama 28.1% 11.1% 7.7% 50.0% 35.2% 42.6% 7.6% 24.8%

#### 5.1.2 ArXiv Papers Seem Ineffective in Improving Mathematical Reasoning