DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models — part 5/5

Cleaned-text code points [61403, 69803). Bibliography omitted here; source.md includes it.

The only difference between RFT and Online RFT is that the outputs of Online RFT are sampled from the real-time policy model π θ \pi_{\theta} , rather than from the SFT model π θ s ​ f ​ t \pi_{\theta_{sft}} . Therefore, the gradient of online RFT is: ∇ θ 𝒥 O ​ n ​ R ​ F ​ T ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o ∼ π θ ​ ( O | q ) ] ​ ( 1 | o | ​ ∑ t = 1 | o | 𝕀 ⁡ ( o ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t | q , o < t ) ) . \nabla_{\theta}\mathcal{J}_{OnRFT}(\theta)=\mathbb{E}[{q\sim P_{sft}(Q),o\sim\pi_{\theta}(O|q)}]\left(\frac{1}{|o|}\sum_{t=1}^{|o|}{\mathbb{I}(o)}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right). (11)

#### A.1.4 Direct Preference Optimization (DPO)

The objective of DPO is: 𝒥 D ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o + , o − ∼ π s ​ f ​ t ​ ( O | q ) ] ​ log ⁡ σ ⁡ ( β ​ 1 | o + | ​ ∑ t = 1 | o + | log ⁡ π θ ​ ( o t + | q , o < t + ) π ref ​ ( o t + | q , o < t + ) − β ​ 1 | o − | ​ ∑ t = 1 | o − | log ⁡ π θ ​ ( o < t − | q , o < t − ) π ref ​ ( o < t − | q , o < t − ) ) \footnotesize\begin{split}\mathcal{J}_{DPO}(\theta)=\mathbb{E}{[q\sim P_{sft}(Q),o^{+},o^{-}\sim\pi_{sft}(O|q)]}\log\sigma\left(\beta\frac{1}{|o^{+}|}\sum_{t=1}^{|o^{+}|}\log\frac{\pi_{\theta}(o^{+}_{t}|q,o^{+}_{<t})}{\pi_{\text{ref}}(o^{+}_{t}|q,o^{+}_{<t})}-\beta\frac{1}{|o^{-}|}\sum_{t=1}^{|o^{-}|}\log\frac{\pi_{\theta}(o^{-}_{<t}|q,o^{-}_{<t})}{\pi_{\text{ref}}(o^{-}_{<t}|q,o^{-}_{<t})}\right)\end{split} (12) The gradient of 𝒥 D ​ P ​ O ​ ( θ ) \mathcal{J}_{DPO}(\theta) is: ∇ θ 𝒥 D ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o + , o − ∼ π s ​ f ​ t ​ ( O | q ) ] ( 1 | o + | ​ ∑ t = 1 | o + | G ​ C D ​ P ​ O ​ ( q , o , t ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t + | q , o < t + ) CLOSE − OPEN 1 | o − | ​ ∑ t = 1 | o − | G ​ C D ​ P ​ O ​ ( q , o , t ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t − | q , o < t − ) ) \footnotesize\begin{split}\nabla_{\theta}\mathcal{J}_{DPO}(\theta)=\mathbb{E}{[q\sim P_{sft}(Q),o^{+},o^{-}\sim\pi_{sft}(O|q)]}&\left(\frac{1}{|o^{+}|}\sum_{t=1}^{|o^{+}|}GC_{DPO}(q,o,t)\nabla_{\theta}\log\pi_{\theta}(o^{+}_{t}|q,o^{+}_{<t})\right.\\ -&\left.\frac{1}{|o^{-}|}\sum_{t=1}^{|o^{-}|}GC_{DPO}(q,o,t)\nabla_{\theta}\log\pi_{\theta}(o^{-}_{t}|q,o^{-}_{<t})\right)\end{split} (13) Data Source: question in SFT dataset with outputs sampled from SFT model. Reward Function: human preference in the general domain (can be ‘Rule’ in mathematical tasks). Gradient Coefficient: G ​ C D ​ P ​ O ​ ( q , o , t ) = σ ⁡ ( β ​ log ⁡ π θ ​ ( o t − | q , o < t − ) π ref ​ ( o t − | q , o < t − ) − β ​ log ⁡ π θ ​ ( o t + | q , o < t + ) π ref ​ ( o t + | q , o < t + ) ) \footnotesize GC_{DPO}(q,o,t)=\sigma\left(\beta\log\frac{\pi_{\theta}(o^{-}_{t}|q,o^{-}_{<t})}{\pi_{\text{ref}}(o^{-}_{t}|q,o^{-}_{<t})}-\beta\log\frac{\pi_{\theta}(o^{+}_{t}|q,o^{+}_{<t})}{\pi_{\text{ref}}(o^{+}_{t}|q,o^{+}_{<t})}\right) (14)

#### A.1.5 Proximal Policy Optimization (PPO)

The objective of PPO is:

𝒥 P ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o ∼ π θ o ​ l ​ d ​ ( O | q ) ] ​ 1 | o | ​ ∑ t = 1 | o | min ⁡ [ π θ ​ ( o t | q , o < t ) π θ o ​ l ​ d ​ ( o t | q , o < t ) ​ A t , clip ​ ( π θ ​ ( o t | q , o < t ) π θ o ​ l ​ d ​ ( o t | q , o < t ) , 1 − ε , 1 + ε ) ​ A t ] . \footnotesize\mathcal{J}_{PPO}(\theta)=\mathbb{E}{[q\sim P_{sft}(Q),o\sim\pi_{\theta_{old}}(O|q)]}\frac{1}{|o|}\sum_{t=1}^{|o|}\min\left[\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{old}}(o_{t}|q,o_{<t})}A_{t},\text{clip}\left(\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{old}}(o_{t}|q,o_{<t})},1-\varepsilon,1+\varepsilon\right)A_{t}\right]. (15) To simplify the analysis, it is assumed that the model only has a single update following each exploration stage, thereby ensuring that π θ o ​ l ​ d = π θ \pi_{\theta_{old}}=\pi_{\theta} . In this case, we can remove the min \min and clip {\rm clip} operation: 𝒥 P ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o ∼ π θ o ​ l ​ d ​ ( O | q ) ] ​ 1 | o | ​ ∑ t = 1 | o | π θ ​ ( o t | q , o < t ) π θ o ​ l ​ d ​ ( o t | q , o < t ) ​ A t . \footnotesize\mathcal{J}_{PPO}(\theta)=\mathbb{E}{[q\sim P_{sft}(Q),o\sim\pi_{\theta_{old}}(O|q)]}\frac{1}{|o|}\sum_{t=1}^{|o|}\frac{\pi_{\theta}(o_{t}|q,o_{<t})}{\pi_{\theta_{old}}(o_{t}|q,o_{<t})}A_{t}. (16) The gradient of 𝒥 P ​ P ​ O ​ ( θ ) \mathcal{J}_{PPO}(\theta) is: ∇ θ 𝒥 P ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , o ∼ π θ o ​ l ​ d ​ ( O | q ) ] ​ 1 | o | ​ ∑ t = 1 | o | A t ​ ∇ θ ​ log ⁡ π θ ​ ( o t | q , o < t ) \footnotesize\begin{split}\nabla_{\theta}\mathcal{J}_{PPO}(\theta)=\mathbb{E}{[q\sim P_{sft}(Q),o\sim\pi_{\theta_{old}}(O|q)]}\frac{1}{|o|}\sum_{t=1}^{|o|}A_{t}\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\end{split} (17) Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function: reward model. Gradient Coefficient: G ​ C P ​ P ​ O ​ ( q , o , t , π θ r ​ m ) = A t , GC_{PPO}(q,o,t,\pi_{\theta_{rm}})=A_{t}, (18) where A t A_{t} is the advantage, which is computed by applying Generalized Advantage Estimation (GAE) ( Schulman et al., 2015 ) , based on the rewards { r ≥ t } \{r_{\geq t}\} and a learned value function V ψ V_{\psi} .

#### A.1.6 Group Relative Policy Optimization (GRPO)

The objective of GRPO is (assume π θ o ​ l ​ d = π θ \pi_{\theta_{old}}=\pi_{\theta} for simplified analysis): 𝒥 G ​ R ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , { o i } i = 1 G ∼ π θ o ​ l ​ d ​ ( O | q ) ] 1 G ​ ∑ i = 1 G 1 | o i | ​ ∑ t = 1 | o i | [ π θ ​ ( o i , t | q , o i , < t ) π θ o ​ l ​ d ​ ( o i , t | q , o i , < t ) ​ A ^ i , t − β ⁡ ( π r ​ e ​ f ​ ( o i , t | q , o i , < t ) π θ ​ ( o i , t | q , o i , < t ) − log ⁡ π r ​ e ​ f ​ ( o i , t | q , o i , < t ) π θ ​ ( o i , t | q , o i , < t ) − 1 ) ] . \footnotesize\begin{split}\mathcal{J}_{GRPO}(\theta)&=\mathbb{E}{[q\sim P_{sft}(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\ &\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left[\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{old}}(o_{i,t}|q,o_{i,<t})}\hat{A}_{i,t}-\beta(\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-\log\frac{\pi_{ref}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-1)\right].\end{split} (19) The gradient of 𝒥 G ​ R ​ P ​ O ​ ( θ ) \mathcal{J}_{GRPO}(\theta) is: ∇ θ 𝒥 G ​ R ​ P ​ O ​ ( θ ) = 𝔼 ⁡ [ q ∼ P s ​ f ​ t ​ ( Q ) , { o i } i = 1 G ∼ π θ o ​ l ​ d ​ ( O | q ) ] 1 G ​ ∑ i = 1 G 1 | o i | ​ ∑ t = 1 | o i | [ A ^ i , t + β ⁡ ( π r ​ e ​ f ​ ( o i , t | o i , < t ) π θ ​ ( o i , t | o i , < t ) − 1 ) ] ​ ∇ θ ​ log ⁡ π θ ​ ( o i , t | q , o i , < t ) . \footnotesize\begin{split}\nabla_{\theta}\mathcal{J}_{GRPO}(\theta)&=\mathbb{E}{[q\sim P_{sft}(Q),\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{old}}(O|q)]}\\ &\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\left[\hat{A}_{i,t}+\beta\left(\frac{\pi_{ref}(o_{i,t}|o_{i,<t})}{\pi_{\theta}(o_{i,t}|o_{i,<t})}-1\right)\right]\nabla_{\theta}\log\pi_{\theta}(o_{i,t}|q,o_{i,<t}).\end{split} (20) Data Source: question in SFT dataset with outputs sampled from policy model. Reward Function: reward model. Gradient Coefficient: G ​ C G ​ R ​ P ​ O ​ ( q , o , t , π θ r ​ m ) = A ^ i , t + β ⁡ ( π r ​ e ​ f ​ ( o i , t | o i , < t ) π θ ​ ( o i , t | o i , < t ) − 1 ) , \footnotesize GC_{GRPO}(q,o,t,\pi_{\theta_{rm}})=\hat{A}_{i,t}+\beta\left(\frac{\pi_{ref}(o_{i,t}|o_{i,<t})}{\pi_{\theta}(o_{i,t}|o_{i,<t})}-1\right), (21) where A ^ i , t \hat{A}_{i,t} is computed based on the group reward scores.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .