DAPO: An Open-Source LLM Reinforcement Learning System at Scale — part 1/3
Cleaned-text code points [0, 15820). Bibliography omitted here; source.md includes it.
# DAPO: An Open-Source LLM Reinforcement Learning System at Scale
###### Abstract
Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical details of state-of-the-art reasoning LLMs are concealed (such as in OpenAI o1 blog and DeepSeek R1 technical report), thus the community still struggles to reproduce their RL training results. We propose the D ecoupled Clip and D ynamic s A mpling P olicy O ptimization ( DAPO ) algorithm, and fully open-source a state-of-the-art large-scale RL system that achieves 50 points on AIME 2024 using Qwen2.5-32B base model. Unlike previous works that withhold training details, we introduce four key techniques of our algorithm that make large-scale LLM RL a success. In addition, we open-source our training code, which is built on the verl framework , along with a carefully curated and processed dataset. These components of our open-source system enhance reproducibility and support future research in large-scale LLM RL.
## 1 Introduction
Test-time scaling such as OpenAI’s o1 [ 1 ] and DeepSeek’s R1 [ 2 ] brings a profound paradigm shift to Large Language Models (LLMs) [ 3 , 4 , 5 , 6 , 7 ] . Test-time scaling enables longer Chain-of-Thought thinking and induces sophisticated reasoning behaviors, which makes the models superior in competitive math and coding tasks like AIME and Codeforces.
The central technique driving the revolution is large-scale Reinforcement Learning (RL), which elicits complex reasoning behaviors such as self-verification and iterative refinement. However, the actual algorithm and key recipe for scalable RL training remains a myth, hidden from technical reports of existing reasoning models [ 1 , 2 , 8 , 9 , 10 , 11 ] . In this paper, we reveal significant obstacles in large-scale RL training and open-source a scalable RL system with fully open-sourced algorithm, training code and dataset that provides democratized solutions with industry-level RL results.
We experiment over Qwen2.5-32B [ 12 ] as the pretrained model for RL. In our initial GRPO run, we achieved only 30 points on AIME — a performance significantly below DeepSeek’s RL (47 points). A thorough analysis reveals that the naive GRPO baseline suffers from several key issues such as entropy collapse, reward noise, and training instability. The broader community has encountered similar challenges in reproducing DeepSeek’s results [ 13 , 14 , 15 , 16 , 17 , 18 , 19 ] suggesting that critical training details may have been omitted in the R1 paper that are required to develop an industry-level, large-scale, and reproducible RL system.
To close this gap, we release an open-source state-of-the-art system for large-scale LLM RL, which achieves 50 points on AIME 2024 based on Qwen2.5-32B model, outperforming previous state-of-the-art results achieved by DeepSeek-R1-Zero-Qwen-32B [ 2 ] (47 points) using 50% training steps (Figure 1 ). We propose the D ecoupled Clip and D ynamic s A mpling P olicy O ptimization ( DAPO ) algorithm, and introduce 4 key techniques to make RL shine in the long-CoT RL scenario. Details are presented in Section 3 . 1. Clip-Higher , which promotes the diversity of the system and avoids entropy collapse;
2. Dynamic Sampling , which improves training efficiency and stability;
3. Token-Level Policy Gradient Loss , which is critical in long-CoT RL scenarios;
4. Overlong Reward Shaping , which reduces reward noise and stabilizes training.
Our implementation is based on verl [ 20 ] . By fully releasing our state-of-the-art RL system including training code and data, we aim to reveal valuable insights to large-scale LLM RL that benefit the larger community.
## 2 Preliminary
### 2.1 Proximal Policy Optimization (PPO)
PPO [ 21 ] introduces a clipped surrogate objective for policy optimization. By constraining the policy updates within a proximal region of the previous policy using clip, PPO stabilizes training and improves sample efficiency. Specifically, PPO updates the policy by maximizing the following objective: 𝒥 PPO ( θ ) = 𝔼 ( q , a ) ∼ 𝒟 , o ≤ t ∼ π θ old ( ⋅ ∣ q ) [ min ( π θ ( o t ∣ q , o < t ) π θ old ( o t ∣ q , o < t ) A ^ t , clip ( π θ ( o t ∣ q , o < t ) π θ old ( o t ∣ q , o < t ) , 1 − ε , 1 + ε ) A ^ t ) ] , \displaystyle\mathcal{J}_{\text{PPO}}(\theta)=\mathbb{E}_{(q,a)\sim\mathcal{D},o_{\leq t}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}\Bigg[\min\Bigg(\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\theta_{\text{old}}}(o_{t}\mid q,o_{<t})}\hat{A}_{t},\ \text{clip}\Bigg(\frac{\pi_{\theta}(o_{t}\mid q,o_{<t})}{\pi_{\theta_{\text{old}}}(o_{t}\mid q,o_{<t})},1-\varepsilon,1+\varepsilon\Bigg)\hat{A}_{t}\Bigg)\Bigg], (1) where ( q , a ) (q,a) is a question-answer pair from the data distribution 𝒟 \mathcal{D} , ε \varepsilon is the clipping range of importance sampling ratio, and A ^ t \hat{A}_{t} is an estimator of the advantage at time step t t . Given the value function V V and the reward function R R , A ^ t \hat{A}_{t} is computed using the Generalized Advantage Estimation (GAE) [ 22 ] : A ^ t GAE ( γ , λ ) = ∑ l = 0 ∞ ( γ λ ) l δ t + l , \displaystyle\hat{A}_{t}^{\text{GAE}(\gamma,\lambda)}=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+l}, (2) where δ l = R l + γ V ( s l + 1 ) − V ( s l ) , 0 ≤ γ , λ ≤ 1 . \delta_{l}=R_{l}+\gamma V(s_{l+1})-V(s_{l}),\quad 0\leq\gamma,\lambda\leq 1. (3)
### 2.2 Group Relative Policy Optimization (GRPO)
Compared to PPO, GRPO eliminates the value function and estimates the advantage in a group-relative manner. For a specific question-answer pair ( q , a ) (q,a) , the behavior policy π θ old \pi_{\theta_{\text{old}}} samples a group of G G individual responses { o i } i = 1 G \{o_{i}\}_{i=1}^{G} . Then, the advantage of the i i -th response is calculated by normalizing the group-level rewards { R i } i = 1 G \{R_{i}\}_{i=1}^{G} : A ^ i , t = r i − mean ( { R i } i = 1 G ) std ( { R i } i = 1 G ) . \hat{A}_{i,t}=\frac{r_{i}-\text{mean}(\{R_{i}\}_{i=1}^{G})}{\text{std}(\{R_{i}\}_{i=1}^{G})}. (4)
Similar to PPO, GRPO adopts a clipped objective, together with a directly imposed KL penalty term: 𝒥 GRPO ( θ ) \displaystyle\mathcal{J}_{\text{GRPO}}(\theta) = 𝔼 ( q , a ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle=\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} (5) [ 1 G ∑ i = 1 G 1 | o i | ∑ t = 1 | o i | ( min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) − β D KL ( π θ | | π ref ) ) ] , \displaystyle\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\Bigg(\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},\ \text{clip}\Big(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon\Big)\hat{A}_{i,t}\Big)-\beta D_{\text{KL}}(\pi_{\theta}||\pi_{\text{ref}})\Bigg)\Bigg], where r i , t ( θ ) = π θ ( o i , t ∣ q , o i , < t ) π θ old ( o i , t ∣ q , o i , < t ) . r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}. (6)
It is also worth noting that GRPO computes the objective at the sample-level. To be exact, GRPO first calculates the mean loss within each generated sequence, before averaging the loss of different samples. As we will be discussing in Section 3.3 , such difference may have an impact on the performance of the algorithm.
### 2.3 Removing KL Divergence
The KL penalty term is used to regulate the divergence between the online policy and the frozen reference policy. In the RLHF scenario [ 23 ] , the goal of RL is to align the model behavior without diverging too far from the initial model. However, during training the long-CoT reasoning model, the model distribution can diverge significantly from the initial model, thus this restriction is not necessary. Therefore, we will exclude the KL term from our proposed algorithm.
### 2.4 Rule-based Reward Modeling
The use of reward model usually suffers from the reward hacking problem [ 24 , 25 , 26 , 27 , 28 , 29 ] . Instead, we directly use the final accuracy of a verifiable task as the outcome reward, computed using the following rule: R ( y ^ , y ) = { 1 , is_equivalent ( y ^ , y ) − 1 , otherwise R(\hat{y},y)=\begin{cases}1,&\texttt{is\_equivalent}(\hat{y},y)\\ -1,&\text{otherwise}\end{cases} (7) where y y is the ground-truth answer and y ^ \hat{y} is the predicted answer. This is proved to be an effective approach to activating the base model’s reasoning capability, as shown in multiple domains such as automated theorem proving [ 30 , 31 , 32 , 33 ] , computer programming [ 34 , 35 , 36 , 37 ] , and mathematics competition [ 2 ] .
## 3 DAPO
We propose the D ecouple Clip and D ynamic s A mpling P olicy O ptimization (DAPO) algorithm. DAPO samples a group of outputs { o i } i = 1 G \{o_{i}\}_{i=1}^{G} for each question q q paired with the answer a a , and optimizes the policy via the following objective: 𝒥 DAPO ( θ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ( q , a ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} (8) [ 1 ∑ i = 1 G | o i | ∑ i = 1 G ∑ t = 1 | o i | min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε low , 1 + ε high ) A ^ i , t ) ] \displaystyle\Bigg[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},\ \text{clip}\Big(r_{i,t}(\theta),1-{\varepsilon_{\text{low}}},1+{\varepsilon_{\text{high}}}\Big)\hat{A}_{i,t}\Big)\Bigg] s.t. \displaystyle\text{s.t.} 0 < | { o i ∣ is_equivalent ( a , o i ) } | < G , \displaystyle 0<\Big|\{o_{i}\mid\texttt{is\_equivalent}(a,o_{i})\}\Big|<G, where r i , t ( θ ) = π θ ( o i , t ∣ q , o i , < t ) π θ old ( o i , t ∣ q , o i , < t ) , A ^ i , t = R i − mean ( { R i } i = 1 G ) std ( { R i } i = 1 G ) . r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})},\quad\hat{A}_{i,t}=\frac{R_{i}-\text{mean}(\{R_{i}\}_{i=1}^{G})}{\text{std}(\{R_{i}\}_{i=1}^{G})}. (9) The full algorithm can be found in Algorithm 1 . In this section, we will introduce the key techniques associated with DAPO.
### 3.1 Raise the Ceiling: Clip-Higher
In our initial experiments using naive PPO [ 21 ] or GRPO [ 38 ] , we observed the entropy collapse phenomenon: the entropy of the policy decreases quickly as training progresses ( Figure 2(b) ). The sampled responses of certain groups tend to be nearly identical. This indicates limited exploration and early deterministic policy, which can hinder the scaling process.
We propose the Clip-Higher strategy to address this issue. Clipping over the importance sampling ratio is introduced in Clipped Proximal Policy Optimization (PPO-Clip) [ 21 ] to restrict the trust region and enhance the stability of RL. We identify that the upper clip can restrict the exploration of the policy, where making an ‘exploitation’ token more probable is much easier yet the probability of an unlikely ‘exploration’ token is too tightly bounded to be uplifted.
Concretely, when ε = 0.2 \varepsilon=0.2 (the default value of most algorithms) and A ^ i , t > 0 \hat{A}_{i,t}>0 (the system tries to increase the probability), consider two actions with probabilities π θ old ( o i ∣ q ) = 0.01 \pi_{\theta_{\text{old}}}(o_{i}\mid q)=0.01 and 0.9 0.9 . The upper bounds of the increased probabilities π θ ( o i ∣ q ) \pi_{\theta}(o_{i}\mid q) are 0.012 0.012 and 1.08 1.08 , respectively ( π θ old ⋅ ( 1 + ϵ ) ) \left(\pi_{\theta_{\text{old}}}\cdot(1+\epsilon)\right) . This implies that ‘exploitation’ tokens with a higher probability ( e.g. , 0.9) are not constrained to get even extremely larger probabilities like 0.999. Conversely, for low-probability ‘exploration’ tokens, achieving a non-trivial increase in probability is considerably more challenging. Empirically, we also observe that the mean probability of up-clipped tokens is low: π θ ( o i ∣ q ) < 0.2 \pi_{\theta}(o_{i}\mid q)<0.2 ( Figure 3(a) ). This finding supports our intuition that the upper clipping threshold indeed restricts the probability increase of low-probability ‘exploration’ tokens, thereby potentially constraining the exploration of the system.
Adhering to the Clip-Higher strategy, we decouple the lower and higher clipping range as ε low \varepsilon_{\text{low}} and ε high \varepsilon_{\text{high}} , as highlighted in Equation 10 : 𝒥 DAPO ( θ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ( q , a ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} (10) [ 1 ∑ i = 1 G | o i | ∑ i = 1 G ∑ t = 1 | o i | min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε low , 1 + ε high ) A ^ i , t ) ] \displaystyle\Bigg[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},\ \text{clip}\Big(r_{i,t}(\theta),1-{\color[rgb]{1,0,0}\varepsilon_{\text{low}}},1+{\color[rgb]{1,0,0}\varepsilon_{\text{high}}}\Big)\hat{A}_{i,t}\Big)\Bigg] s.t. \displaystyle\text{s.t.} 0 < | { o i ∣ is_equivalent ( a , o i ) } | < G . \displaystyle 0<\Big|\{o_{i}\mid\texttt{is\_equivalent}(a,o_{i})\}\Big|<G. We increase the value of ε high \varepsilon_{\text{high}} to leave more room for the increase of low-probability tokens. As shown in Figure 2 , this adjustment effectively enhances the policy’s entropy and facilitates the generation of more diverse samples. We keep ε low \varepsilon_{\text{low}} as it is, because increasing it will suppress the probability of these tokens to 0 0 , resulting in the collapse of the sampling space.
### 3.2 The More the Merrier: Dynamic Sampling
Existing RL algorithm suffers from the gradient-decreasing problem when some prompts have accuracy equal to 1. For example for GRPO, if all outputs { o i } i = 1 G \{o_{i}\}_{i=1}^{G} of a particular prompt are correct and receive the same reward, the resulting advantage for this group is zero . A zero advantage results in zero policy gradients, shrinking the magnitude and increasing the noise sensitivity of the batch gradient, thereby degrading sample efficiency. Empirically, the number of samples with accuracy equal to 1 continues to increase, as shown in Figure 3(b) . This means that the effective number of prompts in each batch keeps decreasing, which can lead to larger variance in gradient and dampens the gradient signals for model training.
To this end, we propose to over-sample and filter out prompts with the accuracy equal to 1 and 0 as illustrated in Equation 11 , leaving all prompts in the batch with effective gradients and keeping a consistent number of prompts. The sampling cost for each batch is dynamic. Before training, we keep sampling until the batch is fully filled with samples whose accuracy is neither 0 nor 1.
𝒥 DAPO ( θ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ( q , a ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} (11) [ 1 ∑ i = 1 G | o i | ∑ i = 1 G ∑ t = 1 | o i | min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε low , 1 + ε high ) A ^ i , t ) ] \displaystyle\Bigg[\frac{1}{\sum_{i=1}^{G}|o_{i}|}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},\ \text{clip}\Big(r_{i,t}(\theta),1-{\varepsilon_{\text{low}}},1+{\varepsilon_{\text{high}}}\Big)\hat{A}_{i,t}\Big)\Bigg] s.t. \displaystyle\text{s.t.} 0 < | { o i ∣ is_equivalent ( a , o i ) } | < G . \displaystyle\color[rgb]{1,0,0}{\displaystyle 0<\Big|\{o_{i}\mid\texttt{is\_equivalent}(a,o_{i})\}\Big|<G}.