DAPO: An Open-Source LLM Reinforcement Learning System at Scale — part 2/3
Cleaned-text code points [15820, 31730). Bibliography omitted here; source.md includes it.
Note that this strategy does not necessarily impede training efficiency, because the generation time is typically dominated by the generation of long-tail samples if the RL system is synchronized and the generation stage is not pipelined. Besides, we find that with dynamic sampling the experiment achieves the same performance faster as shown in Figure 6 .
### 3.3 Rebalancing Act: Token-Level Policy Gradient Loss
The original GRPO algorithm employs a sample-level loss calculation, which involves first averaging the losses by token within each sample and then aggregating the losses across samples. In this approach, each sample is assigned an equal weight in the final loss computation. However, we find that this method of loss reduction introduces several challenges in the context of long-CoT RL scenarios.
Since all samples are assigned the same weight in the loss calculation, tokens within longer responses (which contain more tokens) may have a disproportionately lower contribution to the overall loss, which can lead to two adverse effects. First, for high-quality long samples, this effect can impede the model’s ability to learn reasoning-relevant patterns within them. Second, we observe that excessively long samples often exhibit low-quality patterns such as gibberish and repetitive words. Thus, sample-level loss calculation, due to its inability to effectively penalize those undesirable patterns in long samples, leads to an unhealthy increase in entropy and response length, as shown in Figure 4(a) and Figure 4(b) .
We introduce a Token-level Policy Gradient Loss in the long-CoT RL scenario to address the above limitations: 𝒥 DAPO ( θ ) = \displaystyle\mathcal{J}_{\text{DAPO}}(\theta)= 𝔼 ( q , a ) ∼ 𝒟 , { o i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) \displaystyle\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)} (12) [ 1 ∑ i = 1 G | o i | ∑ i = 1 G ∑ t = 1 | o i | min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε low , 1 + ε high ) A ^ i , t ) ] , \displaystyle\Bigg[\frac{1}{\color[rgb]{1,0,0}\sum_{i=1}^{G}|o_{i}|}{\color[rgb]{1,0,0}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}}\min\Big(r_{i,t}(\theta)\hat{A}_{i,t},\ \text{clip}\Big(r_{i,t}(\theta),1-{\varepsilon_{\text{low}}},1+{\varepsilon_{\text{high}}}\Big)\hat{A}_{i,t}\Big)\Bigg], s.t. \displaystyle\text{s.t.} 0 < | { o i ∣ is_equivalent ( a , o i ) } | < G . \displaystyle 0<\Big|\{o_{i}\mid\texttt{is\_equivalent}(a,o_{i})\}\Big|<G. In this setting, longer sequences can have more influence on the overall gradient update compared to shorter sequences. Moreover, from the perspective of individual tokens, if a particular generation pattern can lead to an increase or decrease in reward, it will be equally prompted or suppressed, regardless of the length of the response in which it appears.
### 3.4 Hide and Seek: Overlong Reward Shaping
In RL training, we typically set a maximum length for generation, with overlong samples truncated accordingly. We find that improper reward shaping for truncated samples can introduce reward noise and significantly disrupt the training process.
By default, we assign a punitive reward to truncated samples. This approach may introduce noise into the training process, as a sound reasoning process can be penalized solely due to its excessive length. Such penalties can potentially confuse the model regarding the validity of its reasoning process.
To investigate the impact of this reward noise, we first apply an Overlong Filtering strategy which masks the loss of truncated samples. We find that this approach significantly stabilizes training and enhances performance, as demonstrated in Figure 5 .
Furthermore, we propose Soft Overlong Punishment (Equation 13 ), a length-aware penalty mechanism designed to shape the reward for truncated samples. Specifically, when the response length exceeds the predefined maximum value, we define a punishment interval. Within this interval, the longer the response, the greater the punishment it receives. This penalty is added to the original rule-based correctness reward, thereby signaling to the model to avoid excessively long responses.
R length ( y ) = { 0 , | y | ≤ L max − L cache ( L max − L cache ) − | y | L cache , L max − L cache < | y | ≤ L max − 1 , L max < | y | R_{\text{length}}(y)=\begin{cases}0,&|y|\leq L_{\text{max}}-L_{\text{cache}}\\ \frac{(L_{\text{max}}-L_{\text{cache}})-|y|}{L_{\text{cache}}},&L_{\text{max}}-L_{\text{cache}}<|y|\leq L_{\text{max}}\\ -1,&L_{\text{max}}<|y|\end{cases} (13)
### 3.5 Dataset Transformation
Our dataset is sourced from the web and official competition homepages through a combination of web scraping and manual annotation. The answers of math dataset typically come in a variety of formats, such as expression, formula and number, which makes it challenging to design comprehensive rules to parse them. To provide accurate reward signals using rules and minimize errors introduced by formula parsers, inspired by AIME, we select and transform the answers into integers, which are easy to parse. For example, if the original answer is expressed in the form of a + b c \frac{a+\sqrt{b}}{c} , we instruct the LLM to modify the question so that the expected answer becomes a + b + c a+b+c . After selection and transformation, we obtained the DAPO -Math-17K dataset, which consists of 17K prompts, each paired with an integer as the answer.
## 4 Experiments
### 4.1 Training Details
In this work, we focus specifically on mathematical tasks to evaluate our algorithm, which can be readily transferred to other tasks. We adopt the verl framework [ 20 ] for training. We use naive GRPO [ 38 ] as our baseline algorithm and estimate advantages using group reward normalization.
For hyper-parameters, we utilize the AdamW [ 39 ] optimizer with a constant learning rate of 1 × 10 − 6 1\times 10^{-6} , incorporating a linear warm-up over 20 rollout steps. For rollout, the prompt batch size is 512 and we sample 16 responses for each prompt. For training, the mini-batch size is set to 512, i.e., 16 gradient updates for each rollout step. For Overlong Reward Shaping , we set the expected maximum length as 16,384 tokens and allocate additional 4,096 tokens as the soft punish cache. Therefore, the maximum number of tokens for generation is set to 20,480 tokens. As for the Clip-Higher mechanism, we set the clipping parameter ε low \varepsilon_{\text{low}} to 0.2 and ε high \varepsilon_{\text{high}} to 0.28, which effectively balance the trade-off between exploration and exploitation. For evaluation on AIME, we repeat the evaluation set for 32 times and report avg@32 for results stability. The inference hyperparameters of evaluation are set to temperature 1.0 and topp 0.7.
### 4.2 Main Results
Experiments on AIME 2024 demonstrate that DAPO has successfully trained the Qwen-32B Base model into a powerful reasoning model, achieving performance superior to DeepSeek’s experiments on Qwen2.5-32B using the R1 approach. In Figure 1 , we observe a substantial improvement of performance on AIME 2024, with accuracy increasing from near 0 0 % to 50%. Notably, this improvement is achieved with only 50% of the training steps required by DeepSeek-R1-Zero-Qwen-32B.
We analyze the contributions of each training technique in our methodology, as detailed in Table 1 . The observed improvements demonstrate the effectiveness of these techniques in RL training, each contributing several accuracy points in AIME 2024. Notably, given the vanilla GRPO setting, only 30% accuracy can be reached by training from a Qwen2.5-32B base model.
For token-level loss, although it brings less performance improvement, we find it enhances training stability and makes the length increase more healthily.
When applying Dynamic Sampling , although more data needs to be sampled due to the filtering out of zero-gradient data, the overall training time is not significantly affected. As shown in Figure 6 , although the number of sampling instances increases, the model’s convergence time is even reduced, due to fewer training steps required.
### 4.3 Training Dynamics
Reinforcement learning on large language models is not only a cutting-edge research direction but also an intrinsically complex systems engineering challenge, characterized by the interdependence of its various subsystems. Modifications to any single subsystem can propagate through the system, leading to unforeseen consequences due to the intricate interplay among these components. Even seemingly minor changes in initial conditions, such as variations in data and hyperparameters, can amplify through iterative reinforcement learning processes, yielding substantial deviations in outcomes. This complexity often confronts researchers with a dilemma: even after meticulous analysis and well-founded expectations that a modification will enhance specific aspects of the training process, the actual results frequently diverge from the anticipated trajectory. Therefore, monitoring of key intermediate results during experimentation is essential for swiftly identifying the sources of discrepancies and, ultimately, for refining the system.
• The Length of Generated Responses is a metric closely related to training stability and performance, as shown in Figure 7(a) . The increase in length provides the model with a larger space for exploration, allowing more complex reasoning behaviors to be sampled and gradually reinforced through training. However, it is important to note that length does not always maintain a continuous upward trend during training. In some considerable periods, it can exhibit a trend of stagnation or even decline, which has also been demonstrated in [ 2 ] . We typically use length in conjunction with validation accuracy as indicators to assess whether an experiment is deteriorating.
• The Dynamics of Reward during training has always been one of the crucial monitoring indicators in reinforcement learning, as shown in Figure 7(b) . In the majority of our experiments, the trend of reward increase is relatively stable and does not fluctuate or decline significantly due to adjustments in experimental settings. This indicates that, given a reliable reward signal, language models can robustly fit the distribution of training set. However, we find that the final reward on the training set often exhibits little correlation with the accuracy on the validation set, which indicates overfitting to the training set.
• The Entropy of the Actor Model and Generation Probability are related to the model’s exploration capability and are key metrics that we closely monitor in our experiments. Intuitively, the model’s entropy needs to be maintained within an appropriate range. An excessively low entropy indicates that the probability distribution is overly sharp, leading to a loss of exploration capability. Conversely, an excessively high entropy is often associated with issues of over-exploration such as gibberish and repetitive generation. For the generation probability, the situation is exactly the opposite. As demonstrated in Section 3.1 , by applying the Clip-Higher strategy, we effectively addressed the issue of entropy collapse. In subsequent experiments, we find that maintaining a slow upward trend in entropy is conducive to the improvement of model performance, shown in Figure 7(c) and Figure 7(d) .
### 4.4 Case Study
During the RL training process, we observe an interesting phenomenon: the reasoning patterns of the actor model evolve dynamically over time. Specifically, the algorithm not only reinforces existing reasoning patterns that facilitate correct problem-solving but also gradually gives rise to entirely new modes of reasoning that were initially absent. This finding reveals the adaptability and exploration capability of RL algorithms and offers new insights into the learning mechanisms of the model.
For example, in the early stages of model training, there was virtually no occurrence of checking and reflecting on previous reasoning steps. However, as training progresses, the model exhibits distinct behaviors of reflection and backtracking, as shown in Table 2 . This observation sheds light on further exploration into interpreting the emergence of reasoning abilities during RL, which we leave for future research.
## 5 Conclusion
In this paper, we release a fully open-sourced system for large-scale LLM RL, including algorithm, code infrastructure, and dataset. The system achieves state-of-the-art large-scale LLM RL performance (AIME 50 using Qwen-32B pretrained model). We propose the D ecoupled Clip and D ynamic s A mpling P olicy O ptimization ( DAPO ) algorithm, and introduce 4 key techniques to make RL powerfully effective and efficient in the long-CoT RL scenario. Additionally, by open-sourcing the training code and dataset, we provide the broader research community and society with practical access to a scalable reinforcement learning solution, enabling all to benefit from these advancements.
## Contributions
Project Lead
Qiying Yu 1,2,4
Algorithm
Qiying Yu 1,2,4 , Zheng Zhang 1 , Ruofei Zhu 1 , Yufeng Yuan 1 , Xiaochen Zuo 1 , Yu Yue 1
Infrastructure ∗
Weinan Dai 1,2,4 , Tiantian Fan 1 , Gaohong Liu 1 , Juncai Liu 1 , Lingjun Liu 1 , Xin Liu 1 , Haibin Lin 1 , Zhiqi Lin 1 , Bole Ma 1 , Guangming Sheng 1,3 , Yuxuan Tong 1,2,4 , Qiying Yu 1,2,4 , Chi Zhang 1 , Mofan Zhang 1 , Ru Zhang 1 , Wang Zhang 1 , Hang Zhu 1 , Jinhua Zhu 1
∗ Last-Name in Alphabetical Order
Dataset
Jiaze Chen 1 , Jiangjie Chen 1,4 , Chengyi Wang 1 , Hongli Yu 1,2,4 , Yuxuan Song 1,2,4 , Xiangpeng Wei 1 , Qiying Yu 1,2,4
Supervision
Hao Zhou 2,4 , Jingjing Liu 2,4 , Wei-Ying Ma 2,4 , Ya-Qin Zhang 2,4 , Lin Yan 1,4 , Mu Qiao 1,4 , Yonghui Wu 1 , Mingxuan Wang 1,4
Affiliation
1 ByteDance Seed
2 Institute for AI Industry Research (AIR), Tsinghua University
3 The University of Hong Kong
4 SIA-Lab of Tsinghua AIR and ByteDance Seed
## Acknowledgments
We thank Zhengyin Du, Shengding Hu, Kai Shen, Tianyang Zhan, Zhen Xiao, Renjie Zheng, Li Han, Kaihua Jiang as well as other colleagues at ByteDance for their support for the DAPO project.
## Appendix A Dataset Transformation
Here is an example of data transformation, where we demonstrate the original problem, the transformed problem, and the model outputs during the transformation process (the original is in markdown text, which has been rendered for readability).
To ensure the model can complete this task with high accuracy, our goal is to have it perform comprehensive reasoning while avoiding hallucinations. We achieve this by providing a clear thinking framework that encourages thorough reasoning. Specifically, we employ Chain of Thought (CoT) reasoning along with four well-defined steps: extract the answer format, rewrite the problem statement, solve the modified problem and provide an integer as the final answer. For each step, we provide few-shot solutions or detailed guidelines to guide the model’s reasoning.
Through this approach, we have observed that in most cases, the LLM can generate reformulations with both format and quality that are satisfactory.
\tcbset colback=seedblue!10!white, colframe=seedblue, width=arc=5mm {tcolorbox}
### Original Problem
Let x x and y y be real numbers such that x 2 + y 2 − 22 x − 16 y + 113 = 0 . x^{2}+y^{2}-22x-16y+113=0. Determine the smallest possible value of x . x. Answer: 11 − 2 6 11-2\sqrt{6}
### Transformed Problem
Let x x and y y be real numbers such that x 2 + y 2 − 22 x − 16 y + 113 = 0 . x^{2}+y^{2}-22x-16y+113=0. Determine the smallest possible value of x x . The original answer is in the form k − m n k-m\sqrt{n} , where k k , m m , and n n are integers. Please find the value of k + m + n k+m+n . Answer: 19 19
### Model’s output (Markdown Text)
#### Transformation Thought Process