The Art of Scaling Reinforcement Learning Compute for LLMs — part 3/5

Cleaned-text code points [31574, 47337). Bibliography omitted here; source.md includes it.
Increasing the generation length from 14k to 32k tokens slows early progress (lower B B and higher C m  i  d C_{mid} ) but consistently lifts the fitted asymptote (A), yielding higher final performance once sufficient compute is provided ( Figure 9 ). This validates long-context RL as a ceiling-raising knob rather than a mere efficiency trade-off. Extrapolations made from the fit correctly forecast the higher 32k-token trajectory when training is extended.

##### Global batch size (prompts)

Smaller-batch runs show early stagnation on downstream benchmarks even as in-distribution validation performance continues to improve. Larger batches reliably improve the asymptote and avoid the downstream stagnation we observe in smaller-batch runs. Figure 10a shows the same qualitative pattern at mid-scale: small batches may appear better early but are overtaken as compute grows. In our largest math run in Figure 1 , moving to batch size of 2048 prompts both stabilized training and yielded a fit that extrapolated from up to 50k GPU hours to the final 100k point.

##### Generations per prompt (fixed total batch)

For a fixed total batch, is it better to allocate more prompts or more generations per prompt? Sweeping generations per prompt 8,16,24,32 and adjusting prompts to keep total batch fixed leaves fitted scaling curves essentially unchanged (Appendix A.13 ), suggesting that, at moderate batch, this allocation is a second-order choice for both A and B. Clearer differences may emerge at much larger batches (e.g., 2k+), which we leave for future work.

## 6 Related Work

We detail two most relevant works to our study in this section. ProRL ( Liu et al., 2025a ) demonstrates that prolonged RL fine-tuning on LLMs ( ∼ 2000 \sim 2000 optimization steps, 64 batch size) for 16K GPU-hours using a mix of reasoning tasks uncovers novel solution strategies beyond a model’s base capabilities. This longer training regimen delivered significant gains on a 1.5B model, rivaling the performance of larger models on some benchmarks. ProRL’s contributions lie in specific heuristics for stability (KL-regularization, policy resetting, entropy controls, etc.) to achieve high performance out of a 1.5B model.

Liu et al. (2025c) offer a complementary perspective and ablates various design choices under consistent conditions on Qwen-3 4B/8B ( Yang et al., 2025 ) , and presents a minimalist combination, LitePPO, that outperforms more complex methods like GRPO ( Shao et al., 2024 ) and DAPO ( Yu et al., 2025 ) on smaller scale models and compute. This yields valuable algorithmic insights, but the focus is on comparative empirical findings, rather than on scaling behaviour.

None of these work study “scaling” properties of these methods. In fact, the main comparisons are done on downstream evaluations, which may not be not the right metric to study predictable scaling. Rather, as done in pre-training and in our work here, we study performance on in-distribution held out eval set. In contrast to the mentioned related works, our work develops and validates a compute-performance framework with predictive fits, while operating at a much larger compute budget (e.g, 6  x 6x larger than ProRL) and model scale compared to the above studies. Additionally, our findings yield a near state-of-the-art RL recipe that can scale predictably to over 100,000 GPU-hours without any stability issues. The rest of the related work is deferred to Appendix A.1 .

## 7 Discussion & Conclusion

In this work, we study the scaling properties of different techniques used in RL for LLMs in pursuit of a predictable scalable recipe. With this mission, we derive a method for fitting predictive scaling fits for accuracy on the validation set that allows us to quantify the asymptotic performance and compute efficiency of an RL method. Using this methodology, our primary contribution is to conduct a careful series of ablations of several algorithmic options that go into the RL recipe. For each ablation, we choose the option with higher asymptotic performance when possible and improved efficiency otherwise. Combining these choices yields the ScaleRL recipe which scales better than all existing recipes in our experiments.

A few observations are in order: • Compute scaling extrapolation . An important insight of our scaling methodology is that we can use smaller-scale ablations in a systematic way to predict performance at larger scales. This allows us to create our final scalable recipe.

• Most important decisions . The off-policy algorithm, loss function, and model precision are the most important decisions from our ablations. Each of the other decisions does not have a large individual effect, but as we see from the leave-one-out experiments, they still do have some cumulative impact (in terms of efficiency) when all combined.

• Asymptotic performance vs. efficiency . For many of our ablations, we found the better option to improve both efficiency and asymptotic performance, but this is not always the case (e.g. for FP32, Figure 5b ). When doing the “forward” ablations starting from the baseline method, we opt for asymptotic performance first and foremost. Interestingly, when doing the “backward” leave-one-out ablations from the ScaleRL recipe, we find very little impact on asymptotic performance from each decision, but each component of the algorithm seems to help efficiency. This shows that the cumulative effect of the changes is quite robust.

• Generalization . While we report transfer to downstream evaluations, our primary focus is on studying predictive scaling, which is characterized through in-distribution performance curves on a held-out dataset from training prompts ( Li et al., 2025b ; Muennighoff et al., 2025 ) . This still leaves the question of how well the LLM would generalize from the training distribution to held out test sets. While a full characterization of generalization is beyond the scope of our work, we do observe correlation between in-distribution validation and downstream generalization performance. However, there are some algorithmic choices that seem to help generalization more, that we want to note here including: larger batch size ( Section A.14 ), reducing truncations ( Section A.15 ), longer generation lengths ( Section 5 , Figure 9 ), and larger model scale ( Section 5 , Figure 1 ).

• Multi-task RL . While our experiments focus mainly on the math domain, we also evaluate ScaleRL under multi-task RL training. As shown in Figure 11 , joint training on math and code yields clean, parallel power-law trends for each domain, with extended runs remaining aligned with the extrapolated curves. While our preliminary results are promising, it would be interesting to thoroughly study predictability of compute scaling for multi-task RL with different training data mixtures.

##### Future work

A natural next step is to derive predictive “scaling laws” for RL across pre-training compute, model size, and RL training data. Future studies can also include other axes of RL compute scaling, such as incorporating structured or dense rewards ( Setlur et al., 2024 ) and more compute-intensive generative verifiers ( Zhang et al., 2025a ) , to find optimal compute allocation for RL training. Finally, the methodological framework introduced here can be applied to study the scaling behavior of other post-training regimes, including multi-turn RL, agentic interaction, and long-form reasoning.

There are of course many design choices in RL, so we don’t think that our ScaleRL recipe is the end of the story. We hope that our focus on scalable RL and methodology for predicting scalability can inspire future work to push the frontier of RL for LLMs even further. To enable future studies to fit compute-performance RL scaling curves, we release a minimal code repository at www.devvrit.com/scalerl_curve_fitting .

## 8 Acknowledgments

The authors would like to thank Sneha Kudugunta and Niladri Chatterji for helpful discussions on pre-training and scaling laws. Additionally, the authors are grateful to Aviral Kumar, Prateek Jain, Lewis Turnstall, Nathan Lambert, Dzmitry Bahdanau, and John Quan for helpful feedback on earlier drafts. Finally, the authors would also like to thank Jenya Lee and Abhinav Jauhri for infrastructure and compute support.

## Appendix A Appendix

### A.1 Extended Related Work

A wave of recent work has applied Reinforcement Learning (RL) to improve the reasoning abilities of large language models (LLMs); often achieving state-of-the-art results on challenging tasks ( OpenAI, 2024 ; Guo et al., 2025 ; Seed et al., 2025 ; Carbonneaux et al., 2025 ) . OpenAI’s o1 series of models established that large-scale RL can substantially enhance long-horizon reasoning, but did not release any details on how these models were trained. Deepseek R1 (and R1-Zero) ( Guo et al., 2025 ) provided the first comprehensive study on training high-performing and long Chain-of-Thought (CoT) models primarily via RL, documenting emergent behaviours under extended RL without any reliance on reward models ( Lightman et al., 2023 ) or Monte Carlo Tree Search (MCTS) ( Xie et al., 2024 ) .

The earliest widely referenced RLVR (verifiable-reward) algorithm underlying this wave of reasoning development is Group Relative Policy Optimization (GRPO), introduced in Shao et al. (2024) . GRPO is a critic-free, group-relative policy gradient with PPO-style clipping that replaces a learned value baseline with group baselines to reduce computational cost and stabilize credit assignment for long CoTs. While GRPO catalyzed rapid progress, subsequent work document its limitations (token-level clipping, model collapse risks) and motivate different group- or sequence- level variants ( Yu et al., 2025 ; Yue et al., 2025 ; Hu et al., 2025b ; Zheng et al., 2025a ) .

Yu et al. (2025) propose the Decoupled clip and Dynamic Sampling Policy Optimization (DAPO), where they decouple ϵ low {\epsilon}_{\texttt{low}} and ϵ high {\epsilon}_{\texttt{high}} clipping in the GRPO objective and do Clip-Higher for ϵ high {\epsilon}_{\texttt{high}} to avoid entropy collapse. Furthermore, they do dynamic sampling of prompts in a given batch to avoid samples with zero variance (or advantage) which contribute zero policy gradients. Finally, they employ token-level loss aggregation unlike GRPO, which uses sample-level loss averaging. With these modifications, they are able to surpass the vanilla GRPO baseline while avoiding entropy collapse in the RL training. In parallel, Yue et al. (2025) develop VAPO; a value-augmented PPO tailored for long CoTs with strong stability and outperforming value-free baselines like GRPO and DAPO. They combine value pre-training and decoupled Generalized Advantage Estimation (GAE) from VC-PPO ( Yuan et al., 2025 ) , loss objective modifications from DAPO, and propose length-adaptive GAE to come up with an open recipe, VAPO, that has been used to train large MoE models in Seed et al. (2025) . Similarly, other technical report like Magistral ( Rastogi et al., 2025 ) , Kimi-k1.5 ( Kimi Team et al., 2025b ) , Minimax-01 ( Li et al., 2025a ) detail various details on their RL training recipes, but don’t share extensive experiments on why their design choices are better than the baselines.

### A.2 RL for LLMs: GRPO and DAPO

##### Group Relative Policy Optimization (GRPO)

GRPO ( Shao et al., 2024 ) adapts PPO Schulman et al. (2017) for LLM fine-tuning with verifiable rewards. For a given prompt x x , the old policy π gen  ( θ old ) \pi_{\text{gen}}(\theta_{\text{old}}) generates G G candidate completions { y i } i = 1 G \{y_{i}\}_{i=1}^{G} , each assigned a scalar reward r i r_{i} . To emphasize relative quality within the group, rewards are normalized as A ^ i = r i − mean ⁡ ( { r j } j = 1 G ) std ⁡ ( { r j } j = 1 G ) + ϵ . \hat{A}_{i}=\frac{r_{i}-\mathrm{mean}(\{r_{j}\}_{j=1}^{G})}{\mathrm{std}(\{r_{j}\}_{j=1}^{G})+\epsilon}. (5) Each completion y i y_{i} of length | y i | |y_{i}| contributes at the token level through ratios ρ i , t  ( θ ) = π t  r  a  i  n  ( y i , t ∣ x , y i , < t , θ ) π g  e  n  ( y i , t ∣ x , y i , < t , θ old ) \rho_{i,t}(\theta)=\frac{\pi_{train}(y_{i,t}\mid x,y_{i,<t},\theta)}{\pi_{gen}(y_{i,t}\mid x,y_{i,<t},\theta_{\text{old}})} (6) The GRPO objective averages across both completions and tokens: 𝒥 GRPO  ( θ ) = 𝔼 x ∼ D , { y i } i = 1 G ∼ π gen ( ⋅ ∣ x , θ old )  [ 1 G  ∑ i = 1 G 1 | y i |  ∑ t = 1 | y i | min ⁡ ( ρ i , t  ( θ )  A ^ i , clip ⁡ ( ρ i , t  ( θ ) , 1 ± ϵ )  A ^ i ) ] \mathcal{J}_{\mathrm{GRPO}}(\theta)=\mathbb{E}_{\begin{subarray}{l}\,x\sim D,\\ \{y_{i}\}_{i=1}^{G}\sim\pi_{\text{gen}(\cdot\mid x,\theta_{\text{old}})}\end{subarray}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\!\left(\rho_{i,t}(\theta)\hat{A}_{i},\;\mathrm{clip}(\rho_{i,t}(\theta),1\pm\epsilon)\hat{A}_{i}\right)\right] (7) Thus GRPO preserves token-level policy ratios as in PPO, while using sequence-level, group-normalized advantages to stabilize learning under sparse rewards.

##### Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)

DAPO ( Yu et al., 2025 ) extends GRPO with two key modifications. First, it replaces symmetric clipping with asymmetric clipping , using distinct thresholds for upward and downward deviations: clip asym  ( ρ , a ) = clip ⁡ ( ρ , 1 − ϵ − , 1 + ϵ + ) \mathrm{clip}_{\mathrm{asym}}(\rho,a)=\mathrm{clip}(\rho,\,1-\epsilon^{-},1+\epsilon^{+}) , where ϵ − \epsilon^{-} and ϵ + \epsilon^{+} are hyper-parameters.

Second, DAPO changes the aggregation scheme to operate at the prompt level . For a given prompt x ∼ D x\sim D , the old policy produces G G completions { y i } i = 1 G \{y_{i}\}_{i=1}^{G} with advantages { A ^ i } \{\hat{A}_{i}\} ( Equation 5 ). Let T = ∑ i = 1 G | y i | T=\sum_{i=1}^{G}|y_{i}| denote the total number of tokens across all completions. With token-level ratios as in Equation 2 . The DAPO surrogate objective is 𝒥 DAPO  ( θ ) = 𝔼 x ∼ D , { y i } i = 1 G ∼ π gen ( ⋅ ∣ x , θ old )  [ 1 T  ∑ i = 1 G ∑ t = 1 | y i | min ⁡ ( ρ i , t  ( θ )  A ^ i , clip asym  ( ρ i , t  ( θ ) )  A ^ i ) ] . \mathcal{J}_{\mathrm{DAPO}}(\theta)=\mathbb{E}_{\begin{subarray}{l}\,x\sim D,\\ \{y_{i}\}_{i=1}^{G}\sim\pi_{\text{gen}(\cdot\mid x,\theta_{\text{old}})}\end{subarray}}\left[\frac{1}{T}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}\min\!\left(\rho_{i,t}(\theta)\hat{A}_{i},\;\mathrm{clip}_{\mathrm{asym}}(\rho_{i,t}(\theta))\hat{A}_{i}\right)\right]. (8) This prompt-level normalization ensures that each token contributes equally to the prompt’s loss, regardless of the number or length of its sampled completions. DAPO also introduces dynamically dropping 0-variance prompts from the batch during training and filling the batch with more prompts until the batch is full. We skip that change here since its effect is similar to having a larger batch size.

### A.3 Training Setup

##### Datasets

For small-scale SFT, we use a curated data mix of reasoning traces. We filter this dataset by removing trivial prompts, discarding solution traces exceeding 12  k 12k tokens, and decontaminating with AIME 2024/2025 ( AoPS, 2025 ) and MATH-500 ( Hendrycks et al., 2021 ) benchmarks. For the RL stage, we use the Polaris-53K dataset ( An et al., 2025 ) for most of our runs; additionally using the Deepcoder dataset ( Luo et al., 2025 ) for runs with both math and code.

##### Supervised Fine-tuning

We run SFT using a batch size of 2M tokens, max sequence length of 12288, and a learning rate of 3 × 10 − 5 3\times 10^{-5} using the AdamW optimizer ( Loshchilov & Hutter, 2019 ) on 32 H100 GPU nodes for approximately 4 epochs and 32B tokens in total.

##### Reinforcement Learning