The Art of Scaling Reinforcement Learning Compute for LLMs — part 1/5

Cleaned-text code points [0, 15808). Bibliography omitted here; source.md includes it.

# The Art of Scaling Reinforcement Learning Compute for LLMs

###### Abstract

Reinforcement learning (RL) has become central to training large language models (LLMs), yet the field lacks predictive scaling methodologies comparable to those established for pre-training. Despite rapidly rising compute budgets, there is no principled understanding of how to evaluate algorithmic improvements for scaling RL compute. We present the first large-scale systematic study, amounting to more than 400,000 GPU-hours, that defines a principled framework for analyzing and predicting RL scaling in LLMs. We fit sigmoidal compute-performance curves for RL training and ablate a wide range of common design choices to analyze their effects on asymptotic performance and compute efficiency. We observe: (1) Not all recipes yield similar asymptotic performance, (2) Details such as loss aggregation, normalization, curriculum, and off-policy algorithm primarily modulate compute efficiency without materially shifting the asymptote, and (3) Stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller-scale runs. Combining these insights, we propose a best-practice recipe, ScaleRL , and demonstrate its effectiveness by successfully scaling and predicting validation performance on a single RL run scaled up to 100,000 GPU-hours. Our work provides both a scientific framework for analyzing scaling in RL and a practical recipe that brings RL training closer to the predictability long achieved in pre-training.

## 1 Introduction

Scaling reinforcement learning (RL) compute is emerging as a critical paradigm for advancing large language models (LLMs). While pre-training establishes the foundations of a model; the subsequent phase of RL training unlocks many of today’s most important LLM capabilities, from test-time thinking ( OpenAI, 2024 ; Guo et al., 2025 ) to agentic capabilities ( Kimi Team et al., 2025a ) . For instance, Deepseek-R1-Zero used 100,000 H800 GPU hours for RL training – 3.75% of its pre-training compute ( Guo et al., 2025 ) . This dramatic increase in RL compute is amplified across frontier LLM generations, with more than 10 × \times increase from o1 to o3 ( OpenAI, 2025 ) and a similar leap from Grok-3 to Grok-4 ( xAI Team, 2025 ) .

While RL compute for LLMs has scaled massively, our understanding of how to scale RL has not kept pace; the methodology remains more art than science. Recent breakthroughs in RL are largely driven by isolated studies on novel algorithms (e.g., Yu et al. (2025, DAPO,) ) and model-specific training reports, such as, MiniMax et al. (2025) and Magistral ( Rastogi et al., 2025 ) . Critically, these studies provide ad-hoc solutions tailored to specific contexts, but not how to develop RL methods that scale with compute. This lack of scaling methodology stifles research progress: with no reliable way to identify promising RL candidates a priori , progress is tied to large-scale experimentation that sidelines most of the academic community.

This work lays the groundwork for science of RL scaling by borrowing from the well-established concept of scaling laws from pre-training. While pre-training has converged to algorithmic recipes that scale predictably with compute ( Kaplan et al., 2020 ; Hoffmann et al., 2022 ; Owen, 2024 ) , the RL landscape lacks a clear standard. As a result, RL practitioners face an overwhelming array of design choices, leaving the fundamental questions of how to scale and what to scale unanswered. To address these questions, we establish a predictive framework for RL performance using a sigmoid-like saturating curve between the expected reward ( R C R_{C} ) on an iid validation set and training compute ( C C ): R C − R 0 ⏞ Reward Gain = ( A − R 0 ) ⏞ Asymptotic Reward Gain × 1 1 + ( C mid / C ) B ⏟ Compute Efficiency \displaystyle\boxed{\overbrace{R_{C}-R_{0}}^{\text{Reward Gain}}=\overbrace{(A-R_{0})}^{\text{Asymptotic Reward Gain}}\times\frac{1}{\underbrace{1+(C_{\text{mid}}/C)^{B}}_{\text{Compute Efficiency}}}} (fixed model and training data) (1)

where 0 ≤ A ≤ 1 0\leq A\leq 1 represents the asymptotic pass rate, B > 0 B>0 is a scaling exponent that determines the compute efficiency, and C mid C_{\text{mid}} sets the midpoint of the RL performance curve. A schematic interpretation of these parameters is provided in Figure 3 .

This framework in Equation 1 allows researchers to extrapolate performance from lower-compute runs to higher compute budgets, enabling them to evaluate scalability of RL methods without incurring the compute cost of running every experiment to its computational limit.

Guided by this framework, we develop ScaleRL , an RL recipe that scales predictably with compute. In a massive 100,000 GPU-hours training run , we show that ScaleRL ’s performance closely matches the scaling curve predicted by our framework (Figure 1 ). Critically, scaling curves extrapolated from only the initial stages of training closely match the final observed performance, confirming the predictive ability of our framework to extreme compute scales.

The design of ScaleRL is grounded in a comprehensive empirical study of RL scaling that spanned over 400,000 GPU-hours (on Nvidia GB200 GPUs). This study explored numerous design choices at an 8B model parameters scale, where individual runs use up to 16,000 GPU-hours, making them 6 × \times cheaper than experimenting at our largest training run scale. This investigation yielded three key principles: • RL Performance Ceilings are Not Universal : As we scale training compute for different methods, they encounter different ceilings on their achievable performance ( A A ). This limit can be shifted by choices such as the loss type and batch size.

• Embracing the Bitter Lesson : Methods that appear superior at small compute budgets can be worse when extrapolated to large-compute regimes ( Figure 2 ). We can still identify scalable methods by estimating the scaling parameters ( A A , B B ) from the early training dynamics using our framework ( Equation 1 ).

• Re-evaluating Common Wisdom : Common interventions thought to improve peak performance (e.g., loss aggregation, data curriculum, length penalty, advantage normalization) mainly adjust compute efficiency ( B B ), while not changing the performance ceiling considerably.

Based on these insights, ScaleRL achieves predictable scaling by integrating existing methods, rather than inventing novel methods. Specifically, ScaleRL combines asynchronous Pipeline-RL setup (§ 3.1 ), forced length interruptions, truncated importance sampling RL loss (CISPO), prompt-level loss averaging, batch-level advantage normalization, FP32 precision at logits, zero-variance filtering, and No-Positive-Resampling – with each component’s contribution validated in a leave-one-out ablation, consuming 16,000 GPU-hours per run.

ScaleRL not only scales predictably but also establishes a new state-of-the-art ( Figure 2 ) – it achieves higher asymptotic performance and compute efficiency compared to established RL recipes. Moreover, ScaleRL maintains predictable scaling when increasing compute across multiple training axes (§ 5 ) – including 2.5 × \times larger batch sizes, longer generation lengths up to 32,768 tokens, multi-task RL using math and code, and larger MoE (Llama-4 17B × 16 \times 16 ); with benefits that consistently transfer to downstream tasks. Overall, this work establishes a rigorous methodology for cost-effectively predicting the scalability of new RL algorithms.

## 2 Preliminaries & Setup

We consider reinforcement learning with LLMs, where prompts x x are sampled from a data distribution D D . Our setup follows a generator–trainer split across GPUs: a subset of GPUs ( generators ) use optimized inference kernels for high-throughput rollout generation, while the remaining GPUs ( trainers ) run the training backend (FSDP) and update parameters. We denote by π gen θ \pi_{\text{gen}}^{\theta} and π train θ \pi_{\text{train}}^{\theta} the model with parameters θ \theta on the generator and training backends, respectively. For each prompt, the old policy π g ​ e ​ n θ o ​ l ​ d \pi_{gen}^{\theta_{old}} on the generator GPUs produces candidate completions, which are then assigned scalar rewards. Policy optimization proceeds by maximizing a clipped surrogate objective, taking expectations over x ∼ D x\sim D and rollouts from π g ​ e ​ n θ o ​ l ​ d \pi_{gen}^{\theta_{old}} .

##### Training Regimen

All experiments are conducted on the RL for reasoning domain, where the model produces a thinking trace enclosed with special tokens ( <think> … \ldots </think> ) and a final solution. Unless noted, training uses a sequence length of 16,384 16,384 tokens: 12,288 12,288 for thinking, 2,048 2,048 for the solution, and an additional 2,048 2,048 for the input prompt. We adopt the 12,288 12,288 thinking budget for faster iteration, and show in Section 5 that ScaleRL extrapolations remain predictive when training with larger thinking budgets ( 32,768 32,768 ). For math RL experiments, we use the Polaris-53K dataset ( An et al., 2025 ) with a batch size of 768 (48 prompts with 16 generations each). In our setup, scaling RL compute corresponds to running multiple epochs over the training prompts. More details about training, including SFT and hyper-parameters, are in Appendix A.3 .

##### Base RL Algorithm

As our starting point in § 3 , we start with a “base” algorithm that resembles GRPO ( Shao et al., 2024 ) without any KL regularization term, in line with large-scale training reports ( Rastogi et al., 2025 ; MiniMax et al., 2025 ) . Additionally, we include the asymmetric DAPO clipping ( Yu et al., 2025 ) , because of its widespread adoption as a default approach to avoid entropy collapse and maintain output diversity.

For a given prompt x x , the old policy π gen ​ ( θ old ) \pi_{\text{gen}}(\theta_{\text{old}}) generates G G candidate completions { y i } i = 1 G \{y_{i}\}_{i=1}^{G} , each assigned a scalar reward r i r_{i} . We compute advantages A ^ i \hat{A}_{i} and group-normalized advantages using: A ^ i = r i − mean ⁡ ( { r j } j = 1 G ) , A ^ i G = A ^ i / ( std ⁡ ( { r j } j = 1 G ) + ϵ ) . \hat{A}_{i}=r_{i}-\mathrm{mean}(\{r_{j}\}_{j=1}^{G}),\quad\hat{A}_{i}^{G}=\hat{A}_{i}/(\mathrm{std}(\{r_{j}\}_{j=1}^{G})+\epsilon). Each completion y i y_{i} of length | y i | |y_{i}| contributes at the token-level importance sampling (IS) ratios ρ i , t ​ ( θ ) \rho_{i,t}(\theta) , with asymmetric upper and lower clipping thresholds, akin to DAPO ( Yu et al., 2025 ) : ρ i , t ​ ( θ ) := π t ​ r ​ a ​ i ​ n θ ​ ( y i , t ∣ x , y i , < t ) π g ​ e ​ n θ old ​ ( y i , t ∣ x , y i , < t ) = π t ​ r ​ a ​ i ​ n θ ​ ( y i , t ) π g ​ e ​ n θ old ​ ( y i , t ) ; clip asym ​ ( ρ , ϵ − , ϵ + ) := clip ⁡ ( ρ , 1 − ϵ − , 1 + ϵ + ) . \rho_{i,t}(\theta):=\frac{\pi^{\theta}_{train}(y_{i,t}\mid x,y_{i,<t})}{\pi^{\theta_{\text{old}}}_{gen}(y_{i,t}\mid x,y_{i,<t})}=\frac{\pi^{\theta}_{train}(y_{i,t})}{\pi^{\theta_{\text{old}}}_{gen}(y_{i,t})};\quad\mathrm{clip}_{\mathrm{asym}}(\rho,\epsilon^{-},\epsilon^{+}):=\mathrm{clip}(\rho,\,1-\epsilon^{-},1+\epsilon^{+}). (2)

We aggregate losses at the sample level , i.e., averaging per-sample token losses before averaging across samples. The surrogate objective is given by: 𝒥 ⁡ ( θ ) = 𝔼 x ∼ D , { y i } i = 1 G ∼ π gen θ old ( ⋅ ∣ x ) ​ [ 1 G ​ ∑ i = 1 G 1 | y i | ​ ∑ t = 1 | y i | min ⁡ ( ρ i , t ​ ( θ ) ​ A ^ i G , clip asym ​ ( ρ i , t ​ ( θ ) , ϵ − , ϵ + ) ​ A ^ i G ) ] . \mathcal{J}(\theta)=\mathbb{E}_{\begin{subarray}{l}\,x\sim D,\\ \{y_{i}\}_{i=1}^{G}\sim{\pi_{\text{gen}}^{\theta_{\text{old}}}(\cdot\mid x)}\end{subarray}}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\!\left(\rho_{i,t}(\theta)\hat{A}_{i}^{G},\;\mathrm{clip_{asym}}(\rho_{i,t}(\theta),\epsilon^{-},\epsilon^{+})\hat{A}_{i}^{G}\right)\right]. (3)

##### Controlling Generation Lengths

To prevent reasoning output lengths from exploding during training, which harms training stability and efficiency, we use interruptions ( GLM-V Team et al., 2025 ; Yang et al., 2025 ) that forcibly stop overly long generations by appending an end-of-thinking phrase (e.g., “ </think> ”), signaling the LLM to terminate its reasoning and produce a final answer. We revisit this choice later in Section 4 and compare it with length-penalty that penalizes long generations ( Yu et al., 2025 ; Kimi Team et al., 2025b ) .

### 2.1 Predictive compute-scaling and fitting curves

Unlike pre-training, which typically uses power-law to fit predictive curves, we model pass rate versus log ⁡ ( compute ) \log(\text{\emph{compute}}) with a sigmoidal function ( Equation 1 ). We do so because we found the sigmoidal fit to be much more robust and stable compared to power law empirically, which we discuss further in Appendix A.4 . Moreover, our choice is consistent with prior work that use sigmoid-like power laws to capture bounded metrics such as accuracy ( Ruan et al., 2024 ; Srivastava et al., 2022 ) .

Similar to pre-training studies ( Li et al., 2025b ; Porian et al., 2025 ) , we find that excluding the very early low-compute regime yields more stable fits, after which training follows a predictable trajectory. Unless noted otherwise, all our scaling fits begin after ∼ \sim 1.5k GPU hours. Further details of the fitting procedure are provided in Appendix A.5 and the robustness of our curve fitting is discussed in Appendix A.7 .

##### Interpreting scaling curves

Intuitively, a sigmoidal curve captures saturating returns - grows slowly in the low-compute regime, accelerates sharply through a mid-range of efficient scaling, and then saturates at high compute. We also provide a schematic interpretation of the parameters A , B , and ​ C m ​ i ​ d A,B,\text{ and }C_{mid} of the sigmoidal curve in Figure 3 . We see that B , C mid B,C_{\text{mid}} primarily affects the efficiency of the run, and A A denotes the asymptotic performance at large compute scale. Further discussion of these parameters is provided in Appendix A.8 .

##### Scaling curve on held-out validation

Consistent with pre-training practice ( Hoffmann et al., 2022 ; Porian et al., 2025 ) , we measure predictive performance on in-distribution validation data. Since our training runs span multiple epochs, we hold out randomly selected 1,000 1,000 prompts from the Polaris-53k dataset for validation and use the remainder for training. The scaling curves are fitted on the validation points, which measure the average pass rate every 100 100 training steps, with 16 16 generations per prompt on the 1,000 1,000 held-out prompts.

## 3 An Empirical Study of RL Scaling

In this section, we conduct RL experiments using an 8B dense model on verifiable math problems. Using the setup described in Section 2 , we study several design axes in terms of their predictable compute-scaling behavior, namely asymptotic performance ( A A ) and compute efficiency ( B B ), as shown in Figure 3 .

We structure our experiments in three stages – we first ablate design choices on top of the baseline at 3.5 ​ k 3.5k to 4 ​ k 4k GPU-hours since some experimental choices destabilize beyond this scale (Appendix A.15 ). Whenever a design change proved stable, we trained it for longer. Then, we combine the best choices into ScaleRL and run leave-one-out (LOO) experiments for 16 ​ k 16k GPU-hours in Section 4 . Here, we assess predictability by fitting on the first 8 ​ k 8k GPU-hours and extrapolating the remainder of the run. Finally, to demonstrate predictable scaling with ScaleRL , we also consider training setups with larger batch sizes, mixture-of-experts model, multiple tasks (math and code), and longer sequence lengths in Section 5 .

### 3.1 Asynchronous RL Setup