The Art of Scaling Reinforcement Learning Compute for LLMs — part 2/5

Cleaned-text code points [15808, 31574). Bibliography omitted here; source.md includes it.

We first investigate the choice of asynchronous off-policy RL setup ( Noukhovitch et al., 2024 ) , as it governs training stability and efficiency, generally independent of all other design choices. Specifically, we consider two approaches for off-policy learning: PPO-off-policy- k k and PipelineRL- k k .

PPO-off-policy- k k is the default approach for asynchronous RL and has been used previously by Qwen3 ( Yang et al., 2025 ) and ProRL ( Liu et al., 2025a ) . In this setup, the old policy π g ​ e ​ n θ old \pi_{gen}^{\theta_{\text{old}}} generates reasoning traces for a batch of B B prompts. Each gradient update processes a mini-batch of B ^ \hat{B} prompts, resulting in k = B / B ^ k=B/\hat{B} gradient updates per batch. In our experiments, we fix B ^ = 48 \hat{B}=48 prompts (with 16 generations each), and vary k ∈ { 1 , 8 } k\in\{1,8\} by setting B = k × 48 B=k\times 48 .

PipelineRL- k k is a recent approach from Piche et al. (2025) and used by Magistral ( Rastogi et al., 2025 ) . In this regimen, generators continuously produce reasoning traces in a streaming fashion. Whenever trainers finish a policy update, the new parameters are immediately pushed to the generators, which continue generating with the updated weights but a stale KV cache from the old policy. Once a full batch of traces is generated, it is passed to the trainers for the next update. In our setup we introduce a parameter k k : the trainers wait if they get k k steps ahead of the generators.

We compare these approaches in Figure 4a . PipelineRL and PPO-off-policy achieve similar asymptotic performance A A , but PipelineRL substantially improves the compute efficiency B B ; thus reaching the ceiling A A faster. This is because PipelineRL reduces the amount of idle time in the training process. This choice yields reliable gains with fewer tokens, making larger sweeps at a lower compute budget possible. We also vary the maximum off-policyness for PipelineRL and find k = 8 k=8 to be optimal as shown in Figure 4b , which we discuss further in Appendix A.11 .

### 3.2 Algorithmic Choices

Building on the results above, we adopt PipelineRL- 8 8 as our updated baseline. We then study six additional algorithmic axes: (a) loss aggregation, (b) advantage normalization, (c) precision fixes, (d) data curriculum, (e) batch definition, and (f) loss type. In Section 4 , we combine the best options into a unified recipe, termed ScaleRL ( Scale -able RL ), and conduct leave-one-out experiments on a larger scale of 16,000 16,000 GPU-Hours.

##### Loss type

We compare the asymmetric DAPO loss (Eq. 8 ) with two recently proposed alternatives: GSPO ( Zheng et al., 2025a ) and CISPO ( MiniMax et al., 2025 ; Yao et al., 2025 ) . GSPO applies importance sampling at the sequence level as opposed to GRPO’s token-level formulation. Specifically, GSPO alters the token-level IS ratio (Eq. 2 ) to sequence-level ratios: ρ i ​ ( θ ) = π t ​ r ​ a ​ i ​ n ​ ( y i | x , θ ) / π g ​ e ​ n ​ ( y i | x , θ old ) \rho_{i}(\theta)=\nicefrac{{\pi_{train}(y_{i}|x,\theta)}}{{\pi_{gen}(y_{i}|x,\theta_{\text{old}})}} . CISPO simply combines truncated IS with vanilla policy gradient ( Ionides, 2008 ) , where sg is the stop-gradient function: 𝒥 CISPO ​ ( θ ) = 𝔼 x ∼ D , { y i } i = 1 G ∼ π g ​ e ​ n ( ⋅ ∣ x , θ old ) ​ [ 1 T ​ ∑ i = 1 G ∑ t = 1 | y i | sg ​ ( min ⁡ ( ρ i , t , ϵ max ) ) ​ A ^ i ​ log ⁡ ( π t ​ r ​ a ​ i ​ n ​ ( y i , t | x , y i < t , θ ) ) ] \hskip-6.99997pt\mathcal{J}_{\mathrm{CISPO}}(\theta)=\hskip-6.99997pt\underset{\begin{subarray}{c}x\sim D,\\ \{y_{i}\}_{i=1}^{G}\sim\pi_{gen}(\cdot\mid x,\theta_{\text{old}})\end{subarray}}{\mathbb{E}}\hskip-3.99994pt\left[\frac{1}{T}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}\texttt{sg}(\mathrm{min}(\rho_{i,t},\epsilon_{\text{max}}))\hat{A}_{i}\,\log\left(\pi_{train}(y_{i,t}|x,y_{i<t},\theta)\right)\right] (4)

Figure 5a shows that both GSPO and CISPO substantially outperform DAPO, improving the asymptotic pass-rate A A by a large margin. CISPO exhibits a prolonged near-linear reward increase, and is marginally better than GSPO later in training, so we opt for CISPO as our best loss type. Further discussion on off-policy loss types, and their hyperparameter robustness is detailed in Section 4 and Appendix A.17 .

##### FP32 Precision for LLM logits

The generators and trainers rely on different kernels for inference and training, leading to small numerical mismatches in their token probabilities ( He & Lab, 2025 ) . RL training is highly sensitive to such discrepancies, since they directly affect the IS ratio in the surrogate objective. MiniMax et al. (2025) identified that these mismatches are especially pronounced at the language model head, and mitigate this by FP32 computations at the head for both the generator and trainer. As shown in Figure 5b , the precision fix dramatically improves the asymptotic performance A A from 0.52 0.52 to 0.61 0.61 . Given this clear benefit, we include the FP32 precision fix in our ScaleRL recipe.

##### Loss Aggregation

We evaluate three strategies for aggregating the RL loss: (a) Sample average where each rollout contributes equally (as in GRPO, Appendix A.2 ). (b) Prompt average where each prompt contributes equally (as in DAPO, Appendix A.2 ). (c) Token average where all token losses in the batch are averaged directly, without intermediate grouping. The comparison results are shown in Appendix A.9 ( Figure 14a ). We find prompt-average achieves the highest asymptotic performance and therefore use this choice for ScaleRL .

##### Advantage Normalization

We compare three variants of advantage normalization: (a) Prompt level where advantages are normalized by the standard deviation of rewards from the rollouts of the same prompt (as in GRPO, Appendix A.2 ). (b) Batch level where advantages are normalized by the standard deviation across all generations in the batch, as used by Hu et al. (2025a) ; Rastogi et al. (2025) . (c) No normalization where advantages are computed as raw rewards centered by the mean reward of the prompt’s generations, without variance scaling (as proposed in Dr. GRPO ( Liu et al., 2025b ) ). A comparison plot is shown in Appendix A.9 ( Figure 14b ), and all three methods are oberseved to yield similar performance. We therefore adopt batch-level normalization as it is theoretically sound and marginally better. This choice is also further corroborated at a larger scale by the leave-one-out experiments in Section 4 .

##### Zero-Variance Filtering

Within each batch, some prompts yield identical rewards across all their generations. These “zero-variance” prompts have zero advantage and therefore contribute zero policy gradient. The default baseline includes such prompts in loss computation, but it is unclear whether they should be included in the effective batch. To test this, we compare the default setting against an effective batch approach, where only prompts with non-zero variance are included in the loss calculation, as done by Seed et al. (2025) . Note that zero-variance filtering differs from dynamic sampling in DAPO ( Yu et al., 2025 ) . The former merely drop the prompts, while latter resamples more prompts until the batch is full. We show in Figure 6a that using the effective batch performs better asymptotically; and we adopt it in our ScaleRL recipe.

##### Adaptive Prompt Filtering

A number of data curriculum strategies have been proposed for RL training to improve sample efficiency ( An et al., 2025 ; Zhang et al., 2025b ; Zheng et al., 2025b ) . Here we evaluate a simple variant, introduced by An et al. (2025) , with the key observation that once a prompt becomes too easy for a policy, it typically remains easy. Since such prompts consume some compute but no longer contribute useful gradient signal ( Section 3.2 ), it is better to exclude them from future training. We implement this by maintaining a history of pass rates and permanently removing any prompt with pass rate ≥ 0.9 \geq 0.9 from subsequent epochs–we call this No-Positive-Resampling . In Figure 6b we compare this curriculum against the default setting where all prompts are resampled uniformly throughout training. We see that the curriculum improves scalability and the asymptotic reward A A .

## 4 ScaleRL : Scaling RL Compute Effectively & Predictably

From the design axes studied above, we consolidate the best-performing settings into a single recipe, which we term ScaleRL ( Scale -able RL ). ScaleRL is an asynchronous RL recipe that uses PipelineRL with 8 8 steps off-policyness , interruption-based length control for truncation, FP32 computation for logits , and optimizes the 𝒥 ScaleRL ​ ( θ ) \mathbf{\mathcal{J}_{\mathrm{\texttt{{ScaleRL}}}}(\theta)} loss . This loss combines prompt-level loss aggregation, batch-level advantage normalization, truncated importance-sampling REINFORCE loss (CISPO) , zero-variance filtering, and no-positive resampling: 𝒥 ScaleRL ​ ( θ ) = 𝔼 x ∼ D , { y i } i = 1 G ∼ π g ​ e ​ n θ o ​ l ​ d ( ⋅ ∣ x ) ​ [ 1 ∑ g = 1 G | y g | ​ ∑ i = 1 G ∑ t = 1 | y i | sg ​ ( min ⁡ ( ρ i , t , ϵ ) ) ​ A ^ i norm ​ log ⁡ π t ​ r ​ a ​ i ​ n θ ​ ( y i , t ) ] , \displaystyle\mathcal{J}_{\mathrm{\texttt{{ScaleRL}}}}(\theta)=\hskip-6.99997pt\underset{\begin{subarray}{c}x\sim D,\\ \{y_{i}\}_{i=1}^{G}\sim\pi_{gen}^{\theta_{old}}(\cdot\mid x)\end{subarray}}{\mathbb{E}}\hskip-3.99994pt\left[\frac{1}{\sum_{g=1}^{G}|y_{g}|}\sum_{i=1}^{G}\sum_{t=1}^{|y_{i}|}\texttt{sg}(\mathrm{min}(\rho_{i,t},\epsilon))\hat{A}_{i}^{\text{norm}}\,\log\pi_{train}^{\theta}(y_{i,t})\right], ρ i , t = π t ​ r ​ a ​ i ​ n θ ​ ( y i , t ) π g ​ e ​ n θ o ​ l ​ d ​ ( y i , t ) , A ^ i norm = A ^ i / A ^ std , 0 < mean ⁡ ( { r j } j = 1 G ) < 1 , pass_rate ​ ( x ) < 0.9 , \displaystyle\quad\rho_{i,t}=\frac{\pi_{train}^{\theta}(y_{i,t})}{\pi_{gen}^{\theta_{old}}(y_{i,t})},\ \hat{A}^{\mathrm{norm}}_{i}=\hat{A}_{i}/\hat{A}_{\text{std}},\ 0<\mathrm{mean}(\{r_{j}\}_{j=1}^{G})<1,\ \text{pass\_rate}(x)<0.9,\ where sg is the stop-gradient function, A ^ std \hat{A}_{\text{std}} is the standard deviation of all advantages A ^ i \hat{A}_{i} in a batch and pass_rate ​ ( x ) \text{pass\_rate}(x) denotes the historical pass rate of a prompt x x . For forced interruptions, we use the end-of-thinking phrase: “Okay, time is up. Let me stop thinking and formulate a final answer now. </think> ”.

##### Leave-One-Out (LOO) Ablations

To validate that these choices remain optimal when combined, we conduct leave-one-out (LOO) experiments: starting from ScaleRL , we revert one axis at a time to its baseline counterpart from Section 2 . This ensures that each design decision contributes positively even in the presence of all others. Figure 7 reports these experiments, each scaled to 16 16 k GPU hours.

Across all axes, ScaleRL consistently remains the most effective configuration, slightly outperforming LOO variants either in asymptotic reward or in compute efficiency (refer to the last column in the Figure 7 table). Since most LOO variants reach similar asymptotic pass rates, we transform the sigmoidal fit to a power-law fit, to highlight efficiency differences via the slope B B (details in Figure 7 ). Concretely, we average the asymptotic reward A A across all runs, re-fit the curves with this fixed A A , and then compare slopes (measuring efficiency) in Figure 7 . The corresponding non-transformed pass-rate vs. compute curves are provided in Appendix A.2 .

##### Error margin in fitting scaling curves

Since RL training is known to exhibit high variance ( Agarwal et al., 2021 ) , we use three independent ScaleRL runs ( Figure 8a ) to estimate the variability in fitted scaling coefficients. The observed variance in asymptotic reward and efficiency parameters serves as our empirical error margin, used to determine whether changes in compute efficiency or asymptotic performance are statistically meaningful for two different runs ( Madaan et al., 2024 ) .

##### Extrapolating Scaling Curves

In all our LOO experiments as well as independent ScaleRL runs, we fit the sigmoidal curve up to 8000 GPU-hours and extrapolate to 16000 GPU-hours, observing that the predicted curves align closely with both training and extended points. This demonstrates the stability and predictability of ScaleRL and other stable, scalable recipes under large-scale RL training.

##### Are the design choices worth it?

In Section 3.2 , certain design choices alter asymptotic performance, such as loss type ( Figure 5a ) and FP32 precision ( Figure 5b ). However, in our LOO experiments with ScaleRL ( Figure 7 ), these components appear less critical individually (last column in the figure). This raises the question of whether certain design choices can be safely left at their “default” values.

We argue the answer is no. Even when a choice seems redundant in the combined recipe, it can still provide stability or robustness that can become decisive in other regimes. For example, while the FP32 precision fix makes little difference with dense 8B trained with ScaleRL ( Figure 7 ), it provides large gains in GRPO/DAPO-style losses by mitigating numerical instabilities. This indicates that its benefits extend beyond the specific ScaleRL configuration we study. To further test this, we ran a leave-one-out experiment on the Scout 17Bx16 MoE and observed that FP32 precision improves overall scalability ( Figure 8b ).

A similar case arises with the loss type. In Figure 7 , reverting to DAPO yields similar asymptotic performance to CISPO within ScaleRL . Nonetheless, as we discuss in Appendix A.17 , CISPO is markedly more robust to the choice of IS-clipping parameter ϵ max \epsilon_{\max} , reducing the sensitivity of training to hyperparameter tuning. Moreover, it’s also more efficient than DAPO, as seen in LOO experiment ( B = 2.01 B=2.01 vs B = 1.77 B=1.77 ). This justifies preferring CISPO, even if a carefully tuned DAPO variant can perform similar asymptotically.

In summary, even when individual design choices appear redundant within the combined recipe, they often enhance training stability, robustness, or efficiency in ways that generalize across models and setups. ScaleRL retains such components not just for marginal gains in a specific configuration, but because they address recurring sources of instability and variance that arise across reinforcement learning regimes.

## 5 Predictable Scaling Returns Across RL Compute Axes

Given a fixed or growing compute budget, which scaling knob –context length, batch size, generations per prompt, and model size – buys the most reliable performance gain, and how early can we predict that return? We answer this by (i) fitting the saturating power-law in eq. 1 early in training for each setting (precisely, half the target budget), (ii) extrapolating to the target budget, and (iii) extending training to verify the forecast. Across all axes below we observe clean, predictive fits whose extrapolated curves align with the extended trajectories, mirroring the behavior seen in our 100,000 GPU-hour run ( Figure 1 ), and the cross-recipe comparison in Figure 2 .

##### Model scale (MoE)

Does ScaleRL remain predictive and stable on larger models? Training the 17B×16 Llama-4 Scout MoE with ScaleRL exhibits the same predictable scaling behavior as the 8B model, with low truncation rates and no instability pathologies (Appendix A.15 , A.17 ). Figure 1 shows the training curve. The extended points align with the fitted curve, supporting the model-scale invariance of our recipe. Moreover, the larger 17B×16 MoE exhibits much higher asymptotic RL performance than the 8B dense model, outperforming the 8B’s performance using only 1 / 6 1/6 of its RL training compute.

##### Generation length (context budget)