The Art of Scaling Reinforcement Learning Compute for LLMs — part 4/5

Cleaned-text code points [47337, 62524). Bibliography omitted here; source.md includes it.

We allocate 14 14 k generation budget during RL training, where 12 ​ k 12\text{k} tokens are allocated to the intermediate reasoning (“thinking”), followed by 2 ​ k 2\text{k} tokens for the final solution and answer. We sample 48 48 prompts in each batch, each with 16 16 generations per prompt. Thus, we get the total batch size as 768 768 completions per gradient update step. The rewards are given as ± 1 \pm 1 to correct and incorrect traces respectively. We use a constant learning rate of 5 × 10 − 7 5\times 10^{-7} , AdamW optimizer ( Loshchilov & Hutter, 2019 ) with ϵ = 10 − 15 \epsilon=10^{-15} , weight decay of 0.01 (default in AdamW), and a linear warmup of 100 steps. The lower ϵ \epsilon is to avoid gradient clipping (epsilon underflow) ( Wortsman et al., 2023 ) .

We use automated checkers like Sympy ( Meurer et al., 2017 ) or Math-Verify 1 1 1 https://github.com/huggingface/Math-Verify for assessing the correctness of the final answer for math problems after stripping out the thinking trace ( <think> ⋯ </think> \texttt{<think>}\cdots\texttt{</think>} ). We use a custom code execution environment for coding problems involving unit tests and desired outputs.

We used 80 Nvidia GB200 GPU for a single run, with a compute budget ranging from 3.5-4K GPU hours for establishing different design choices in Section 3.2 , 16K for the leave-one-out experiments ( Section 4 ), and finally 30k-100K GPU hours for our larger scale runs ( Section 5 ). We adopt a generator–trainer split between GPUs. For 80 GPU experiments, we set 64 64 of those as generators , responsible for the generation of reasoning trace using the optimized inference codebase. The remaining 16 GPUs act as trainers , which receive generated trajectories, perform policy updates, and periodically broadcast updated parameters back to the generators.

### A.4 What curve to fit?

Pre-training curves are usually fit using power-law equation ( Li et al., 2025b ; Kaplan et al., 2020 ; Muennighoff et al., 2025 ) , which in our case would model performance as R C = A − D / C B , C ≥ C 0 R_{C}=A-D/C^{B},C\geq C_{0} , where D D is a constant, and C 0 C_{0} marks the compute threshold beyond which the law holds. Intuitively, this implies that each multiplicative increase in compute yields a constant proportional gain in performance. For RL post-training, however, we find a sigmoidal fit ( eq. 1 ) more appropriate for several reasons. First, for bounded metrics such as accuracy or reward, sigmoidal curves provide better predictive fits ( Ruan et al., 2024 ; Srivastava et al., 2022 ) ; we observe the same, with accurate extrapolation to higher compute ( Figure 1 ). Second, power laws are unbounded at low compute and are typically fit only beyond a threshold C 0 C_{0} . In RL, where total training spans far fewer steps (e.g., only ∼ \sim 75 evaluation points to fit only in Figure 1 ), discarding early points a lot further reduces the already limited data available for fitting. Third, empirically, sigmoidal fits are substantially more robust and stable than power-law fits. Concretely, consider the 100 ​ k 100\text{k} GPU-hour run on the 8B dense model shown in Figure 1 . When we fit a power-law curve between 1.5 ​ k 1.5\text{k} – 50 ​ k 50\text{k} GPU hours, it predicts an asymptotic performance of A = 1.0 A=1.0 , which is clearly incorrect - the actual curve saturates near 0.65 0.65 . In contrast, the sigmoidal fit yields an accurate prediction of A = 0.645 A=0.645 . Moreover, the power-law fit is highly sensitive to the chosen fitting regime: fitting over ( 5 ​ k , 50 ​ k ) (5\text{k},50\text{k}) GPU hours instead gives A = 0.74 A=0.74 , while the sigmoidal fit remains robust and still predicts A = 0.645 A=0.645 . Power-law models only recover the correct asymptote when fitted exclusively in the high-compute regime (e.g., 30 ​ k 30\text{k} – 60 ​ k 60\text{k} GPU hours). However, our goal is to predict large-scale performance from lower-compute regimes, where such long runs are unavailable.

Given these considerations, we use the sigmoidal form throughout our analysis. Intuitively, a sigmoidal curve captures saturating returns - grows slowly in the low-compute regime, accelerates sharply through a mid-range of efficient scaling, and then saturates at high compute as it approaches a finite performance ceiling.

One thing to note is that at high compute regime, sigmoidal curve behaves same as power-law. Concretely, we can have the following approximation of sigmoidal curve: R C \displaystyle R_{C} = R 0 + A − R 0 1 + ( C m ​ i ​ d / C ) B \displaystyle=R_{0}+\frac{A-R_{0}}{1+(C_{mid}/C)^{B}} (sigmoidal curve from eq. 1 ) ⟹ R C \displaystyle\implies R_{C} ≈ R 0 + ( A − R 0 ) ​ ( 1 − C m ​ i ​ d B C B ) \displaystyle\approx R_{0}+(A-R_{0})\left(1-\frac{C_{mid}^{B}}{C^{B}}\right) (For C >> C m ​ i ​ d C>>C_{mid} , high compute regime) = A − ( A − R 0 ) ​ C m ​ i ​ d B C B \displaystyle=A-\frac{(A-R_{0})C_{mid}^{B}}{C^{B}} = A − D C B \displaystyle=A-\frac{D}{C^{B}} where above D = ( A − R 0 ) ​ C m ​ i ​ d B D=(A-R_{0})C_{mid}^{B} . And this is the same form of power-law mentioned at the start of this section.

### A.5 Fitting scaling curves

We fit the sigmoid-law equation in Equation 1 to the mean reward on our held-out validation set. This set consists of 1,000 1,000 prompts held out from the Polaris-53k ( An et al., 2025 ) math dataset, with 16 16 generations sampled every evaluation step performed at 100 100 steps intervals.

Directly fitting all three parameters { A , B , C m ​ i ​ d } \{A,B,C_{mid}\} is challenging. Instead, we perform a grid search over A ∈ { 0.450 , 0.455 , 0.460 , … , 0.800 } A\in\{0.450,0.455,0.460,\ldots,0.800\} and C m ​ i ​ d ∈ [ 100 , 40000 ] C_{mid}\in[100,40000] (searching over 100 linearly separated values), and for each candidate A , C m ​ i ​ d A,C_{mid} fit B B . The best fit (measured by sum of squared residuals) across this grid is selected as the final curve. We use SciPy’s curve_fit with default initialization; varying the initialization strategies produced identical results. To enable future studies to fit compute–performance RL scaling curves, we release a minimal code repository at www.devvrit.com/scalerl_curve_fitting .

To estimate the error margin of our fits, we trained three independent ScaleRL runs with a batch size of 768 768 and generation length of 14 ​ k 14\text{k} (as used in Section 4 ), shown in Figure 8a . We found that the fit values of A A varied by at most ± 0.015 \pm 0.015 , suggesting 0.02 0.02 as a reasonable error margin on the estimates of asymptotic performance. Estimating the error margin for the fitted value B B is difficult, as different algorithms with different A A values can have different error margins for B B . However, for the purpose of comparing algorithms, we can safely deduce that if two methods achieve similar A A values (within 0.02 0.02 ), the one with higher B B when a refit is done with the average of A A values is at least as good in terms of efficient scalability.

### A.6 Comparing algorithms

Consistent with observations in large-scale pre-training, where the loss exhibits a sharp initial drop before settling into a predictable power-law decay ( Li et al., 2025b ) , we observe a similar two-phase behavior in RL. The mean reward increases rapidly, almost linearly, during the ∼ \sim first epoch ( ∼ 1 ​ k \sim 1\text{k} steps, or ∼ \sim 1.5k GPU Hours for most runs), after which the curve follows sigmoidal-law behavior (see Figure 15 to see the “sigmoid” like curve). Our sigmoidal-law fits are applied to this latter portion of the training curve.

Unlike pre-training, our main goal is not to predict the performance of a fixed recipe, but to identify which algorithms and design choices scale reliably, and to design algorithm that exhibits predictive nature. Achieving highly robust fits typically requires very large runs with hundreds or thousands of evaluation points, which is impractical in our setting for two reasons. First, running all ablations at such scale would be computationally prohibitive. Second, many RL algorithms we compare are themselves not scalable to such extreme budgets: they often saturate much earlier or even degrade with more compute due to instability. For example, our baseline method ( Section 3.2 ) destabilizes beyond ∼ 3500 \sim 3500 GPU-hours, since overlong generation truncations exceed 10 % 10\% of generations - reducing the effective batch size. More discussion on this is in Section A.15 .

As we ablate across different axes in Section 3.2 , we discover design choices that improve stability at higher compute. Some ablated variants can scale further, e.g., ∼ 5 ​ k \sim 5\text{k} GPU hours for ϵ = 0.26 \epsilon=0.26 in DAPO, ∼ 6 ​ k \sim 6\text{k} GPU hours with the FP32 precision fix ( Section 3.2 ), and ∼ 7 ​ k \sim 7\text{k} GPU hours for CISPO. Once we combine the best design choices, we obtain a stable and scalable recipe, which allows us to run leave-one-out (LOO) experiments for ∼ 1600 \sim 1600 GPU hours per run.

### A.7 Robustness of fits

One may wonder how robust our fitted curves are. We address a few relevant points below:

• For stable and scalable experiments, including all runs from Section 4 onward, changing the fitting regime (e.g., including or excluding the initial 1.5 1.5 k GPU-hour range) yields similar predictable results. For instance, in the 100k GPU-hour run on the 8B dense model, fitting over ( 1.5 ​ k , 50 ​ k ) (1.5\text{k},50\text{k}) gives B = 1.70 B=1.70 , A = 0.645 A=0.645 , while ( 0,100 ​ k ) (0,100\text{k}) gives B = 1.56 B=1.56 , A = 0.655 A=0.655 , ( 0 , 50 ​ k ) (0,50\text{k}) gets B = 1.7 , A = 0.645 B=1.7,A=0.645 , and ( 5 ​ k , 50 ​ k ) (5\text{k},50\text{k}) gives B = 1.67 B=1.67 , A = 0.645 A=0.645 . Across these regimes, parameter values remain within the expected error margin ( Section 7 ).

• We nonetheless skip the low-compute regime because early training phases, especially in less stable setups from Section 3.2 , often plateau prematurely or deviate from the sigmoidal trend due to transient instabilities (see Appendix A.6 , A.15 ). Excluding this region allows the fit to focus on the mid-to-high compute range where saturation behavior is clearer and more consistent.

• The 1.5 1.5 k GPU-hour threshold is a heuristic chosen empirically: it approximately corresponds to one epoch for most experiments in Section 3.2 . Larger cutoffs reduced the number of fitting points, while smaller ones often introduced noise. We found 1.5 1.5 k GPU hours to provide the best balance between fit stability and sample coverage, consistent with practices of skipping low-FLOPs regime in pre-training scaling analyses and fitting ( Li et al., 2025b ) .

### A.8 Interpreting Sigmoidal Curves

Figure 3 presented an example fit illustrating the influence of parameters A A , B B , and C mid C_{\text{mid}} . Here, we extend this with additional illustrations: Figure 12a , Figure 12b , and Figure 13a vary B B , C mid C_{\text{mid}} , and A A respectively, while keeping the other parameters fixed. We observe that B B and C mid C_{\text{mid}} primarily affect the efficiency of scaling, whereas A A determines the asymptotic performance achievable at large compute. In Figure 13b we see a case of two runs where one is much more efficient, hence shows initial promising gains, but converges to a lower asymptote, while the other progresses more slowly yet ultimately surpasses it due to a higher A A . In practice, scaling strategies should prioritize design choices that raise the asymptotic ceiling A A , and only then optimize for efficiency parameters such as B B or C mid C_{\text{mid}} .

### A.9 Forward and LOO Ablations

We show additional results for Section 3.2 in Figures 14a - 14b . We also plot the pass rate vs compute leave one out experiments from Section 4 in Figure 15 .

### A.10 Controlling generation length

One common concern in reasoning RL is to control exploding generation lengths, which harms both training efficiency and stability (Appendix A.15 ). We consider two approaches: (a) interruptions , used in works like GLM-4.1V ( GLM-V Team et al., 2025 ) , and Qwen3 ( Yang et al., 2025 ) and (b) length penalties , used in works like DAPO ( Yu et al., 2025 ) , Kimi ( Kimi Team et al., 2025b ) , Magistral ( Rastogi et al., 2025 ) , and Minimax-M1 ( MiniMax et al., 2025 ) .

Interruptions forcibly stop generation by appending a marker phrase such as “Okay, time is up. Let me stop thinking and formulate a final answer </think> ”, signaling the model to terminate its reasoning and produce a final answer. In our setup, the interruptions tokens are placed randomly in between [ 10 ​ k , 12 ​ k ] [10k,12k] token length, to induce generalization to different generation lengths.

Length penalties instead reshape the reward. Following DAPO ( Yu et al., 2025 ) , we penalize overly long completions with a tolerance interval L cache L_{\text{cache}} : R length ​ ( y ) = c ​ l ​ i ​ p ​ ( L max − | y | L cache − 1 , − 1 , 0 ) R_{\text{length}}(y)=clip\left(\dfrac{L_{\max}-|y|}{L_{\text{cache}}}-1,-1,0\right) (9) This penalty is added only to the correct traces, discouraging excessively long generations. In the length penalty experiment, we set L max = 14 ​ k L_{\max}=14\text{k} tokens and L cache = 2 ​ k L_{\text{cache}}=2\text{k} tokens.

In Section 4 , we compare length penalty and interruption at a scale of 16k GPU-Hours. We find that replacing interruption with length penalty in our final ScaleRL recipe does not improve performance.

### A.11 PipelineRL

Using the baseline setup, we ablated the off-policy parameter in PipelineRL ( Figure 4b ). Both 4 4 and 8 8 off-policyness performed equally well, and we adopt 8 8 as the default setting when updating the baseline in Section 3.1 .

Why does PipelineRL consistently outperform the classic PPO-off-policy approach ( Sections 3.1 and 4 )? We attribute this to its closer alignment with on-policy training. In PPO-off-policy, generation and training proceed in alternating phases: the trainer operates strictly on batches that are as off-policy as the chosen parameter k k , making updates based on stale rollouts. In contrast, PipelineRL operates in a streaming fashion. As soon as a batch is available, it is passed to the trainer; likewise, as soon as a model update is ready, it is shared back to the generators, who immediately use it—including in the continuation of partially generated traces. This tight feedback loop keeps training closer to the on-policy regime, reducing the mismatch between generator and trainer distributions.

Importantly, this distinction affects the asymptotic performance c c of the scaling curve, not just the efficiency exponent b b . Very few axes shift the asymptote in this way, making the choice of off-policy algorithm one of the most consequential design decisions in RL post-training.

### A.12 Entropy Curves: Scaling Batch Size

We tracked entropy on the held-out validation set throughout training. Across all experiments—spanning variations in batch size, number of tasks, generation length, and model scale—we observed a consistent overall decrease in entropy.