The Art of Scaling Reinforcement Learning Compute for LLMs — part 5/5

Cleaned-text code points [62524, 73766). Bibliography omitted here; source.md includes it.

An interesting finding is that entropy may not always offer a predictive insight into the performance, as proposed by some recent works like Cui et al. (2025) . In Figure 16 , we plot entropy for ScaleRL runs with batch sizes 768 768 and 2048 2048 . Despite the 2048-batch size run achieving much stronger downstream performance at every stage ( Figure 10b ), both runs followed nearly identical entropy trajectories per step ( Figure 16 ). This highlights an important point - although entropy is sometimes used as a proxy for exploration, simply maintaining higher entropy does not translate into better generalization. Instead, larger batches reduced effective exploration similar to smaller batches, per step, yet still yielded substantially better performance - underscoring batch size as an important decisive factor.

Overall, our findings suggest that while entropy decreases consistently during training, it is not necessarily a reliable predictor of downstream performance. This observation reinforces the need to focus on algorithmic and scaling choices (e.g., batch size, off-policy method) in adddition to entropy dynamics when aiming for improved performance, both on training distribution as well as downstream task distribution.

### A.13 Scaling on multiple axes

We provide the remaining scaling to different axes figure here in Figure 17 , and the corresponding downstream evaluation in Figure 18 . We also provide the value of A , B , C m ​ i ​ d A,B,C_{mid} in Table 1 .

### A.14 Downstream performance

In Figure 1 , 9 , 10b , and 18 , we report a representative set of downstream evaluation curves. These include ScaleRL runs with batch sizes { 512,768 , 2048 } \{512,768,2048\} , long-context training run with 32 ​ k 32\text{k} generation length, the large-model (Scout) training run, a multi-task run (math + code), and different number of generations per prompt (with fixed batch size) run. For each setting we plot performance against compute. Moreover, we see downstream performance better for experiments like larger batch sizes, longer generation length, and large model size - mirroring similar order for validation set curves.

### A.15 Truncations and training instabilities

Across our experiments we found that training instabilities were often linked to truncations. As generation length grew, many RL runs exhibited fluctuating truncation rates that sometimes increased over training. At batch size 768 768 , we observed that truncations in the range of 10 10 – 15 % 15\% typically destabilized training, with performance degrading and not recovering without intervention. Examples include the extended GRPO run in Figure 2 , where instability correlated with rising truncation rates, and the updated baseline used in Section 3.2 .

By contrast, ScaleRL runs were more stable. On the 8 8 B model, truncations remained below 5 % 5\% for over 90 % 90\% of training. At batch size 2048 2048 , truncations were slightly higher, occasionally approaching ∼ 7 % \sim 7\% . This increase was largely attributable to longer average generation lengths observed during training, which naturally raise the chance of exceeding the budget. Nevertheless, because the effective batch size (after excluding truncated samples) remained large, training stability was preserved. Intuitively, larger generation length budget should help reduce truncations. Training with 34 34 k generation length (batch 768 768 ) remained stable - truncations briefly spiked to ∼ 4 % \sim 4\% but quickly fell below 2 % 2\% .

Larger models were even more robust. On the Scout run, truncations remained consistently below 2 % 2\% , and for > 90 % >90\% of training steps were under 1 % 1\% . This likely reflects both the inherent ability of larger models to regulate generation length and their stronger instruction-following ability, which made interruption signals more effective.

Overall, we suggest practitioners monitor truncation rates closely. Our findings indicate that high truncation rates are a reliable warning signal of instability, while larger models, higher generation budgets, and careful design choices (as in ScaleRL ) substantially mitigate this risk.

### A.16 Comparing Prevalent Methods

In Figure 2 we compared some popular training recipes with ScaleRL . We briefly describe these existing recipes here.

##### DeepSeek (GRPO)

This recipe mostly follows the DeepSeek Guo et al. (2025) work. We use GRPO as the loss function ( Section A.2 ) with ϵ m ​ i ​ n = ϵ m ​ a ​ x = 0.2 \epsilon_{min}=\epsilon_{max}=0.2 , sample average loss aggregation, and PPO-offpolicy-8 algorithm. We saw the training became unstable post 6k GPU Hours due to truncations ( Section A.15 ).

##### Qwen2.5 (DAPO)

This recipe follows DAPO Yu et al. (2025) . It includes the DAPO loss function (Appendix A.2 ) with ϵ m ​ i ​ n = 0.2 , ϵ m ​ a ​ x = 0.26 \epsilon_{min}=0.2,\epsilon_{max}=0.26 (Appendix A.17.1 ). This recipe uses PPO-offpolicy-8, and prompt average loss aggregation. The only change from the original DAPO paper ( Yu et al., 2025 ) was regarding dynamically filling in the batch. Specifically DAPO drops 0-variance prompts and samples more prompts until the batch is full. In our codebase, this was not efficent because for PPO-offpolicy algorithm, we had generators pre-decide that each generator will generate rollouts for # ​ prompts / # ​ generators \nicefrac{{\#\text{prompts}}}{{\#\text{generators}}} . Therefore, if a specific generator had more 0-variance prompts, it sampled further prompts to complete its share of # ​ prompts / # ​ generators \nicefrac{{\#\text{prompts}}}{{\#\text{generators}}} . This could lead to other generators being stalled and an overall slowdown. Hence, to get around this issue, we rather kept a larger batch size of 1280 1280 (80 prompts, 16 generations each), and dropped 0-variance prompts from the batch. We noted that post-dropping, the effective batch was still greater than 768, what we used for ScaleRL . Therefore, if at all, we gave some advantage to the DAPO recipe.

##### Magistral

This refers to the recipe used in Rastogi et al. (2025) . It includes similar recipe as DAPO with the main difference being PipelineRL used as the off-policy algorithm.

##### MiniMax

This refers to the recipe used in MiniMax et al. (2025) . It uses CISPO loss, FP32 precision fix at the LM head, PPO-offpolicy algorithm, and prompt average. Similar to DAPO, it drops 0-variance prompts as well and hence we give it a larger batch size of 1280 1280 as well.

### A.17 Loss Type - Stability and Robustness

As discussed below, GRPO/DAPO-style losses are highly sensitive to the choice of clipping ratio hyperparameter ϵ max \epsilon_{\max} . In contrast, CISPO and GSPO show far greater robustness. For example, in Appendix A.17.2 , varying ϵ max \epsilon_{\max} for CISPO between { 4 , 5 , 8 } \{4,5,8\} produced no significant differences in performance. For GSPO, the 10 − 4 10^{-4} clipping scale used in the original paper ( Zheng et al., 2025a ) did not work well in our setting. We therefore ablated across broader scales and found that once the correct order of magnitude was identified (e.g., × 10 − 3 4\!\times\!10^{-3} and higher), performance was stable and largely insensitive to fine-grained changes (e.g., { × 10 − 3 , × 10 − 3 \{4\!\times\!10^{-3},5\!\times\!10^{-3} ).

#### A.17.1 DAPO clipping ratios

In this section, we analyze the role of the clipping threshold ϵ max \epsilon_{\max} in DAPO Loss Function ( eq. 8 ). The hyper-parameter sensitivity of ϵ m ​ a ​ x \epsilon_{max} has been observed in prior work, for example, GRPO typically sets ϵ max = 0.2 \epsilon_{\max}=0.2 , while DAPO uses 0.28 0.28 . However, beyond tuning sensitivity, we find that ϵ max \epsilon_{\max} directly alters the scaling behavior of the algorithm. As ϵ max \epsilon_{\max} increases, the terminal reward A A increases until an optimal range is reached, after which A A decreases again. This is a striking effect: unlike many hyper-parameters that merely shifts the convergence speed, ϵ max \epsilon_{\max} governs the asymptotic error itself.

#### A.17.2 CISPO Clipping Ratios

We ablate the higher clipping ratio for CISPO, keeping the lower clipping ratio fixed at 0 0 ( Figure 19b ). Across a wide range of values, we find little difference in performance, indicating that CISPO is largely insensitive to this hyperparameter. This robustness mirrors our findings for GSPO ( Section A.17.3 ), and stands in contrast to DAPO/GRPO-style objectives, which are highly sensitive to the exact choice of clipping threshold. Such stability under hyperparameter variation makes CISPO a strong candidate for default use in large-scale training.

#### A.17.3 GSPO ablations

We ablate the clipping-ratio scale used in GSPO, as shown in Figure 20a . The default 10 − 4 10^{-4} scale as given in the GSPO paper Zheng et al. (2025a) does not scale the best for our 8B model. The 10 − 3 10^{-3} scale performs as well as, or better than, alternatives ( Figure 20a ) Given this scale, we further varied the upper clipping ratio in { × 10 − 3 , × 10 − 3 } \{4\!\times\!10^{-3},5\!\times\!10^{-3}\} and found { × 10 − 3 } \{5\!\times\!10^{-3}\} yielded slightly better fit ( Figure 20b ).

An important observation is that GSPO is quite robust to the choice of clipping ratio. Once the correct scale is identified, most nearby values or even larger scale perform similarly. This robustness contrasts sharply with DAPO-style losses, which are highly sensitive to the exact value of the higher clipping ratio, as noted in Section 3.2 .

#### A.17.4 GSPO vs CISPO

Despite hyperparameter robustness, we encountered stability issues with GSPO. On multiple occasions, GSPO runs diverged mid-training, leading to sudden drops in performance. For 8B models, restarting from a stable checkpoint allowed recovery, but this strategy failed on larger models such as Scout, where instability persisted despite repeated resetting to a stable checkpoint. While we checked to the best of our ability for any implementation bugs, we did not find one.

Overall, while all three loss families can be competitive under tuned settings, CISPO offers the best balance of stability and robustness to hyperparameters, making it our recommended choice.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .