The Art of Scaling Reinforcement Learning Compute for LLMs

arxiv:2510.13786

Original source · Paper JSON

Post-training · 2025-10-15

Cites (7) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• The Art of Scaling Reinforcement Learning Compute for LLMs
• Abstract
• 1 Introduction
• 2 Preliminaries & Setup
• Training Regimen
• Base RL Algorithm
• Controlling Generation Lengths
• 2.1 Predictive compute-scaling and fitting curves
• Interpreting scaling curves
• Scaling curve on held-out validation
• 3 An Empirical Study of RL Scaling
• 3.1 Asynchronous RL Setup
• 3.2 Algorithmic Choices
• Loss type
• FP32 Precision for LLM logits
• Loss Aggregation
• Advantage Normalization
• Zero-Variance Filtering
• Adaptive Prompt Filtering
• 4 ScaleRL : Scaling RL Compute Effectively & Predictably
• Leave-One-Out (LOO) Ablations
• Error margin in fitting scaling curves
• Extrapolating Scaling Curves
• Are the design choices worth it?
• 5 Predictable Scaling Returns Across RL Compute Axes
• Model scale (MoE)
• Generation length (context budget)
• Global batch size (prompts)
• Generations per prompt (fixed total batch)
• 6 Related Work
• 7 Discussion & Conclusion
• Future work
• 8 Acknowledgments
• Appendix A Appendix
• A.1 Extended Related Work
• A.2 RL for LLMs: GRPO and DAPO
• Group Relative Policy Optimization (GRPO)
• Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO)
• A.3 Training Setup
• Datasets
• Supervised Fine-tuning
• Reinforcement Learning
• A.4 What curve to fit?
• A.5 Fitting scaling curves
• A.6 Comparing algorithms
• A.7 Robustness of fits
• A.8 Interpreting Sigmoidal Curves
• A.9 Forward and LOO Ablations
• A.10 Controlling generation length
• A.11 PipelineRL
• A.12 Entropy Curves: Scaling Batch Size
• A.13 Scaling on multiple axes
• A.14 Downstream performance
• A.15 Truncations and training instabilities
• A.16 Comparing Prevalent Methods
• DeepSeek (GRPO)
• Qwen2.5 (DAPO)
• Magistral
• MiniMax
• A.17 Loss Type - Stability and Robustness
• A.17.1 DAPO clipping ratios
• A.17.2 CISPO Clipping Ratios
• A.17.3 GSPO ablations
• A.17.4 GSPO vs CISPO
• Instructions for reporting errors