Single-stream Policy Optimization

arxiv:2509.13232

Original source · Paper JSON

Post-training · 2025-09-16

Cites (6) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Single-stream Policy Optimization
• 1 Introduction
• 2 Related Work
• 3 Background
• 3.1 Policy Gradient and the REINFORCE Algorithm
• 3.2 Variance Reduction Baselines for Large Language Models
• 4 Method
• 4.1 A KL-Adaptive Value Tracker
• 4.2 Advantage Estimation and Policy Optimization
• 4.3 Prioritized Prompt Sampling
• 4.4 Advantages over GRPO
• 5 Experiments
• 5.1 Experimental Setup
• 5.2 Empirical Comparison with GRPO
• 5.3 Analysis of Signal Efficiency and Stability
• 5.4 Agentic Training Demonstrations
• 6 Conclusions
• Appendix A SPO Initialization
• Appendix B Batch Extensions
• Appendix C Comparisons against GRPO
• C.1 Inefficiency of Dynamic Sampling
• C.2 Variance Reduction for Policy Gradient
• Appendix D Training and Evaluation Details
• Appendix E Ablation Studies
• Instructions for reporting errors