Group Sequence Policy Optimization

arxiv:2507.18071

Original source · Paper JSON

Post-training · 2025-07-24

Cites (2) · Cited by (18)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Group Sequence Policy Optimization
• Abstract
• 1 Introduction
• 2 Preliminaries
• Notation
• Proximal Policy Optimization (PPO)
• Group Relative Policy Optimization (GRPO)
• 3 Motivation
• 4 Algorithm
• 4.1 GSPO: Group Sequence Policy Optimization
• 4.2 Gradient Analysis
• 4.3 GSPO-token: A Token-level Objective Variant
• 5 Experiments and Discussion
• 5.1 Empirical Results
• 5.2 Curious Observation on Clipping Fractions
• 5.3 Benefit of GSPO for MoE Training
• Background
• Our Previous Approach
• Benefit of GSPO
• 5.4 Benefit of GSPO for RL Infrastructure
• 6 Conclusion