Group Sequence Policy Optimization — part 1/2

Cleaned-text code points [0, 13674). Bibliography omitted here; source.md includes it.

# Group Sequence Policy Optimization

###### Abstract

This paper introduces Group Sequence Policy Optimization (GSPO), our stable, efficient, and performant reinforcement learning algorithm for training large language models. Unlike previous algorithms that adopt token-level importance ratios, GSPO defines the importance ratio based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. We demonstrate that GSPO achieves superior training efficiency and performance compared to the GRPO algorithm, notably stabilizes Mixture-of-Experts (MoE) RL training, and has the potential for simplifying the design of RL infrastructure. These merits of GSPO have contributed to the remarkable improvements in the latest Qwen3 models.

## 1 Introduction

Reinforcement learning (RL) has emerged as a pivotal paradigm for scaling language models ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; Qwen, 2025b ; Qwen, 2025a ) . Through large-scale RL, language models develop the capability to tackle sophisticated problems, such as competition-level mathematics and programming, by undertaking deeper and longer reasoning processes.

To successfully scale RL with greater computational investment, the foremost prerequisite is maintaining stable and robust training dynamics. However, current state-of-the-art RL algorithms, exemplified by GRPO ( Shao et al., 2024 ) , exhibit severe stability issues when training gigantic language models, often resulting in catastrophic and irreversible model collapse ( Qwen, 2025a ; MiniMax, 2025 ) . This instability hinders efforts to push the capability boundaries of language models through continued RL training.

In this paper, we identify that the instability of GRPO stems from the fundamental misapplication and invalidation of importance sampling weights in its algorithmic design. This introduces high-variance training noise that progressively accumulates with increased response length and is further amplified by the clipping mechanism, ultimately precipitating model collapse.

To address these core limitations, we propose Group Sequence Policy Optimization (GSPO) , a new RL algorithm for training large language models. The key innovation of GSPO lies in its theoretically grounded definition of importance ratio based on sequence likelihood ( Zheng et al., 2023 ) , aligning with the basic principle of importance sampling. Additionally, GSPO computes the normalized rewards as the advantages of multiple responses to a query, ensuring the alignment between sequence-level rewarding and optimization.

Our empirical evaluation demonstrates the significant superiority of GSPO over GRPO in training stability, efficiency, and performance. Critically, GSPO has inherently resolved the stability challenges in the RL training of large Mixture-of-Experts (MoE) models, eliminating the need for complex stabilization strategies, and shows the potential for simplifying RL infrastructure. These merits of GSPO ultimately contributed to the exceptional performance improvements in the latest Qwen3 models. We envision GSPO as a robust and scalable algorithmic foundation that will enable the continued advancement of large-scale RL training with language models.

## 2 Preliminaries

#### Notation

In this paper, an autoregressive language model parameterized by θ \theta is defined as a policy π θ \pi_{\theta} . We use x x to denote a query and 𝒟 \mathcal{D} as the query set. Given a response y y to a query x x , its likelihood under the policy π θ \pi_{\theta} is denoted as π θ ​ ( y | x ) = ∏ t = 1 | y | π θ ​ ( y t | x , y < t ) \pi_{\theta}(y|x)=\prod_{t=1}^{|y|}\pi_{\theta}(y_{t}|x,y_{<t}) where | y | |y| denotes the number of tokens in y y . A query-response pair ( x , y ) (x,y) can be scored by a verifier r r , resulting in a reward r ⁡ ( x , y ) ∈ [ 0 , 1 ] r(x,y)\in[0,1] .

#### Proximal Policy Optimization (PPO)

Using samples generated from the old policy π θ old \pi_{\theta_{\text{old}}} , PPO ( Schulman et al., 2017 ) constrains the policy update within a proximal region of the old policy through the clipping mechanism. Specifically, PPO employs the following objective for policy optimization (we omit the KL regularization term hereinafter for brevity, as it is not the focus of this paper): 𝒥 PPO ( θ ) = 𝔼 x ∼ 𝒟 , y ∼ π θ old ( ⋅ | x ) [ 1 | y | ∑ t = 1 | y | min ( w t ( θ ) A ^ t , clip ( w t ( θ ) , 1 − ε , 1 + ε ) A ^ t ) ] , \displaystyle\mathcal{J}_{\text{PPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{|y|}\sum_{t=1}^{|y|}\min\left(w_{t}(\theta)\widehat{A}_{t},\,\mathrm{clip}\left(w_{t}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{t}\right)\right], (1) where the importance ratio of the token y t y_{t} is defined as w t ​ ( θ ) = π θ ​ ( y t | x , y < t ) π θ old ​ ( y t | x , y < t ) w_{t}(\theta)=\frac{\pi_{\theta}(y_{t}|x,y_{<t})}{\pi_{\theta_{\text{old}}}(y_{t}|x,y_{<t})} , the advantage A ^ t \widehat{A}_{t} of y t y_{t} is estimated by another value model, and ε \varepsilon is the clipping range of importance ratios.

The core challenge of PPO in practice lies in its heavy reliance on the value model. Specifically, the value model usually has a similar size to the policy model, introducing a considerable memory and computational burden. Furthermore, the algorithmic effectiveness hinges on the reliability of its value estimate. While acquiring a reliable value model is inherently challenging, ensuring its scalability to longer responses and more complex tasks presents an even greater challenge.

#### Group Relative Policy Optimization (GRPO)

GRPO ( Shao et al., 2024 ) bypasses the need for the value model by computing the relative advantage of each response within a group of responses to the same query. Specifically, GRPO optimizes the following objective: 𝒥 GRPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( w i , t ( θ ) A ^ i , t , clip ( w i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) ] , \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\left(w_{i,t}(\theta)\widehat{A}_{i,t},\,\mathrm{clip}\left(w_{i,t}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{i,t}\right)\right], (2) where G G is the number of generated responses to each query x x (i.e., the group size), and the importance ratio w i , t ​ ( θ ) w_{i,t}(\theta) and advantage A ^ i , t \widehat{A}_{i,t} of token y i , t y_{i,t} are: w i , t ​ ( θ ) = π θ ​ ( y i , t | x , y i , < t ) π θ old ​ ( y i , t | x , y i , < t ) , A ^ i , t = A ^ i = r ⁡ ( x , y i ) − mean ⁡ ( { r ⁡ ( x , y i ) } i = 1 G ) std ⁡ ( { r ⁡ ( x , y i ) } i = 1 G ) , \displaystyle w_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})},\quad\ \widehat{A}_{i,t}=\widehat{A}_{i}=\frac{r(x,y_{i})-\mathrm{mean}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}{\mathrm{std}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}, (3) respectively, where all the tokens in y i y_{i} share the same advantage as A ^ i \widehat{A}_{i} .

## 3 Motivation

The growth in model size, sparsity (e.g., in Mixture-of-Experts models), and response length necessitates a large rollout batch size to maximize hardware utilization during RL. To improve sample efficiency, it is standard practice to partition a large batch of rollout data into multiple mini-batches for gradient updates. This procedure inevitably introduces an off-policy learning setting, where responses y y are sampled from an old policy π θ old \pi_{\theta_{\text{old}}} rather than the current policy π θ \pi_{\theta} being optimized. This also explains the necessity of the clipping mechanism in PPO and GRPO, which prevents overly “off-policy” samples from being involved in gradient estimation.

While mechanisms like clipping aim to manage this off-policy discrepancy, we identify a more fundamental issue in GRPO: its objective is ill-posed . This problem becomes particularly acute when training large models on long-response tasks, leading to catastrophic model collapse. The ill-posed nature of the GRPO objective stems from a misapplication of importance sampling weights. The principle of importance sampling is to estimate the expectation of a function f f under a target distribution π tar \pi_{\text{tar}} by re-weighting samples drawn from a behavior distribution π beh \pi_{\text{beh}} : 𝔼 z ∼ π tar ​ [ f ⁡ ( z ) ] = 𝔼 z ∼ π beh ​ [ π tar ​ ( z ) π beh ​ ( z ) ​ f ​ ( z ) ] . \displaystyle\mathbb{E}_{z\sim\pi_{\text{tar}}}\left[f(z)\right]=\mathbb{E}_{z\sim\pi_{\text{beh}}}\left[\frac{\pi_{\text{tar}}(z)}{\pi_{\text{beh}}(z)}\,f(z)\right]. (4) Crucially, this relies on averaging over multiple samples ( N ≫ 1 N\gg 1 ) from the behavior distribution π beh \pi_{\text{beh}} for the importance weight π tar ​ ( z ) π beh ​ ( z ) \frac{\pi_{\text{tar}}(z)}{\pi_{\text{beh}}(z)} to effectively correct for the distributional mismatch.

In contrast, GRPO applies the importance weight π θ ​ ( y i , t | x , y i , < t ) π θ old ​ ( y i , t | x , y i , < t ) \frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})} at each token position t t . Since this weight is based on a single sample y i , t y_{i,t} from each next-token distribution π θ old ( ⋅ | x , y i , < t ) \pi_{\theta_{\text{old}}}(\cdot|x,y_{i,<t}) , it fails to perform the intended distribution-correction role. Instead, it introduces high-variance noise into the training gradients, which accumulates over long sequences and is exacerbated by the clipping mechanism. We have empirically observed that this can lead to model collapse that is often irreversible. Once the collapse occurs, resuming training is unavailing, even when reverting to a previous checkpoint and meticulously tuning hyperparameters (e.g., the clipping ranges), extending generation length, or switching the RL queries.

The above observation suggests a fundamental issue in GRPO’s design. The failure of the token-level importance weight points to a core principle: the unit of optimization objective should match the unit of reward . Since the reward is granted to the entire sequence, applying off-policy correction at the token level appears problematic. This motivates us to forego the token-level objective and explore utilizing importance weights and performing optimization directly at the sequence level .

## 4 Algorithm

### 4.1 GSPO: Group Sequence Policy Optimization

While the token-level importance weight π θ ​ ( y i , t | x , y i , < t ) π θ old ​ ( y i , t | x , y i , < t ) \frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})} is problematic in GRPO, we observe that in the context of language generation, the sequence-level importance weight π θ ​ ( y | x ) π θ old ​ ( y | x ) \frac{\pi_{\theta}(y|x)}{\pi_{\theta_{\text{old}}}(y|x)} has a clear theoretical meaning: it reflects how far the response y y sampled from π θ old ( ⋅ | x ) \pi_{\theta_{\text{old}}}(\cdot|x) deviates from π θ ( ⋅ | x ) \pi_{\theta}(\cdot|x) , which naturally aligns with the sequence-level reward and can also serve as a meaningful indicator of the clipping mechanism.

Based on this straightforward observation, we propose the Group Sequence Policy Optimization (GSPO) algorithm. GSPO employs the following sequence-level optimization objective: 𝒥 GSPO ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G min ( s i ( θ ) A ^ i , clip ( s i ( θ ) , 1 − ε , 1 + ε ) A ^ i ) ] , \displaystyle\mathcal{J}_{\text{GSPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\min\left(s_{i}(\theta)\widehat{A}_{i},\,\mathrm{clip}\left(s_{i}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{i}\right)\right], (5) where we adopt the group-based advantage estimation: A ^ i = r ⁡ ( x , y i ) − mean ⁡ ( { r ⁡ ( x , y i ) } i = 1 G ) std ⁡ ( { r ⁡ ( x , y i ) } i = 1 G ) , \displaystyle\widehat{A}_{i}=\frac{r(x,y_{i})-\mathrm{mean}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}{\mathrm{std}\left(\{r(x,y_{i})\}_{i=1}^{G}\right)}, (6) and define the importance ratio s i ​ ( θ ) s_{i}(\theta) based on sequence likelihood ( Zheng et al., 2023 ) : s i ​ ( θ ) = ( π θ ​ ( y i | x ) π θ old ​ ( y i | x ) ) 1 | y i | = exp ⁡ ( 1 | y i | ​ ∑ t = 1 | y i | log ⁡ π θ ​ ( y i , t | x , y i , < t ) π θ old ​ ( y i , t | x , y i , < t ) ) . \displaystyle s_{i}(\theta)=\left(\frac{\pi_{\theta}(y_{i}|x)}{\pi_{\theta_{\text{old}}}(y_{i}|x)}\right)^{\frac{1}{|y_{i}|}}=\exp\left(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\log\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})}\right). (7) Therefore, GSPO applies clipping to entire responses instead of individual tokens to exclude the overly “off-policy” samples from gradient estimation, which matches both the sequence-level rewarding and optimization. Note that we adopt length normalization in s i ​ ( θ ) s_{i}(\theta) to reduce the variance and to control s i ​ ( θ ) s_{i}(\theta) within a unified numerical range. Otherwise, the likelihood changes of a few tokens can result in dramatic fluctuations of the sequence-level importance ratio, and the importance ratios of responses with different lengths will require varying clipping ranges. We also note that the clipping ranges in GSPO and in previous algorithms (e.g., GRPO) typically differ in order of magnitude due to the distinct definitions of importance ratios.

### 4.2 Gradient Analysis