Group Sequence Policy Optimization — part 2/2
Cleaned-text code points [13674, 28061). Bibliography omitted here; source.md includes it.
We can derive the gradient of the GSPO objective as follows (clipping is omitted for brevity): ∇ θ 𝒥 GSPO ( θ ) = \displaystyle\nabla_{\theta}\mathcal{J}_{\text{GSPO}}(\theta)= ∇ θ 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G s i ( θ ) A ^ i ] \displaystyle\ \nabla_{\theta}\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}s_{i}(\theta)\widehat{A}_{i}\right] (8) = \displaystyle= 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G s i ( θ ) A ^ i ⋅ ∇ θ log s i ( θ ) ] \displaystyle\ \mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}s_{i}(\theta)\widehat{A}_{i}\cdot\nabla_{\theta}\log s_{i}(\theta)\right] (9) = \displaystyle= 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G ( π θ ( y i | x ) π θ old ( y i | x ) ) 1 | y i | A ^ i ⋅ 1 | y i | ∑ t = 1 | y i | ∇ θ log π θ ( y i , t | x , y i , < t ) ] . \displaystyle\ \mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\left(\frac{\pi_{\theta}(y_{i}|x)}{\pi_{\theta_{\text{old}}}(y_{i}|x)}\right)^{\frac{1}{|y_{i}|}}\widehat{A}_{i}\cdot\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\nabla_{\theta}\log\pi_{\theta}(y_{i,t}|x,y_{i,<t})\right]. (10) For comparison, the gradient of the GRPO objective is as follows (note that A ^ i , t = A ^ i \widehat{A}_{i,t}=\widehat{A}_{i} ): ∇ θ 𝒥 GRPO ( θ ) = \displaystyle\nabla_{\theta}\mathcal{J}_{\text{GRPO}}(\theta)= ∇ θ 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | w i , t ( θ ) A ^ i , t ] \displaystyle\ \nabla_{\theta}\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}w_{i,t}(\theta)\widehat{A}_{i,t}\right] (11) = \displaystyle= 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G A ^ i ⋅ 1 | y i | ∑ t = 1 | y i | π θ ( y i , t | x , y i , < t ) π θ old ( y i , t | x , y i , < t ) ∇ θ log π θ ( y i , t | x , y i , < t ) ] . \displaystyle\ \mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\widehat{A}_{i}\cdot\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})}\nabla_{\theta}\log\pi_{\theta}(y_{i,t}|x,y_{i,<t})\right]. (12) Therefore, the fundamental distinction between GSPO and GRPO lies in how they weight the gradients of the log likelihoods of tokens . In GRPO, the tokens are weighted according to their respective “importance weight” π θ ( y i , t | x , y i , < t ) π θ old ( y i , t | x , y i , < t ) \frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})} . However, these unequal weights, which can vary among ( 0 , 1 + ε ] (0,1+\varepsilon] (for A ^ i > 0 \widehat{A}_{i}>0 ) or [ 1 − ε , + ∞ ) [1-\varepsilon,+\infty) (for A ^ i < 0 \widehat{A}_{i}<0 ), are not negligible, and their impact can accumulate and lead to unpredictable consequences as training progresses. In contrast, GSPO weights all the tokens in a response equally, eliminating this instability factor of GRPO.
### 4.3 GSPO-token: A Token-level Objective Variant
In scenarios like multi-turn RL, we may desire a finer-grained advantage adjustment than the sequence level. To this end, we introduce a token-level objective variant of GSPO, namely GSPO-token , to allow token-wise advantage customization: 𝒥 GSPO-token ( θ ) = 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( s i , t ( θ ) A ^ i , t , clip ( s i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) ] , \displaystyle\mathcal{J}_{\text{GSPO-token}}(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\left(s_{i,t}(\theta)\widehat{A}_{i,t},\,\mathrm{clip}\left(s_{i,t}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{i,t}\right)\right], (13) where s i , t ( θ ) = sg [ s i ( θ ) ] ⋅ π θ ( y i , t | x , y i , < t ) sg [ π θ ( y i , t | x , y i , < t ) ] , \displaystyle s_{i,t}(\theta)=\mathrm{sg}\left[s_{i}(\theta)\right]\cdot\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i,t}|x,y_{i,<t})\right]}, (14) and sg [ ⋅ ] \mathrm{sg}[\cdot] denotes only taking the numerical value but stopping the gradient, corresponding to the detach operation in PyTorch. The gradient of GSPO-token can be derived as: ∇ θ 𝒥 GSPO-token ( θ ) = \displaystyle\nabla_{\theta}\mathcal{J}_{\text{GSPO-token}}(\theta)= ∇ θ 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | s i , t ( θ ) A ^ i , t ] \displaystyle\ \nabla_{\theta}\mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}s_{i,t}(\theta)\widehat{A}_{i,t}\right] (15) = \displaystyle= 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G s i ( θ ) ⋅ 1 | y i | ∑ t = 1 | y i | A ^ i , t ∇ θ π θ ( y i , t | x , y i , < t ) π θ ( y i , t | x , y i , < t ) ] \displaystyle\ \mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}s_{i}(\theta)\cdot\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\widehat{A}_{i,t}\frac{\nabla_{\theta}\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}\right] (16) = \displaystyle= 𝔼 x ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | x ) [ 1 G ∑ i = 1 G ( π θ ( y i | x ) π θ old ( y i | x ) ) 1 | y i | ⋅ 1 | y i | ∑ t = 1 | y i | A ^ i , t ∇ θ log π θ ( y i , t | x , y i , < t ) ] . \displaystyle\ \mathbb{E}_{x\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|x)}\left[\frac{1}{G}\sum_{i=1}^{G}\left(\frac{\pi_{\theta}(y_{i}|x)}{\pi_{\theta_{\text{old}}}(y_{i}|x)}\right)^{\frac{1}{|y_{i}|}}\cdot\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\widehat{A}_{i,t}\nabla_{\theta}\log\pi_{\theta}(y_{i,t}|x,y_{i,<t})\right]. (17) Note that the term π θ ( y i , t | x , y i , < t ) sg [ π θ ( y i , t | x , y i , < t ) ] \frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\mathrm{sg}\left[\pi_{\theta}(y_{i,t}|x,y_{i,<t})\right]} has a numerical value of 1, so s i , t ( θ ) s_{i,t}(\theta) is numerically equal to s i ( θ ) s_{i}(\theta) . Comparing Equation ( 5 ) and ( 13 ), and Equation ( 10 ) and ( 17 ), GSPO-token and GSPO are numerically identical in the optimization objective, clipping condition, and theoretical gradient when we set the advantages of all the tokens in the response y i y_{i} to the same value (i.e., A ^ i , t = A ^ i \widehat{A}_{i,t}=\widehat{A}_{i} ), while GSPO-token enjoys the higher flexibility of adjusting the advantages per token.
## 5 Experiments and Discussion
### 5.1 Empirical Results
We experiment with a cold-start model fine-tuned from Qwen3-30B-A3B-Base, and report the training reward curves as well as the model performance curves on the AIME’24 (average Pass@1 over 32 samplings), LiveCodeBench (202410-202502, average Pass@1 over 8 samplings), and CodeForces (Elo Rating) benchmarks. During the RL training, each batch of rollout data is partitioned into four mini-batches for gradient updates. In GSPO, we set the left and right clipping ranges in Equation ( 5 ) to 3e-4 and 4e-4, respectively. We compare against GRPO as the baseline and set the left and right clipping ranges in Equation ( 2 ) to 0.2 and 0.27, respectively, which we have carefully tuned to ensure a fair comparison. Note that GRPO necessitates the Routing Replay training strategy for the normal convergence of MoE RL, which we will additionally discuss in § 5.3 , while GSPO has obviated the need for this strategy .
Figure 1 shows that the training with GSPO proceeds stably throughout. We observe that GSPO can deliver continuous performance improvement through increasing the training compute, regularly updating the query set, and extending the generation length . Moreover, GSPO also demonstrates superior training efficiency over GRPO, achieving better training accuracy and benchmark performance under the same training compute and consumed queries. Finally, we have successfully applied GSPO to the RL training of the latest Qwen3 models, strongly proving the efficacy of GSPO in unleashing the power of RL scaling for large language models.
### 5.2 Curious Observation on Clipping Fractions
A key distinction of GSPO compared to GRPO is its practice of clipping entire responses rather than individual tokens. Particularly, as shown in Figure 2 , we observe a difference of two orders of magnitude in the fractions of clipped tokens between GSPO and GRPO (while adjusting the clipping ranges does not alter the disparity in magnitude). However, despite clipping significantly more tokens and consequently using fewer for training (or gradient estimation), GSPO still achieves higher training efficiency than GRPO. This counter-intuitive finding — that clipping a much larger fraction of tokens leads to superior training efficiency — further indicates that GRPO’s token-level gradient estimates are inherently noisy and inefficient for sample exploitation. In contrast, GSPO’s sequence-level approach provides a more reliable and effective learning signal.
### 5.3 Benefit of GSPO for MoE Training
#### Background
Compared to the RL training of dense models, the sparse activation nature of MoE models introduces unique stability challenges. In particular, we found that when adopting the GRPO algorithm, the expert-activation volatility of MoE models can prevent RL training from converging properly. To be specific, after one or more gradient updates, the experts activated for the same response can change significantly. For example, with the 48-layer Qwen3-30B-A3B-Base model, after each RL gradient update and for the same rollout sample, there are roughly 10% of the experts activated under the new policy π θ \pi_{\theta} that are different from those under the old policy π θ old \pi_{\theta_{\text{old}}} . This phenomenon, which becomes more prominent in deeper MoE models, makes the token-level importance ratios w i , t ( θ ) = π θ ( y i , t | x , y i , < t ) π θ old ( y i , t | x , y i , < t ) w_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})} fluctuate drastically and further invalidates them, as discussed in § 3 and 4.2 , consequently hindering the normal convergence of RL training.
#### Our Previous Approach
To tackle this challenge, we previously employed the Routing Replay training strategy. Specifically, we cache the activated experts in π θ old \pi_{\theta_{\text{old}}} and “replay” these routing modes in π θ \pi_{\theta} when computing the importance ratios w i , t ( θ ) = π θ ( y i , t | x , y i , < t ) π θ old ( y i , t | x , y i , < t ) w_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}|x,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t})} . In this way, for each token y i , t y_{i,t} , π θ ( y i , t | x , y i , < t ) \pi_{\theta}(y_{i,t}|x,y_{i,<t}) and π θ old ( y i , t | x , y i , < t ) \pi_{\theta_{\text{old}}}(y_{i,t}|x,y_{i,<t}) share the same activated network, so that we can restore the stability of the token-level importance ratios and ensure optimization of the consistent activated network across gradient updates. Figure 3 demonstrates that Routing Replay serves as an essential technique in the normal convergence of the GRPO training of MoE models.
#### Benefit of GSPO
Although Routing Replay enables the GRPO training of MoE models to converge properly, its practice of reusing routing modes incurs additional memory and communication overhead and can also limit the actual capacity of the MoE model. In contrast, as shown in Figure 1 , GSPO eliminates the dependency on Routing Replay and is fully capable of computing the importance ratios s i ( θ ) s_{i}(\theta) conventionally, converging normally, and optimizing stably. The key insight is that GSPO focuses only on the sequence likelihood (i.e., π θ ( y i | x ) \pi_{\theta}(y_{i}|x) ) and is not sensitive to the individual token likelihood (i.e., π θ ( y i , t | x , y i , < t ) \pi_{\theta}(y_{i,t}|x,y_{i,<t}) ). Since the MoE model always maintains its language modeling capability, the sequence likelihood will not fluctuate drastically. In summary, GSPO fundamentally resolves the expert-activation volatility issue in MoE models, obviating the need for complex workarounds like Routing Replay. This not only simplifies and stabilizes the training process but also allows the model to leverage its full capacity without artificial constraints.
### 5.4 Benefit of GSPO for RL Infrastructure
Given the precision discrepancies between training engines (e.g., Megatron) and inference engines (e.g., SGLang and vLLM), in practice, we typically use the training engine to recompute the likelihoods of sampled responses under the old policy π θ old \pi_{\theta_{\text{old}}} . However, GSPO uses only sequence-level, rather than token-level, likelihoods for optimization, and intuitively, the former is much more tolerant of precision discrepancies. Hence, GSPO makes it possible to directly use the likelihoods returned by the inference engine for optimization, thereby avoiding the need for recomputation with the training engine. This can be especially beneficial in scenarios like partial rollout and multi-turn RL and in the training-inference disaggregated frameworks.
## 6 Conclusion
We propose Group Sequence Policy Optimization (GSPO), a new reinforcement learning algorithm for training large language models. Following the basic principle of importance sampling, GSPO defines importance ratios based on sequence likelihood and performs sequence-level clipping, rewarding, and optimization. GSPO demonstrates notably superior training stability, efficiency, and performance compared to GRPO and exhibits particular efficacy for the large-scale RL training of MoE models, laying the foundation for the exceptional improvements in the latest Qwen3 models. With GSPO as a scalable algorithmic cornerstone, we will continue to scale RL and look forward to the resulting fundamental advances in intelligence.