Soft Adaptive Policy Optimization — part 1/3
Cleaned-text code points [0, 14124). Bibliography omitted here; source.md includes it.
# Soft Adaptive Policy Optimization
###### Abstract
Reinforcement learning (RL) plays an increasingly important role in enhancing the reasoning capabilities of large language models (LLMs), yet stable and performant policy optimization remains challenging. Token-level importance ratios often exhibit high variance—a phenomenon exacerbated in Mixture-of-Experts models—leading to unstable updates. Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping, making it difficult to maintain both stability and effective learning. We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate that adaptively attenuates off-policy updates while preserving useful learning signals. Compared with GSPO and GRPO, SAPO is both sequence-coherent and token-adaptive. Like GSPO, SAPO maintains sequence-level coherence, but its soft gating forms a continuous trust region that avoids the brittle hard clipping band used in GSPO. When a sequence contains a few highly off-policy tokens, GSPO suppresses all gradients for that sequence, whereas SAPO selectively down‑weights only the offending tokens and preserves the learning signal from the near-on‑policy ones, improving sample efficiency. Relative to GRPO, SAPO replaces hard token‑level clipping with smooth, temperature-controlled scaling, enabling more informative and stable updates. Empirical results on mathematical reasoning benchmarks indicate that SAPO exhibits improved training stability and higher Pass@1 performance under comparable training budgets. Moreover, we employ SAPO to train the Qwen3-VL model series, demonstrating that SAPO yields consistent performance gains across diverse tasks and different model sizes. Overall, SAPO provides a more reliable, scalable, and effective optimization strategy for RL training of LLMs.
## 1 Introduction
Reinforcement learning (RL) has become a key driver of recent advances in large language models (LLMs), enabling deeper and longer reasoning for challenging tasks in mathematics, programming, and multimodal understanding ( OpenAI, 2024 ; DeepSeek-AI, 2025 ; Qwen, 2025 ) . Among RL methods, group-based policy optimization has emerged as a practical recipe: multiple responses are sampled per query, sequence-level rewards are normalized within the group, and policy updates are weighted by importance ratios between the current and behavior policies ( Shao et al., 2024 ; Zheng et al., 2025 ) .
A central challenge in this setting is the high variance of token-level importance ratios, especially in Mixture-of-Experts (MoE) models where routing heterogeneity and long responses can amplify deviations across tokens. Such variance increases the likelihood of unstable updates. Hard clipping, as used in GRPO ( Shao et al., 2024 ) , constrains large deviations by zeroing gradients outside a fixed band. While effective at curbing excessive steps, hard clipping makes it difficult to strike a favorable trade-off: overly tight clipping limits the number of valid samples for gradient computation, while looser clipping introduces noisy gradients coming from off-policy samples.
To address the brittleness of hard clipping in group-based policy optimization, we propose Soft Adaptive Policy Optimization (SAPO) , a smooth and adaptive policy-gradient method that replaces hard clipping with a temperature-controlled soft gate, as shown in Figure 1 . SAPO weights token-level updates by a bounded, sigmoid-shaped function of the importance ratio, centered at the on-policy point. This implements a continuous trust region: near on-policy, gradients are preserved to encourage useful updates and exploration; as the ratio deviates, gradients are attenuated smoothly rather than truncated, maintaining the learning signal for moderate deviations while reducing optimization noise. To further enhance robustness in large vocabularies, SAPO employs asymmetric temperatures for positive and negative tokens to make gradients on negative tokens decay more rapidly, reflecting their distinct stability profiles: negative updates tend to increase the logits of many inappropriate tokens and are therefore more prone to introduce instability than positive updates.
Conceptually, SAPO is designed to be sequence-coherent and token-adaptive. Under mild and empirically common conditions—small on-policy steps and low dispersion of token log-ratios within a sequence—the average token gate concentrates to a smooth sequence-level gate, aligning optimization with sequence-level rewards in the spirit of sequence-based methods such as GSPO ( Zheng et al., 2025 ) . When these conditions are violated due to heterogeneous or outlier tokens, SAPO selectively down-weights only the offending tokens while retaining informative gradients from near on-policy tokens within the same sequence. This selective attenuation mitigates the signal loss associated with hard clipping, improving sampling efficiency while maintaining stable updates.
Empirically, SAPO provides improved stability and task performance compared with GSPO and GRPO. While all methods may ultimately exhibit signs of instability, SAPO sustains coherent learning for a longer duration and reaches higher Pass@1 accuracy before divergence. This stems from SAPO’s ability to preserve informative gradients beyond the hard-clip threshold while selectively suppressing high-variance token updates. Furthermore, our temperature ablations further reveal that the asymmetric design—using a larger temperature for negative-token updates—is critical: it dampens high-variance negative gradients and significantly reduces the likelihood of early collapse. Beyond controlled settings, SAPO also proves effective in practical training of Qwen3-VL models across a broad mixture of text and multimodal tasks and across different model scales and architectures. Together, these results demonstrate that SAPO’s smooth gating and asymmetric temperature control enable a more reliable and productive RL training of large language models.
## 2 Preliminaries
#### Notation.
We model an autoregressive language model parameterized by θ \theta as a stochastic policy π θ \pi_{\theta} over token sequences. Let q q denote a query and 𝒟 \mathcal{D} the query set. For a response y y to q q , its likelihood under π θ \pi_{\theta} factorizes as π θ ( y ∣ q ) = ∏ t = 1 | y | π θ ( y t ∣ q , y < t ) \pi_{\theta}(y\mid q)\;=\;\prod_{t=1}^{|y|}\pi_{\theta}\!\left(y_{t}\mid q,y_{<t}\right) , where | y | |y| is the number of tokens in y y .
#### Group Relative Policy Optimization (GRPO).
For each query q ∼ 𝒟 q\sim\mathcal{D} , GRPO ( Shao et al., 2024 ) samples a group of G G responses { y 1 , … , y G } \{y_{1},\ldots,y_{G}\} from the behavior policy π θ old \pi_{\theta_{\text{old}}} , computes their rewards { R 1 , … , R G } \{R_{1},\ldots,R_{G}\} , and maximizes the following token-level objective: 𝒥 GRPO ( θ ) = 𝔼 q ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | q ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | min ( r i , t ( θ ) A ^ i , t , clip ( r i , t ( θ ) , 1 − ε , 1 + ε ) A ^ i , t ) ] , \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\left[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\min\left(r_{i,t}(\theta)\widehat{A}_{i,t},\,\mathrm{clip}\left(r_{i,t}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{i,t}\right)\right], (1) where
r i , t ( θ ) = π θ ( y i , t | q , y i , < t ) π θ old ( y i , t | q , y i , < t ) , A ^ i , t = A ^ i = R i − mean ( { R j } j = 1 G ) std ( { R j } j = 1 G ) , \displaystyle r_{i,t}(\theta)=\frac{\pi_{\theta}(y_{i,t}|q,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|q,y_{i,<t})},\quad\ \widehat{A}_{i,t}=\widehat{A}_{i}=\frac{R_{i}-\text{mean}(\{R_{j}\}_{j=1}^{G})}{\mathrm{std}\left(\{R_{j}\}_{j=1}^{G}\right)}, (2)
ε > 0 \varepsilon>0 is the clipping range, G G is the number of responses in a group, and A ^ i , t \widehat{A}_{i,t} is the group-normalized advantage (shared across tokens within a response).
#### Group Sequence Policy Optimization (GSPO)
GSPO ( Zheng et al., 2025 ) employs the following sequence-level optimization objective: 𝒥 GSPO ( θ ) = 𝔼 q ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ | q ) [ 1 G ∑ i = 1 G min ( s i ( θ ) A ^ i , clip ( s i ( θ ) , 1 − ε , 1 + ε ) A ^ i ) ] , \displaystyle\mathcal{J}_{\text{GSPO}}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\,\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot|q)}\left[\frac{1}{G}\sum_{i=1}^{G}\min\left(s_{i}(\theta)\widehat{A}_{i},\,\mathrm{clip}\left(s_{i}(\theta),1-{\varepsilon},1+{\varepsilon}\right)\widehat{A}_{i}\right)\right], (3) where s i ( θ ) = ( π θ ( y i | q ) π θ old ( y i | q ) ) 1 | y i | = exp ( 1 | y i | ∑ t = 1 | y i | log π θ ( y i , t | q , y i , < t ) π θ old ( y i , t | q , y i , < t ) ) , A ^ i = R i − mean ( { R j } j = 1 G ) std ( { R j } j = 1 G ) \displaystyle s_{i}(\theta)=\left(\frac{\pi_{\theta}(y_{i}|q)}{\pi_{\theta_{\text{old}}}(y_{i}|q)}\right)^{\frac{1}{|y_{i}|}}=\exp\left(\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}\log\frac{\pi_{\theta}(y_{i,t}|q,y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|q,y_{i,<t})}\right),\quad\ \widehat{A}_{i}=\frac{R_{i}-\text{mean}(\{R_{j}\}_{j=1}^{G})}{\mathrm{std}\left(\{R_{j}\}_{j=1}^{G}\right)} (4) GSPO applies clipping at the sequence level rather than per token. The length normalization in s i ( θ ) s_{i}(\theta) reduces variance and places it on a consistent numerical scale across responses.
## 3 Soft Adaptive Policy Optimization
Soft Adaptive Policy Optimization (SAPO) is a smooth and adaptive policy-gradient method for RL fine-tuning, which replaces hard clipping with a temperature-controlled soft gate. Smooth gating functions have been explored in traditional RL settings ( Chen et al., 2023 ) . In SAPO, we incorporate this idea into the group‑based RL paradigm for LLMs and extend it with two additional components that are essential for large‑scale language model training: (1) a token‑level soft trust region that naturally yields sequence‑level coherence, and (2) an asymmetric temperature design motivated by the distinct behaviors of positive and negative token updates. Specifically, SAPO maximizes the following objective: 𝒥 ( θ ) = 𝔼 q ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | f i , t ( r i , t ( θ ) ) A ^ i , t ] , \displaystyle\mathcal{J}(\theta)=\mathbb{E}_{q\sim\mathcal{D},\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}f_{i,t}(r_{i,t}(\theta))\widehat{A}_{i,t}\Bigg], (5) where f i , t ( x ) = σ ( τ i , t ( x − 1 ) ) ⋅ 4 τ i , t , τ i , t = { τ pos , if A ^ i , t > 0 τ neg , otherwise , f_{i,t}(x)=\sigma\left(\tau_{i,t}\,(x-1)\right)\cdot\frac{4}{\tau_{i,t}},\quad\tau_{i,t}=\begin{cases}\tau_{\text{pos}},&\text{if }\widehat{A}_{i,t}>0\\ \tau_{\text{neg}},&\text{otherwise}\end{cases}, (6) A ^ i , t \widehat{A}_{i,t} and r i , t ( θ ) r_{i,t}(\theta) are computed as in Equation ( 2 ), τ pos \tau_{\text{pos}} and τ neg \tau_{\text{neg}} are the temperatures for positive and negative tokens, respectively, and σ ( x ) = 1 / ( 1 + e − x ) \sigma(x)=1/(1+e^{-x}) is the sigmoid function.
Differentiating ( 5 ) yields a weighted log-policy gradient: ∇ θ 𝒥 ( θ ) \displaystyle\nabla_{\theta}\mathcal{J}(\theta) = 𝔼 q ∼ 𝒟 , { y i } i = 1 G ∼ π θ old ( ⋅ ∣ q ) [ 1 G ∑ i = 1 G 1 | y i | ∑ t = 1 | y i | w i , t ( θ ) r i , t ( θ ) ∇ θ log π θ ( y i , t ∣ q , y i , < t ) A ^ i , t ] \displaystyle=\mathbb{E}_{q\sim\mathcal{D},\{y_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|y_{i}|}\sum_{t=1}^{|y_{i}|}w_{i,t}(\theta)\,r_{i,t}(\theta)\,\nabla_{\theta}\log\pi_{\theta}(y_{i,t}\mid q,y_{i,<t})\widehat{A}_{i,t}\Bigg] (7) where w i , t ( θ ) = 4 p i , t ( θ ) ( 1 − p i , t ( θ ) ) , p i , t ( θ ) = σ ( τ i , t ( r i , t ( θ ) − 1 ) ) , w_{i,t}(\theta)=4\,p_{i,t}(\theta)\,\bigl(1-p_{i,t}(\theta)\bigr),\quad p_{i,t}(\theta)=\sigma\left(\tau_{i,t}\,(r_{i,t}(\theta)-1)\right), (8) which peaks at r i , t ( θ ) = 1 r_{i,t}(\theta)=1 with a value of 1 1 and decays smoothly and approximately exponentially as r i , t ( θ ) r_{i,t}(\theta) deviates from 1 1 , implementing a soft trust region and preventing both gradient vanishing and excessively large updates, as shown in Figure 1 . Notably, at r i , t ( θ ) = 1 r_{i,t}(\theta)=1 , the soft-gated gradient equals that of the unclipped objective r i , t ( θ ) A ^ i , t r_{i,t}(\theta)\widehat{A}_{i,t} , regardless of τ i , t \tau_{i,t} , preserving on-policy behavior. This also explains the presence of the 4 / τ i , t 4/\tau_{i,t} factor in f i , t f_{i,t} .
Compared with GSPO ( Zheng et al., 2025 ) and GRPO ( Shao et al., 2024 ) , SAPO provides both sequence-level coherence and token-level adaptivity: (1) Under mild assumptions—small on-policy steps and low dispersion of token log-ratios within a sequence—the average token gate concentrates to a smooth sequence-level gate g ( log s i ( θ ) ) = sech 2 ( τ i 2 log s i ( θ ) ) g(\log s_{i}(\theta))=\mathrm{sech}^{2}\!\big(\tfrac{\tau_{i}}{2}\log s_{i}(\theta)\big) . Thus, SAPO reduces to a GSPO-like sequence formulation but with a continuous trust region. Crucially, when a few off-policy tokens push s i s_{i} beyond GSPO’s hard band, GSPO suppresses gradients for the many near-on-policy tokens in that sequence, hurting sample efficiency. SAPO, in contrast, preserves informative gradients by down-weighting only the offending tokens while keeping near-on-policy tokens influential. (2) Relative to GRPO, SAPO avoids hard token-level clipping that zeroes gradients outside a fixed range. Instead, SAPO scales updates smoothly, providing a more balanced way to retain useful learning signals while preventing unstable policy shifts. See more details in Section 4 .
#### Why Different Temperatures for Positive and Negative Advantages