Soft Adaptive Policy Optimization — part 3/3

Cleaned-text code points [29869, 34749). Bibliography omitted here; source.md includes it.

These RL algorithms primarily differ in how they handle off-policy tokens for which r i , t ​ ( θ ) r_{i,t}(\theta) deviates from 1 1 . From a token-level perspective, SAPO provides a smooth downweighting mechanism; from a sequence-level perspective, SAPO suppresses gradients from extreme off-policy tokens in sequences, thereby constructing more effective sequences for training. In contrast, GRPO and GSPO rely on hard clipping, which is less adaptive than SAPO for optimization.

## 5 Experiments

### 5.1 Controlled Experiments

We conduct experiments using a cold-start model fine-tuned from Qwen3-30B-A3B-Base on mathematical reasoning queries. We report both the training reward and the validation performance (average Pass@1 over 16 samples) on the AIME25 ( AIME, 2025 ) , HMMT25 ( HMMT, 2025 ) , and BeyondAIME ( Seed et al., 2025 ) benchmarks. During RL training, each batch of rollout data is divided into four mini-batches for gradient updates. For SAPO, we set τ pos = 1.0 \tau_{\text{pos}}=1.0 and τ neg = 1.05 \tau_{\text{neg}}=1.05 in Equation ( 6 ). We compare SAPO with GSPO and GRPO-R2 (i.e., GRPO equipped with routing replay), using the same hyperparameter configurations as in Zheng et al. (2025) .

Figure 4 shows that SAPO consistently improves model performance across all benchmarks, achieving both higher stability and stronger final performance compared with GSPO and GRPO-R2. While GSPO and GRPO-R2 exhibit early-stage training collapse, SAPO maintains stable training dynamics and ultimately attains superior performance. Notably, SAPO does not rely on routing replay for stabilization or strong performance, which improves exploration and reduces engineering overhead for RL systems.

To empirically examine the effect of choosing τ neg > τ pos \tau_{\text{neg}}>\tau_{\text{pos}} , we evaluate three configurations: τ neg = 1.05 > τ pos = 1.0 \tau_{\text{neg}}=1.05>\tau_{\text{pos}}=1.0 , τ neg = τ pos = 1.0 \tau_{\text{neg}}=\tau_{\text{pos}}=1.0 , and τ neg = 0.95 < τ pos = 1.0 \tau_{\text{neg}}=0.95<\tau_{\text{pos}}=1.0 . As shown in Figure 5 , training is most stable when negative tokens are assigned a higher temperature ( τ neg = 1.05 \tau_{\text{neg}}=1.05 ) and most unstable when they are assigned a lower temperature ( τ neg = 0.95 \tau_{\text{neg}}=0.95 ). These results suggest that gradients associated with negative tokens contribute more strongly to training instability, and that SAPO’s asymmetric temperature design effectively mitigates this issue.

### 5.2 Qwen3‑VL Training

We apply SAPO to train models in the Qwen3-VL family to assess its effectiveness in practical large-scale settings. Our experiments show that SAPO consistently improves performance across models of varying sizes and across both MoE and dense architectures. We train on a broad collection of text and multimodal tasks, including mathematics, coding, and logical reasoning. To support multi-task learning, we maintain a fixed sampling ratio for each task within each batch. We also employ a large batch size, splitting each batch of rollout data into two mini-batches for gradient updates, ensuring that each mini-batch provides sufficient learning signal for all tasks.

To highlight SAPO’s advantages over GSPO and GRPO-R2, we evaluate all three reinforcement learning algorithms starting from a preliminary cold-start checkpoint of Qwen3-VL-30B-A3B. We report training rewards and mean validation performance on four benchmarks: AIME25 ( AIME, 2025 ) (Pass@1 with 32 samples), LiveCodeBench v6 ( Jain et al., 2024 ) (Pass@1 with 8 samples), ZebraLogic ( Lin et al., 2025 ) , and MathVision ( Wang et al., 2024 ) . As shown in Figure 6 , SAPO achieves steady performance gains throughout training and outperforms both baselines under equal compute budgets.

## 6 Conclusion

We introduce Soft Adaptive Policy Optimization (SAPO), a smooth and token-adaptive reinforcement learning algorithm designed to address the instability and inefficiencies associated with hard-clipped policy optimization in large language models. By replacing discontinuous clipping with a temperature-controlled soft gate and employing asymmetric temperatures to better regulate negative-token gradients, SAPO provides a more stable and informative optimization signal. Empirical results on several mathematical reasoning benchmarks demonstrate that SAPO extends the duration of stable training and achieves higher Pass@1 performance under comparable budgets. Beyond controlled settings, large-scale experiments on Qwen3‑VL models further demonstrate that SAPO delivers consistent improvements across diverse text and multimodal tasks and across different model sizes and architectures. These findings suggest that smooth and adaptive gating mechanisms offer a promising direction for improving the robustness and effectiveness of RL training for large language models.