Soft Adaptive Policy Optimization

arxiv:2511.20347

Original source · Paper JSON

Post-training · 2025-11-25

Cites (2) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Soft Adaptive Policy Optimization
• Abstract
• 1 Introduction
• 2 Preliminaries
• Notation.
• Group Relative Policy Optimization (GRPO).
• Group Sequence Policy Optimization (GSPO)
• 3 Soft Adaptive Policy Optimization
• Why Different Temperatures for Positive and Negative Advantages
• 4 A Gating‑Function Perspective on SAPO’s Connections to GRPO and GSPO
• Unified surrogate
• Algorithm-specific f i , t f_{i,t}
• Gradient form for SAPO/GRPO
• 4.1 SAPO–GSPO Connection: Reduction to a Sequence-Level Soft Gate
• SAPO’s token-level soft gate
• Assumptions
• Average token gates ⇒ \Rightarrow sequence gate
• Do the two assumptions (A1) and (A2) hold?
• Advantages over GSPO
• 4.2 SAPO–GRPO Connection: Smooth Token Gates vs. Hard Token Clipping
• GRPO’s piecewise-hard token gate
• Advantages over GRPO
• 4.3 Summary
• 5 Experiments
• 5.1 Controlled Experiments
• 5.2 Qwen3‑VL Training
• 6 Conclusion