Single-stream Policy Optimization — part 2/4

Cleaned-text code points [14865, 29992). Bibliography omitted here; source.md includes it.

The tracker adapts to policy changes by dynamically adjusting its memory of past rewards. When the policy changes significantly, older observations become less relevant and should be downweighted. After each new observation r ⁡ ( x , y ) ∈ { 0 , 1 } r(x,y)\in\{0,1\} , we discount the prior Beta parameters ( α − 1 , β − 1 ) (\alpha_{-1},\beta_{-1}) by a factor ρ ⁡ ( x ) \rho(x) before incorporating the new evidence r ⁡ ( x , y ) r(x,y) : α ⁡ ( x ) = ρ ⁡ ( x ) ​ α − 1 ​ ( x ) + r ⁡ ( x , y ) , β ⁡ ( x ) = ρ ⁡ ( x ) ​ β − 1 ​ ( x ) + ( 1 − r ⁡ ( x , y ) ) , v ^ ​ ( x ) = α ⁡ ( x ) α ⁡ ( x ) + β ⁡ ( x ) . \alpha(x)=\rho(x)\alpha_{-1}(x)+r(x,y),\quad\beta(x)=\rho(x)\beta_{-1}(x)+(1-r(x,y)),\quad\hat{v}(x)=\frac{\alpha(x)}{\alpha(x)+\beta(x)}. (5) The discount factor ρ ( x ) = 2 − D ( x ) / D half \rho(x)=2^{-D(x)/D_{\text{half}}} is determined by the KL divergence D ⁡ ( x ) D(x) between the current policy and the last policy that acted on prompt x x , causing the tracker to forget faster as the policy changes more significantly. The hyperparameter D half D_{\text{half}} controls this forgetting rate ρ ∈ [ ρ min , ρ max ] \rho\in[\rho_{\text{min}},\rho_{\text{max}}] .

Initialization. To initialize, we collect n 0 n_{0} samples to compute an initial value estimate v ^ 0 ​ ( x ) \hat{v}_{0}(x) . To avoid transient instability, we set the initial effective sample size to its expected equilibrium, N 0 = 1 / ( 1 − ρ min ) N_{0}=1/(1-\rho_{\min}) , where ρ min \rho_{\min} is the minimum allowed forgetting factor. The initial parameters are then: α 0 ​ ( x ) = N 0 ⋅ v ^ 0 ​ ( x ) , β 0 ​ ( x ) = N 0 ⋅ ( 1 − v ^ 0 ​ ( x ) ) . \alpha_{0}(x)=N_{0}\cdot\hat{v}_{0}(x),\qquad\beta_{0}(x)=N_{0}\cdot(1-\hat{v}_{0}(x)). (6)

This Bayesian update is equivalent to an adaptive Exponential Moving Average (EMA) on the value estimate: v ^ ​ ( x ) = v ^ − 1 ​ ( x ) + η ⁡ ( x ) ​ ( r ⁡ ( x , y ) − v ^ − 1 ​ ( x ) ) , \hat{v}(x)=\hat{v}_{-1}(x)+\eta(x)(r(x,y)-\hat{v}_{-1}(x)), (7) where the learning rate η ⁡ ( x ) = ( ρ ⁡ ( x ) ​ N eff , − 1 ​ ( x ) + 1 ) − 1 \eta(x)=(\rho(x)N_{\text{eff},-1}(x)+1)^{-1} naturally adapts to both policy shifts (via ρ ⁡ ( x ) \rho(x) ) and statistical confidence (via N eff = α ⁡ ( x ) + β ⁡ ( x ) + 1 N_{\text{eff}}=\alpha(x)+\beta(x)+1 ). This formulation highlights how our tracker balances new evidence against accumulated knowledge. For general rewards beyond binary ones, we can just use the same EMA formulation to directly track v ^ \hat{v} , rather than relying on α \alpha and β \beta in the binary cases.

### 4.2 Advantage Estimation and Policy Optimization

SPO uses the tracker’s estimate v ^ \hat{v} as a baseline for advantage calculation in a policy gradient algorithm. At iteration i i , for a single reward r ⁡ ( x , y ) r(x,y) obtained with policy π θ i \pi_{\theta_{i}} , the advantage is computed using the pre-update baseline (denoted with subscript -1 ): A ⁡ ( x , y ) = r ⁡ ( x , y ) − v ^ − 1 ​ ( x ) . A(x,y)=r(x,y)-\hat{v}_{-1}(x). (8) Using the baseline from the previous step ensures that it is independent of the action taken at step i i , preserving the unbiasedness of the policy gradient estimate. While the reward r ⁡ ( x , y ) r(x,y) is typically a direct outcome signal, SPO’s framework is also compatible with more sophisticated reward functions. For instance, recent work like InfAlign [ 4 ] demonstrates how to calibrate and transform the reward signal to be “inference-aware,” directly optimizing for procedures like Best-of- N N sampling. Such transformed rewards can be seamlessly integrated into SPO by replacing the standard r ⁡ ( x , y ) r(x,y) in the advantage calculation. Since v − 1 ​ ( x ) v_{-1}(x) is independent of y ∼ π θ i ( ⋅ | x ) y\sim\pi_{\theta_{i}}(\cdot|x) , 𝔼 ⁡ [ ( r − v i − 1 ​ ( x ) ) ​ ∇ θ ​ log ⁡ π ] = ∇ J ​ ( θ ) \mathbb{E}\!\left[(r-v_{i-1}(x))\,\nabla_{\theta}\log\pi\right]=\nabla J(\theta) [ 39 ] . Instead of normalizing advantages on a per-prompt basis in a group [ 34 , 11 ] , SPO normalizes them across an entire batch of prompts ℬ \mathcal{B} [ 19 , 31 , 3 , 23 ] . The normalized advantage A ~ ​ ( x , y ) \tilde{A}(x,y) is computed as: A ~ ​ ( x , y ) = A ⁡ ( x , y ) − μ ℬ σ ℬ , \tilde{A}(x,y)=\frac{A(x,y)-\mu_{\mathcal{B}}}{\sigma_{\mathcal{B}}}, (9) where μ ℬ \mu_{\mathcal{B}} and σ ℬ \sigma_{\mathcal{B}} are the mean and standard deviation of advantages in the batch { A ⁡ ( x , y ) } x ∈ ℬ \{A(x,y)\}_{x\in\mathcal{B}} . We then apply the advantage A ~ ​ ( x , y ) \tilde{A}(x,y) to each token in the response sequence y y and update the policy parameters using a standard PPO-Clip policy loss [ 31 ] 3 3 3 The term “PPO” is frequently used with ambiguity. It may denote the entire algorithm suite (e.g., clipped policy and value losses), refer narrowly to just the clipped policy objective, or describe the broader training framework, including mechanisms like mini-batch updates. : L CLIP ​ ( θ ) \displaystyle L^{\text{CLIP}}(\theta) = 𝔼 s , t ​ [ min ⁡ ( π θ ​ ( a t ∣ s t ) π θ old ​ ( a t ∣ s t ) ​ A t ~ , clip ⁡ ( π θ ​ ( a t ∣ s t ) π θ old ​ ( a t ∣ s t ) , 1 − ε , 1 + ε ) ​ A ~ t ) ] . \displaystyle=\mathbb{E}_{s,t}\!\left[\min\!\Bigg(\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})}\,\tilde{A_{t}},\;\operatorname{clip}\!\Big(\frac{\pi_{\theta}(a_{t}\mid s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}\mid s_{t})},\,1-\varepsilon,\,1+\varepsilon\Big)\,\tilde{A}_{t}\Bigg)\right]. (10) Methods like Clip-Higher [ 42 ] , Clip-Cov [ 10 ] and KL-Cov [ 10 ] to retain policy entropy are applicable here. Other policy optimization algorithms like CISPO [ 27 ] (similar to vtrace [ 12 , 40 ] ) and GSPO [ 43 ] (use sequence-level likelihood instead of token-level) are compatible with our advantage estimator. Advanced methods to control policy behaviors like ASPO [ 22 ] can be utilized to modulate the advantage values. We note that if we use “no baseline” (i.e., v ^ = 0 \hat{v}=0 ), it is an extremely simple and valid algorithm but may suffer from high policy gradient variance.

### 4.3 Prioritized Prompt Sampling

To further enhance data efficiency, SPO employs a curriculum learning strategy by prioritizing prompts with the highest learning potential [ 30 , 36 ] . At each iteration, we sample a batch of prompts based on a score that emphasizes prompts with high uncertainty, while ensuring a minimum level of exploration. The sampling weight w i ​ ( x ) w_{i}(x) for prompt x x is defined as: w i ​ ( x ) ∝ v ^ − 1 ​ ( x ) ​ ( 1 − v ^ − 1 ​ ( x ) ) + ϵ . \addcontentsline{lla}{section}{\numberline qn:sampling_{c}oef}w_{i}(x)\propto\sqrt{\hat{v}_{-1}(x)\bigl(1-\hat{v}_{-1}(x)\bigr)}+\epsilon. (11) The first term corresponds to the estimated standard deviation of a Bernoulli outcome, which naturally allocates more weight to prompts that are neither almost always solved ( v ^ ≈ 1 \hat{v}\approx 1 ) nor almost always failed ( v ^ ≈ 0 \hat{v}\approx 0 ). The exploration bonus ϵ \epsilon , set to 0.05 0.05 by default, prevents curriculum collapse by ensuring that every prompt retains a non-zero probability of being sampled, thereby maintaining broad coverage of the data distribution. The complete SPO training procedure is outlined in Algorithm 1 .

### 4.4 Advantages over GRPO

Group-Free for Scalable Infrastructure. SPO’s design is inherently “group-free”, a significant advantage in distributed training frameworks for LLMs. Each sample, consisting of a single stream of (prompt, response) pair, is a self-contained data point for the policy update. GRPO, however, requires the generation and evaluation of an entire group of G G samples for a single prompt before any training signal can be computed. We provide our illustrations in Figure 1 . In a distributed setting, this introduces a synchronization barrier: the processing of a given prompt is not complete until all G G responses have been generated. This is particularly problematic in the presence of long-tail generation times, where a single slow response generation can stall the processing for its entire group . For constructing a training batch, SPO only needs to collect B B independent (prompt, response) pairs, which is far more flexible and efficient than waiting for B B entire groups to complete. This makes SPO’s architecture significantly more infrastructure-friendly and scalable. The advantage is amplified in agentic training, especially in settings that require multi-turn interactions with tools [ 15 , 9 ] or long-horizon agent rollouts [ 45 , 41 ] . The scale of these interactions can be substantial: state-of-the-art open-source models ( gpt-oss-120b ) may average 20 search turns per task [ 9 ] , with other agentic sessions reaching over 40 tool calls and generating up to 150,000 tokens of context [ 15 ] .

Adaptive Curriculum. To further enhance training efficiency, SPO integrates a prioritized sampling scheme. This mechanism naturally creates an adaptive curriculum by focusing computational resources on prompts with the highest learning potential. This ensures that the model’s training is concentrated on the most informative examples at any given point in time. GRPO, in its standard formulation, typically relies on uniform sampling of prompts. This may waste computation on prompts that are already mastered or are currently too difficult to yield useful learning signals. While dynamic sampling [ 42 ] and repeat strategies [ 2 ] have been proposed to mitigate this issue, they often discard samples after generation, wasting computation. SPO’s prioritized sampling addresses the scheduling problem before response generation, leading to a more natural and efficient training process.

More discussions on the inefficiency of dynamic sampling and the variance reduction of policy gradient are outlined in Appendix C , where we provide detailed analysis.

## 5 Experiments

### 5.1 Experimental Setup

The SPO algorithm is broadly applicable in LLM reasoning tasks [ 11 ] and Agentic training. We evaluate Tool-Integrated Reasoning (TIR) [ 13 , 22 ] scenarios, where the LLMs can utilize external Python interpreter to help solve hard problems. We conduct experiments using a moderately sized LLM, Qwen3-8B [ 29 ] . For training data, we use the English subset from the DAPO dataset [ 42 ] . Only outcome reward is applied for RLVR, without the format rewards. We evaluate performance on the challenging math competition benchmarks, i.e., AIME 24, AIME 25, BeyondAIME [ 32 ] , BRUMO 25 [ 5 ] , and HMMT 25 [ 5 ] . See Appendix D for training and evaluation details.

We distinguish our goal from that of “hill-climbing” on benchmark leaderboards. The latter often necessitates resource-intensive and highly specialized techniques, including SFT from frontier models [ 24 ] , mid-training [ 38 ] , multi-stage RL pipelines [ 26 , 18 , 8 ] , curated hard datasets with intricate processing [ 2 , 33 ] , test-time scaling techniques [ 14 ] and extremely large generation group sizes [ 45 ] . Our work, instead, concentrates on the fundamental efficiency and scalability of the RL algorithm itself.

### 5.2 Empirical Comparison with GRPO

Method AIME 24 AIME 25 BeyondAIME BRUMO 25 HMMT 25 Average maj@32 avg@32 maj@32 avg@32 maj@32 avg@32 maj@32 avg@32 maj@32 avg@32 maj@32 avg@32 Qwen3-8B 77.8 64.4 70.5 58.4 45.2 38.0 55.1 49.4 36.8 30.3 57.1 48.1 GRPO 83.3 77.6 72.1 64.2 45.6 39.0 56.7 56.9 44.2 40.9 60.4 55.7 SPO 84.0 74.9 76.5 65.0 46.9 40.3 64.0 59.0 47.5 40.6 63.8 56.0

Our experiments demonstrate that SPO outperforms the GRPO baseline on aggregate metrics when training the Qwen-8B model. As shown in Table 1 , SPO achieves superior weighted average scores on both primary metrics. It obtains a maj@32 of 63.8 63.8 compared to GRPO’s 60.4 60.4 , a significant improvement of + 3.4 ​ percentage points ​ ( pp ) +3.4\penalty\ \text{percentage points}\penalty\ (\mathrm{pp}) . This aggregate strength is driven by remarkable consistency, as SPO outperforms GRPO on the maj@32 metric across all five benchmarks. The performance gap is most pronounced on BRUMO 25 , where SPO achieves a substantial + 7.3 ​ pp +7.3\penalty\ \mathrm{pp} ( 64.0 64.0 vs. 56.7 56.7 ). Further significant gains are seen on AIME 25 ( + 4.4 ​ pp +4.4\penalty\ \mathrm{pp} ) and HMMT 25 ( + 3.3 ​ pp +3.3\penalty\ \mathrm{pp} points), underscoring the robustness of SPO’s improvements. Notably, these benchmarks have minimal data contamination [ 5 ] , allowing them to serve as a true test of generalization . This demonstrates that our SPO method improves the model’s ability to generalize rather than simply overfit to the training data, a risk exemplified by the DAPO dataset’s strong correlation with AIME 24. While GRPO remains competitive on the avg@32 metric in some cases, SPO’s consistent and significant advantage in maj@32 suggests it learns more robust and repeatable solutions, a key goal for reliable reasoning models.

These findings are mirrored in the pass@ k k performance shown in Figure 2 . The weighted average curve (Figure 2f ) shows a clear and consistent advantage for SPO across all values of k k , translating to an average improvement of approximately 2.4 ​ pp 2.4\penalty\ \mathrm{pp} . While the performance on avg@32 is more competitive on a per-benchmark basis, SPO’s strong overall performance underscores the stability and effectiveness of its learning signal. We provide additional ablation studies on A ∗ A^{*} -PO, SPO with no baseline, and SPO with no offline initialization in Appendix E .

### 5.3 Analysis of Signal Efficiency and Stability

(a) Ineffective Gradient Ratios (b) Advantage Variance Comparison

To empirically assess the architectural advantages of SPO, we conduct a two-part analysis of the unnormalized advantage signals produced by SPO and GRPO (Figure 3 ). First, we quantify complete signal loss arising from degenerate groups. Second, we measure the variance of the remaining learning signals. Together, these metrics characterize each method’s efficiency and stability.

Signal Efficiency and Information Loss. Figure 3a reports the fraction of ineffective samples. For GRPO (blue), the share of samples in degenerate groups rises from roughly 60% to over 80%, yielding zero advantage and no gradient. For SPO, we instead track the proportion of near-zero advantages under two diagnostic tolerances, | A | ≤ τ \lvert A\rvert\leq\tau , with values of τ = 10 − 4 \tau=10^{-4} (red) and τ = 0.02 \tau=0.02 (green). Advantages under the tight tolerance τ = 10 − 4 \tau=10^{-4} remain rare throughout training (red line), while the | A | ≤ 0.02 \lvert A\rvert\leq 0.02 share (green) gradually increases as the value tracker v ^ \hat{v} becomes more accurate and residuals shrink on mastered prompts. This trend is expected and desirable: it reflects accurate prediction rather than signal loss. Unlike GRPO’s degenerate groups, these SPO samples are not discarded, they still produce well-defined gradients and contribute to learning. Notably, even under the stricter τ = 0.02 \tau=0.02 tolerance, SPO’s near-zero ratio remains far below GRPO’s degenerate rate, underscoring SPO’s efficient use of compute.