Single-stream Policy Optimization — part 4/4
Cleaned-text code points [45740, 51334). Bibliography omitted here; source.md includes it.
• maj@ k k : the correctness of the majority-voted answer [ 37 ] . This metric first identifies the most frequent answer among k k responses for each problem. The score is 1 if that modal answer is correct, and 0 otherwise. Let a i , j a_{i,j} be the final answer string for the j j -th response to problem i i , and let r ( ⋅ ) r(\cdot) be the reward function for a given answer string. The metric is defined as: maj@ k = 1 M ∑ i = 1 M r ( mode { a i , j } j = 1 k ) . \text{maj@}k=\frac{1}{M}\sum_{i=1}^{M}r\left(\operatorname{mode}\{a_{i,j}\}_{j=1}^{k}\right).
## Appendix E Ablation Studies
We conduct a series of ablation studies to dissect the core components of SPO and validate our design choices. To facilitate efficient experimentation, these studies are performed under a streamlined setting compared to our main experiments. Specifically, we utilize a batch size of 256 256 prompt-response pairs, and the model is updated with 4 4 gradient steps for each collected batch. All ablation results are reported on the AIME 25 benchmark, using the avg@16 metric with a maximum generation length of 16,384 16,384 tokens.
(a) SPO vs. A ∗ A^{*} -PO (b) Baseline Ablation (c) Offline Initialization Ablation
SPO vs . A ∗ A^{*} -PO . This experiment, presented in Figure 6a , compares our proposed SPO with A ∗ A^{*} -PO [ 6 ] . A ∗ A^{*} -PO utilizes a static baseline derived from a pre-computed optimal value function, V ∗ V^{*} , which is tied to the KL-regularized objective with respect to an initial reference policy, π ref \pi_{\text{ref}} . While this approach is highly efficient, its central assumption may be challenged in tool-calling scenarios. In these tasks, learning involves acquiring new functional capabilities, leading to a significant policy drift where the learned policy, π t \pi_{t} , diverges substantially from π ref \pi_{\text{ref}} . Consequently, the pre-computed V ∗ V^{*} may become a less representative baseline for the current policy’s true value function, V π t V_{\pi_{t}} , potentially affecting the accuracy of the advantage estimates. In contrast, SPO’s baseline is adaptive, dynamically tracking an estimate of V π t V_{\pi_{t}} as the policy evolves. The empirical results, which show SPO’s superior performance, suggest that this adaptability is crucial. By maintaining a baseline that remains relevant to the current policy, SPO provides a more stable and effective learning signal in environments that demand significant policy evolution. Finally, from a practical perspective, π ref \pi_{\text{ref}} computation during A ∗ A^{*} -PO policy update occupies an extra trunk of GPU memory, making it less appealing than the proposed SPO algorithm.
Baseline Ablation . Figure 6b presents a crucial ablation that validates the fundamental principle of using a baseline for variance reduction. In this experiment, we remove the value tracker component v ^ − 1 ( x ) \hat{v}_{-1}(x) from the advantage calculation, causing the algorithm to rely solely on the globally batch-normalized raw reward r ( x , y ) r(x,y) as its learning signal. However, the substantial performance degradation observed is a classic illustration of the remaining challenges. While global normalization effectively controls the overall scale of rewards, the raw reward signal is still noisy on a per-sample basis as it fails to account for prompt-specific difficulty. SPO’s history-informed baseline is designed to subtract this expected difficulty, thereby effectively reducing variance and providing a much cleaner, more reliable gradient for learning. This experiment confirms that the adaptive value tracker is the most critical component for SPO’s success, directly addressing the core challenge of variance in single-stream policy optimization.
Offline Initialization Ablation . In Figure 6c , we analyze the impact of the value tracker’s initialization phase. The standard SPO algorithm initializes the value tracker with estimates computed from a small set of n 0 n_{0} offline samples, giving it a “warm start”. The ablation removes this step, forcing the tracker to learn from scratch online. The results clearly demonstrate the benefit of the offline initialization. Without it, the tracker begins with a highly inaccurate baseline, leading to high-variance gradients and significant instability in the initial training phase, as evidenced by the performance dip. Although the model eventually recovers, it fails to reach the same level of performance as the properly initialized model, underscoring the importance of a good initial value estimate for stable and effective learning.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .