Skip-Connected Policy Optimization for Implicit Advantage — part 4/4

Cleaned-text code points [45197, 56129). Bibliography omitted here; source.md includes it.

These findings explain why SKPO’s architectural design is effective. In the upstream stage, by using single-stream policy optimization (SPO) with temporal baselines instead of group-relative advantages, SKPO entirely avoids the sign inconsistency problem for intermediate reasoning segments. The temporal baseline V ^ ​ ( q ) \hat{V}(q) provides stable value estimates without requiring multiple concurrent samples. In the downstream stage, with G G rollouts per upstream segment, group-relative estimation becomes reliable. The downstream responses are closer to terminal tokens where outcome variance is naturally lower, and the group size provides sufficient samples for stable advantage computation. This structural partitioning (not circumvention) enables SKPO to genuinely optimize at the segment level, which is why our advanced off-policy analysis can reveal implicit advantage in intermediate reasoning steps that other methods cannot observe.

## Appendix B Training Dynamics Analysis

We analyze the optimization behavior across different training configurations, examining convergence patterns, policy stability indicators, and computational efficiency metrics throughout the training process. All training curves presented in this section are from experiments on Qwen2.5-Math-7B base model. For visual clarity, we select a representative subset of methods in each figure rather than displaying all baselines.

### B.1 Performance Metrics

Figure 9 shows the mean accuracy (Mean@32) across training, which reflects the average quality across all sampled responses, serving as an indicator of consistent behavior and sample efficiency. SKPO demonstrates superior convergence compared to baseline methods, reaching higher asymptotic performance while maintaining stable training trajectories. Furthermore, SKPO exhibits faster initial improvement, which we attribute to the dense reward signals provided by averaging downstream outcomes for upstream segment optimization; traditional outcome-based methods receive sparse feedback only at trajectory endpoints, while SKPO’s segment-level rewards provide intermediate reward signals that accelerate policy learning in the critical early training phase.

### B.2 Optimization Stability

#### Entropy Analysis.

Figure 10 shows the actor policy entropy across training. Most methods exhibit decreasing entropy over the 500-step training schedule, with DAPO being the exception due to its explicit entropy regularization. This entropy reduction is a common behavior as policies converge toward higher-reward solutions. During our exploration of various algorithmic components, we also designed and experimented with some entropy-increasing mechanisms. Unfortunately, we found no clear correlation between entropy trends and final performance, suggesting that our method may not be well-suited for improvements in this direction.

#### Training Time.

Figure 11 compares the wall-clock training time per optimization step across methods. Critique-GRPO serves as a representative example of two-batch approaches: it first generates 7 direct rollouts, then performs a separate batch dispatch to generate 1 self-critique response conditioned on the best direct response. This sequential processing incurs substantial overhead. A naive two-batch implementation of SKPO would similarly suffer from this overhead. Our engineering optimization eliminates this by modifying the vLLM inference engine to perform mid-generation KV cache rewriting within a single continuous rollout, achieving computational parity with GRPO-family baselines.

## Appendix C Experimental Setup and Hyperparameters

This section provides comprehensive details on the experimental configuration, training procedures, and evaluation protocols used throughout this work. All experiments are designed to ensure fair comparison across methods and reproducibility of results.

#### Dataset.

We train all models on a curated mathematical reasoning dataset containing 17,398 problems after deduplication (dapo-math-17k). The training data consists of diverse mathematical problems spanning arithmetic, algebra, geometry, number theory, and competition mathematics. We evaluate on standard benchmarks including AIME 2024/2025 (American Invitational Mathematics Examination), AMC (American Mathematics Competition), MATH, Minerva Math, and Olympiad-level problems. For out-of-domain evaluation, we use MMLU-Pro (general reasoning across multiple subjects) and LiveCodeBench (code generation tasks). All training data is formatted with a consistent prompt structure, and responses are truncated to fit within maximum length constraints.

#### Model Architectures.

We conduct experiments on two base models: Qwen2.5-Math-7B , a 7-billion parameter language model specialized for mathematical reasoning, and Llama-3.2-3B-Instruct , a 3-billion parameter instruction-tuned model. Both models use the same tokenizer as their respective base model families. For SKPO, the upstream and downstream phases share the same model parameters, with only the input conditioning (original problem q q vs. augmented input [ s , q ] [s,q] ) differing between phases.

#### Optimization Hyperparameters.

We train all models for 500 optimization steps with a batch size of 128 prompts per step. The learning rate is set to 1 × 10 − 6 1\times 10^{-6} with linear warmup over the first 10 steps and no decay thereafter. We use the AdamW optimizer with weight decay 0.1 0.1 , β 1 = 0.9 \beta_{1}=0.9 , and β 2 = 0.999 \beta_{2}=0.999 . Gradient norms are clipped at 1.0 1.0 to improve training stability. We employ mixed-precision training (bfloat16) to accelerate computation and reduce memory usage. The policy gradient mini-batch size is set to 32 prompts, meaning each optimization step processes 4 mini-batches ( 128 / 32 = 4 128/32=4 ). Loss aggregation uses token-mean normalization, where gradients from all tokens contribute equally after normalizing by response length.

#### SKPO Configuration.

For SKPO, we set the downstream group size to G = 8 G=8 responses per upstream segment, providing sufficient samples for stable Monte Carlo reward estimation. The upstream segment termination position is sampled uniformly from [ 1 6 ​ L p , 1 2 ​ L p ] [\frac{1}{6}L_{p},\frac{1}{2}L_{p}] , where L p L_{p} is the per-prompt average response length maintained using the same SPO-style KL-adaptive update mechanism as the value tracker (initialized to 1024 tokens). This statistical approach adapts to varying problem complexity without relying on formatting conventions; maximum segment length is implicitly bounded by the maximum response length (3072 tokens). The asymmetric optimization strategy employs different configurations for each phase. Upstream (SPO) uses temporal baseline V ^ ​ ( q ) \hat{V}(q) with KL-adaptive forgetting factor, with ρ min = 0.875 \rho_{\min}=0.875 , ρ max = 0.96 \rho_{\max}=0.96 , and half-life τ 1 / 2 = 8.0 \tau_{1/2}=8.0 steps. Unlike the original SPO which collects n 0 n_{0} samples before initializing the tracker, we initialize the baseline to zero (neutral in reward space) and update immediately upon the first observation for practical training efficiency; we enable prioritized sampling with ϵ = 0.05 \epsilon=0.05 for curriculum learning; batch size is 128 prompts × \times 1 response = 128 samples. Downstream (GRPO variant) uses group-relative advantages with importance sampling clipping, with clipping threshold ε = 0.2 \varepsilon=0.2 for the standard range [ 0.8 , 1.2 ] [0.8,1.2] and an additional upper bound at 1.28 1.28 (Clip-Higher); batch size is 128 prompts × \times 8 responses = 1024 samples. We balance upstream and downstream objectives with equal weights ( w up = w down = 0.5 w_{\text{up}}=w_{\text{down}}=0.5 ), normalize advantages using batch statistics ( A ~ = ( A − μ ℬ ) / σ ℬ \tilde{A}=(A-\mu_{\mathcal{B}})/\sigma_{\mathcal{B}} ), and disable KL penalty ( β = 0 \beta=0 ) and entropy regularization ( α ent = 0 \alpha_{\text{ent}}=0 ) to focus purely on outcome-based optimization.

#### Baseline Configurations.

All baselines use the same training schedule (500 steps), batch size (128 prompts), learning rate ( 1 × 10 − 6 1\times 10^{-6} ), and base model to ensure fair comparison. Since SPO generates only 1 rollout per prompt, its batch size is scaled to 1024 prompts to match the total sample count of other methods with group size 8. Baseline-specific hyperparameters follow their respective original papers.

## Appendix D Prompt for Implicit Advantage Analysis

For the implicit advantage analysis in Section 4.3 , we use an external model (GPT-5-nano) to perform Monte Carlo continuation sampling. The prompt used for continuation is shown below:

where {problem} is the mathematical problem statement, and {segment} is the partial reasoning from the trained model at a specific relative position. The external model is instructed to continue from exactly where the partial solution left off, without repeating prior work. By sampling multiple continuations and computing the success rate, we obtain the segment-level reward estimate that reflects “how easy is it to reach a correct answer from this intermediate state.”

#### Why Instruction-Based Continuation?

Ideally, one would simply concatenate the problem and partial response as a prefix, then let the model generate completions directly. However, current commercial LLM APIs do not support raw text completion without chat templates, as they require messages formatted according to their instruction-following interface. Therefore, we cannot perform true prefix-based continuation. Instead, we rely on the external model’s instruction-following capability by explicitly instructing it to “continue from exactly where this solution left off” and “do NOT repeat any work already done.” This approach leverages the strong instruction-following abilities of advanced models to approximate the desired continuation behavior.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .