Skip-Connected Policy Optimization for Implicit Advantage — part 3/4

Cleaned-text code points [29395, 45197). Bibliography omitted here; source.md includes it.

As shown in Table 1 , SKPO achieves strong improvements on Qwen2.5-Math-7B, where it leads in nearly all benchmarks with an average score of 42.5%, surpassing the second-best method CISPO (40.9%) by 1.6 percentage points and outperforming vanilla GRPO (30.2%) by 12.3 percentage points. On Llama-3.2-3B-Instruct, SKPO yields the best overall average of 24.1%, exceeding the second-best DAPO (22.7%) by 1.4 percentage points and GRPO (18.9%) by 5.2 percentage points, despite competitive per-task variation. On In-Domain mathematical benchmarks, SKPO substantially outperforms strong baselines including CISPO and DAPO, demonstrating that the skip-connected architecture effectively integrates fine-grained rewards with group-relative estimation. More importantly, these gains transfer to Out-of-Domain tasks: on MMLU-Pro and LiveCodeBench, SKPO maintains its advantage, demonstrating that the hierarchical reasoning structure generalizes beyond pure mathematics to broader reasoning domains including code generation.

#### Shortcut Validation.

The skip connection [ s , q ] [s,q] concatenates the upstream segment and the original problem to balance upstream influence and downstream exploration freedom. We compare three conditioning strategies: Unconditional generating from q q alone, Continual using s s as a hard prefix, and Skip (ours) conditioning on [ s , q ] [s,q] . Using base models (Qwen2.5-Math-7B and Llama-3.2-3B-Instruct) without any RL training, we first generate complete responses for each problem, then extract segments s s at six relative positions spanning from 1 6 \frac{1}{6} to 1 2 \frac{1}{2} of the total response length. We then sample G = 8 G=8 downstream continuations under each conditioning strategy on a mixture of all mathematical evaluation benchmarks (AIME24, AIME25, AMC, MATH, Minerva, and Olympiad). We measure three metrics: Diversity as the average number of distinct final answers among 8 rollouts per problem; Advantage Zero Rate as the probability that all 8 rollouts yield identical outcomes (all correct or all incorrect), which collapses the advantage signal to zero; and Response Length as the generation length excluding the conditioning prefix. As shown in Figure 3 , Continual conditioning progressively constrains exploration as segments extend, resulting in lower diversity, higher advantage zero rate, and shorter generation length. In contrast, Skip maintains diversity and advantage zero rate comparable to Unconditional, demonstrating that the skip-connected structure effectively preserves downstream exploration freedom while incorporating upstream guidance.

### 4.2 Ablation Studies

We conduct ablation studies to validate the effectiveness of key components in SKPO. We compare SKPO against four variants: ❶ w/o Skip : The upstream segment s s is directly used as a prefix for downstream generation without the skip connection to the original problem. ❷ w/o MC Estimation : The upstream stage receives rewards from its own complete generation without aggregating downstream outcomes. ❸ Split@Fixed : Fixed split position alternating between 1 6 \frac{1}{6} and 1 2 \frac{1}{2} across training steps, replacing our adaptive sampling from [ 1 6 ​ L q , 1 2 ​ L q ] [\frac{1}{6}L_{q},\frac{1}{2}L_{q}] . ❹ Selection:Extreme : Select the upstream s s with minimum or maximum negative log-probability alternating across steps, instead of the median. The results are summarized in Table 2 .

The results in Table 2 validate all key components. w/o Skip leads to a significant performance drop: without the shortcut connection to the original problem, downstream responses sampled along the same upstream path explore a constrained space, substantially increasing the probability of homogeneous outcomes and advantage collapse to zero. w/o MC Estimation also underperforms: when the upstream stage receives rewards only from its own complete generation rather than aggregating downstream outcomes, it cannot learn reasoning patterns that specifically facilitate downstream solving. For segmentation position, Split@Fixed shows degraded performance: alternating between fixed positions loses the benefit of adaptive sampling, as splitting too early provides insufficient upstream context while splitting too late reduces downstream exploration freedom. Our adaptive sampling from [ 1 6 ​ L q , 1 2 ​ L q ] [\frac{1}{6}L_{q},\frac{1}{2}L_{q}] dynamically balances both considerations. For upstream selection, Selection:Extreme alternating between min and max perplexity underperforms because extreme selections either represent overconfident but potentially flawed reasoning (min) or uncertain responses lacking useful guidance (max). The median strategy avoids both extremes, selecting a representative upstream segment that facilitates diverse downstream exploration.

### 4.3 Implicit Advantage Analysis

Following the Monte Carlo trajectory sampling approach for implicit reward estimation, we design an analysis to reveal how different optimization objectives shape intermediate reasoning quality. We collect all correct responses from SKPO, CISPO, and DAPO across all mathematical test benchmarks (AIME24, AIME25, AMC, MATH, Minerva, and Olympiad), splitting each into 100 segments based on relative token position. We select CISPO and DAPO as comparison baselines because they achieve relatively strong performance among all methods in our main experiments. For each segment, we use GPT-5-nano as the external evaluator to perform Monte Carlo continuation sampling 8 times (see Appendix D for the prompt template), obtaining segment-level reward estimates that reflect “how easy is it to reach a correct answer from this intermediate state.” We then compute inter-method relative advantages at each position following A ^ method = ( R method − R ¯ ) / σ R \hat{A}_{\text{method}}=(R_{\text{method}}-\bar{R})/\sigma_{R} , where the comparison group consists of all methods at the same relative position for the same problem. We choose GPT-5-nano because its cost is acceptable and its mathematical reasoning capability is sufficient for producing meaningful visualizations.

#### Why External Evaluator?

A natural alternative would be mutual evaluation among trained models. However, since rewards derive from Monte Carlo sampling, models trained on identical data with only algorithmic differences exhibit similar continuation distributions, making inter-method differences difficult to detect without prohibitively large sample sizes. Using a substantially stronger external model addresses this: its higher capability provides a wider dynamic range in reward estimates, amplifying subtle quality differences in intermediate reasoning that would otherwise be obscured by sampling variance.

As visualized in Figure 4 , responses from SKPO consistently maintain significantly higher advantages in the early-to-middle stages (20–50% position range) compared to those from CISPO and DAPO, across both model families. This cross-model consistency reinforces the finding: SKPO establishes superior structural foundations during the critical problem-setup phase, where our hierarchical optimization explicitly provides learning signals. The convergence of all curves toward zero at trajectory end is expected, as outcome correctness becomes determined and intermediate advantages diminish.

## 5 Conclusion

We identify a key challenge in integrating fine-grained rewards with group-relative optimization: insufficient Monte Carlo sampling for token-level reward estimation degrades performance below outcome-only baselines. To address this, we propose SKPO, which partitions reasoning into upstream and downstream phases via a skip-connected architecture. The upstream phase generates early-stopped reasoning segments; the downstream phase performs trajectory sampling from the skip connection of the segment and original problem, preserving upstream influence while maintaining exploration freedom. We employ asymmetric optimization: single-stream algorithms for the sample-constrained upstream phase and group-relative algorithms for the downstream phase. Experiments show SKPO outperforms strong baselines on In-Domain mathematics and generalizes to Out-of-Domain tasks. Our off-policy continuation analysis reveals an implicit advantage phenomenon: SKPO-generated segments exhibit systematically higher advantages in intermediate steps when comparing correct responses across methods, indicating more robust forward reasoning patterns.

#### Limitations.

Our study still has several limitations. First, the main training experiments are conducted on 3B and 7B models with mathematical RLVR data, so broader validation on larger scales and non-math training domains remains future work. Second, SKPO currently instantiates Monte Carlo sampling at a single split point sampled from a restricted range; richer multi-split or later-split designs may further improve granularity at higher cost. Third, our implicit-advantage analysis relies on an external evaluator with instruction-based continuation rather than raw prefix completion, so the resulting reward estimates should be interpreted comparatively rather than as absolute values. Finally, our single-pass KV-cache rewriting is an efficiency optimization rather than a correctness requirement, but reproducing the most efficient version still requires custom inference-engine support.

Future work can explore later split positions with higher resampling costs for finer granularity, and integration with multi-stage RL for agentic tasks.

## Impact Statement

This paper presents work whose goal is to advance the field of Deep Learning, specifically in reinforcement learning for large language models. The proposed method aims to improve sample efficiency and training stability, which could reduce computational costs and environmental impact. However, there are potential risks if more capable reasoning models are used for malicious purposes. We encourage responsible development and deployment of such technologies.

## Appendix A Why Fine-Grained Rewards Fail with Group-Relative Estimation

As claimed in Section 1 , directly integrating fine-grained rewards with group-relative advantage estimation leads to a fundamental sign inconsistency problem: with limited Monte Carlo samples, estimated advantages frequently have incorrect signs, causing the optimization to reinforce wrong behaviors or penalize correct ones. This appendix provides simulation evidence quantifying this phenomenon. The severity of this problem explains why existing methods circumvent rather than solve it. For example, PRIME ( Cui et al., 2025 ) aggregates token rewards to response-level before computing group-relative advantages, while VSRM ( Yue et al., 2025 ) abandons group-relative estimation entirely in favor of PPO. SKPO takes a different approach: structurally partitioning the problem so that upstream optimization avoids group-relative computation while downstream optimization has sufficient samples for stable estimation.

### A.1 Position-Dependent Variance

The effectiveness of group-relative advantage estimation varies dramatically with trajectory position. For very early tokens (problem restatement and initial setup), the reward is largely determined by the model’s own policy distribution rather than the specific tokens generated, as the reasoning space has not yet been meaningfully pruned. Trajectory sampling from these positions yields high variance, but this variance is symmetric across methods, explaining why our implicit advantage analysis shows similar rewards at the 0–20% position range. Intermediate tokens (the decisive reasoning steps) are where methods diverge. Quality differences emerge as some reasoning paths make productive progress while others take detours or make errors. However, Monte Carlo estimation at these positions suffers from prohibitive variance: distinguishing a 55% success probability prefix from a 60% one requires hundreds of samples. This is precisely where SKPO’s implicit advantage emerges, as our upstream-downstream structure provides stable optimization signals for these intermediate segments. For terminal tokens (final computation and answer extraction), the outcome is semantically determined; sampling variance is minimal and single samples provide reliable gradients, so all methods perform comparably.

### A.2 Simulation Setup and Results

We simulate a group of G = 8 G=8 trajectories where each prefix has a true success probability p i p_{i} . Advantages are estimated by sampling N N continuations per prefix and applying group-relative normalization. An estimate is considered correct if the signs of all G G advantages match their ground-truth signs. Simulation results in Figures 5 and 6 demonstrate that sample efficiency degrades rapidly as group quality spread narrows. When reasoning steps provide only incremental improvements, even N = 512 N=512 samples fail to yield 50% probability of correct sign assignment for the entire group. Individual prefix analysis in Figure 6(a) confirms that prefixes near the group mean are most vulnerable to noise-induced sign flips, which are precisely the prefixes that represent incremental but genuine reasoning progress.

### A.3 Correctness Distribution Analysis

To further illustrate the sign inconsistency problem, we systematically examine how advantage estimation accuracy varies across different correctness distributions within a group of G = 8 G=8 responses. We sample random trajectories and categorize them by the number of correct responses (from 0/8 to 8/8), visualizing the true token rewards, advantages, and estimation accuracy for each distribution type. Representative examples are shown in Figure 7 .

The results reveal a striking pattern: reliable advantage estimation occurs only in extreme cases . When all responses converge to the same outcome (0/8 or 8/8), both GRPO and MC estimation achieve near-perfect sign accuracy, as there is no meaningful quality distinction to be made within the group. However, this apparent success is misleading: although sign accuracy appears high, the MAE between MC and true advantages remains comparable to GRPO in these degenerate cases, since all advantages are effectively zero and provide no learning signal.

In contrast, the informative mixed cases (1/8 through 7/8) exhibit persistent sign inconsistency problems: • Early tokens : Sign accuracy frequently drops below 50%, meaning random guessing would perform equally well.

• Trajectory crossings : When reward trajectories of different-outcome responses cross (common in early-to-mid reasoning), MC estimation cannot reliably distinguish which response is on a better path.

• Narrow spreads : Cases near 4/8 (balanced correct/incorrect) show the worst estimation quality, as the group quality spread is minimal.

However, as shown in Figure 8 , with sufficient sampling budget, MC estimation does eventually surpass GRPO. The figure demonstrates that as K K increases from 8 to 8192, MC sign accuracy improves steadily, and the Mean Absolute Error (MAE) between estimated and true advantages decreases substantially. At K = 8192 K=8192 , MC achieves near-perfect sign accuracy with MAE approaching zero, significantly outperforming GRPO’s constant-advantage approximation. This confirms that the theoretical advantage of fine-grained credit assignment is real; the challenge lies purely in the prohibitive sampling cost required to realize it.

This analysis underscores our central claim: fine-grained Monte Carlo estimation with practical sampling budgets only works reliably when it provides no useful signal, and fails precisely when fine-grained credit assignment would be most valuable.

### A.4 How SKPO Structurally Partitions the Problem