Skip-Connected Policy Optimization for Implicit Advantage — part 1/4
Cleaned-text code points [0, 13842). Bibliography omitted here; source.md includes it.
# Skip-Connected Policy Optimization for Implicit Advantage ###### Abstract Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness. ###### Keywords: ## 1 Introduction Reinforcement Learning with Verifiable Rewards (RLVR) ( Shao et al., 2024 ; Hu et al., 2025 ; Ahmadian et al., 2024 ; Li et al., 2024 ) has emerged as the dominant paradigm for enhancing LLM reasoning. Among these methods, the representative Group Relative Policy Optimization (GRPO) achieves strong training results through outcome-based rewards and within-group relative advantage computation with moderate sampling budgets. While this coarse-grained binary design demonstrates stability, fine-grained dense rewards and advantages remain theoretically appealing for their higher performance upper bounds; yet limited inference budgets pose a practical challenge: how to make stable progress toward the upper bound promised by dense fine-grained rewards and advantages, under the existing RLVR paradigm . Our preliminary analysis reveals a counterintuitive finding that partially explains this dilemma. By employing Monte Carlo sampling to estimate step-level rewards through sampling multiple continuations from existing reasoning and aggregating their outcome rewards ( Yue et al., 2025 ) , we obtain the results shown in Figure 1 . Even with substantial sampling costs, MC estimation combined with group-relative advantage computation yields high-variance advantages with frequent sign errors for early reasoning tokens, stably underperforming outcome-only GRPO. This finding may explain why existing methods must make various trade-offs between fine-grained and coarse-grained, dense and sparse, to circumvent this estimation challenge. Existing methods make different sacrifices in their designs. ❶ Multi-stage methods represented by reflection and ensembling ( Gandhi et al., 2025 ) sacrifice fine-grained information: while they structure reasoning into explicit steps, step quality still reduces to outcome correctness, i.e., whether reflection corrects to the right answer or the ensemble converges to the correct solution. ❷ Fine-grained reward methods face a similar dilemma. Algorithm-based approaches typically either aggregate token rewards within responses to compute group-relative advantages and subsequently broadcast these to every token (which is equivalent to dense outcome-level methods), or entirely sacrifice group-relative estimation in favor of algorithms like PPO. Monte Carlo estimation, while more interpretable and offering higher upper bounds for fine-grained refinement, requires prohibitive sampling for reliable advantage estimation. ➤ In summary, existing approaches face a structural dilemma between fine-grained validity and computational efficiency . To break this dilemma, we propose SK ip-connected P olicy O ptimization (SKPO), which decomposes reasoning into upstream and downstream phases. The key insight is that by instantiating Monte Carlo sampling at a single intermediate position, we can obtain dense rewards for upstream while reusing the same samples for group-relative optimization in downstream. Specifically: ❶ The upstream phase generates early-stopped reasoning segments and receives Monte Carlo rewards aggregated from downstream continuations, using single-stream optimization ( Xu and Ding, 2025 ) to reduce instability on fine-grained signals by lowering upstream per-prompt sample count and estimating more stable baselines from historical samples. ❷ The downstream phase maintains group-relative optimization with a skip connection [ s , q ] [s,q] , concatenating both the upstream segment and the original problem to balance exploration freedom and upstream influence. ❸ An engineering optimization achieves computational parity with GRPO through single-pass rollout. Interestingly, this architecture also yields an implicit advantage : models trained with SKPO generate trajectories with significantly higher intermediate-step quality even when matched for final outcome correctness. This benefit emerges from the fine-grained optimization structure without external supervision. Our contributions are: • We identify why fine-grained Monte Carlo rewards fail under practical budgets: high-variance advantages whose signs frequently flip, underperforming outcome-only GRPO. • We propose SKPO, decomposing reasoning into upstream and downstream phases, instantiating Monte Carlo sampling at a single intermediate position to provide dense rewards for upstream, while using a skip connection [ s , q ] [s,q] that enables downstream to leverage upstream reasoning while preserving exploration freedom for group-relative optimization. • Experiments show substantial improvements on math benchmarks with robust OOD generalization. Analysis reveals an implicit advantage : higher intermediate-step quality emergent from the optimization structure. ## 2 Related Work ### 2.1 Group-Based Policy Optimization Group Relative Policy Optimization (GRPO) has emerged as a dominant paradigm for reinforcement learning from verifiable rewards ( Shao et al., 2024 ; Guo et al., 2025 ; Yu et al., 2025b ; Zheng et al., 2025 ; Wu et al., 2025 ; Yao et al., 2025 ; Jain et al., 2025b ) . By sampling multiple responses within a group and computing relative advantages, GRPO eliminates the need for explicit value models while effectively guiding policy updates. Recent research has systematically explored and optimized various components within GRPO ( Khatri et al., 2025 ; He et al., 2025b ) . Variants of GRPO explore different combinations of design choices within the framework. ❶ Reward and advantage computation encompasses methods that improve sample efficiency through decoupled reward modeling and modified advantage computation, while also modeling the influence of entropy mechanisms and step-level fine-grained information on training dynamics ( Ahmadian et al., 2024 ; Liu et al., 2025b ; Xiong et al., 2025 ; Li et al., 2025a ; Jain et al., 2025b ; He et al., 2025c ; He et al., 2025a ; Chen et al., 2025d ) . ❷ Policy ratio design and aggregation includes clipping mechanisms that enhance training stability through dynamic token-level bounds and gradient-preserving designs ( Yu et al., 2025b ; Yang et al., 2025b ) , as well as normalization strategies that stabilize policy updates through ratio normalization and geometric mean aggregation, or reduce sensitivity to outliers through group expectation sampling ( Zhao et al., 2025 ; Zhang et al., 2025a ; Sheng et al., 2025 ) . ❸ Alternative baseline estimation addresses the limitation that group-relative baselines require multiple concurrent samples per prompt. SPO ( Xu and Ding, 2025 ) replaces the spatial baseline (group mean from concurrent rollouts) with a temporal baseline via per-prompt value trackers, enabling optimization when only a single sample is available, which motivates our adoption of temporal baselines for upstream optimization where only a single segment is generated per prompt. Among these design choices, we are particularly interested in the construction of step-level information. ### 2.2 Fine-Grained Reward/Advantage A well-known characteristic of vanilla GRPO is its uniform credit assignment: the same advantage is distributed equally across all tokens in a response, which may lead to length bias and inefficiency ( Liu et al., 2025a ) . Recent work addresses this limitation through two primary paradigms. ❶ Algorithm-based methods derive dense reward signals from existing rollouts through algorithmic formulations rather than additional sampling, including reformulating the Bellman equation to decompose action-level Q-values into token-level components via soft updates, and synthesizing process rewards from policy log-ratios, entropy weights, and learnable token preferences ( Wen et al., 2024 ; Chen et al., 2025b ; Cui et al., 2025 ; Tan et al., 2025 ; Xie et al., 2025 ; Deng et al., 2025 ; Li et al., 2025b ; Liu et al., 2025a ; Lee and Tong, 2025 ) ; other works infer implicit process rewards directly from policy rollouts without explicit reward model training ( Fan et al., 2025 ; Fei et al., 2025 ; Sullivan, 2025 ; Prakash and Buvanesh, 2025 ; Ma et al., 2025 ) ; additionally, some methods selectively optimize only high-entropy tokens or identify pivotal reasoning steps for targeted credit assignment ( Wang et al., 2025b ; Wang et al., 2025a ; Yu et al., 2025a ) . ❷ Sampling-based methods estimate segment-level or token-level values through additional Monte Carlo sampling at intermediate states, utilizing vine sampling at segment boundaries, tree-based trajectory organization, or repeated rollouts to obtain unbiased value estimates ( Kazemnejad et al., 2024 ; Brantley et al., 2025 ; Lin et al., 2025 ) . Algorithm-based approaches achieve fine-grained credit assignment with minimal overhead, while sampling-based methods provide accurate estimation at a computational cost proportional to the number of estimation points. However, the instability introduced by fine-grained information may outweigh its benefits compared to coarse-grained but stable optimization over long training processes. ### 2.3 Multi-Stage Pipeline Designs GRPO-based methods also integrate with pipeline designs to enhance training effectiveness ( Yan et al., 2025 ; Chen et al., 2025a ) . Beyond single-response optimization discussed above, multi-stage pipelines involve multiple responses or attempts to solve a task. ❶ Structured decomposition exploits multi-response generation to improve credit assignment, organizing multiple rollouts into search trees to track retrospective credit and identify pivotal nodes, assigning credit to high-level thought templates across hierarchical reasoning traces, or extending Monte Carlo sampling with tree-based trajectory organization for unbiased value estimation over multiple attempts ( Zhang et al., 2025c ; Yang et al., 2025a ; Kazemnejad et al., 2024 ; Brantley et al., 2025 ) . ❷ Internal feedback constructs multi-turn refinement loops where models learn from their own previous outputs, employing multi-stage RL training with reward shaping to prevent behavior collapse, using layered generation processes where later layers explicitly correct earlier outputs under implicit process rewards, or introducing verify-then-revise mechanisms that alternate between policy and generative verifier roles ( Kumar et al., 2025 ; Ding et al., 2025 ; Jiang et al., 2025 ; Ekbote et al., 2025 ; Li et al., 2025c ) . ❸ External signals guide multi-stage reasoning through curriculum or data generation, utilizing challenger-solver co-evolution patterns that generate questions at the solver’s capability frontier, formulating curriculum selection as non-stationary multi-armed bandits for adaptive difficulty scheduling, updating model parameters during inference using pseudo-labels from majority voting, or enabling self-play through questioner-responder-verifier role alternation ( Huang et al., 2025 ; Chen et al., 2025c ; Zuo et al., 2025 ; Yang et al., 2025c ) . Our work differs from these approaches in that, while adopting an upstream-downstream two-stage generation pipeline, the actual inference cost does not exceed a single rollout, making it directly comparable to on-policy single-rollout methods rather than multi-stage approaches, while similarly introducing no external information to guide the policy model. Meanwhile, we still adopt a sampling-based approach to obtain fine-grained dense rewards, but by instantiating upstream-downstream sampling at a single position and aligning with the group sampling count, we incur no additional cost while achieving both efficient sampling and stage-wise information density. ## 3 Methodology Figure 2 illustrates the overall algorithm. This section formalizes the key components of SKPO: Section 3.1 first analyzes ideal token-level Monte Carlo reward estimation, explaining why early-position estimation suffers from high variance while late-position estimation naturally converges to outcome rewards, a property that justifies GRPO’s conservative broadcasting as a reasonable baseline. We then present the upstream-downstream architecture that instantiates Monte Carlo sampling at a single position, detailing the dual functions of the skip connection [ s , q ] [s,q] , and describe an engineering optimization that achieves computational parity with GRPO. Section 3.2 formalizes the asymmetric optimization strategy with explicit policy gradient objectives for both phases. ### 3.1 Monte Carlo Trajectory Sampling #### Ideal Token-Level Reward Estimation for Analysis.