Skip-Connected Policy Optimization for Implicit Advantage

arxiv:2604.08690

Original source · Paper JSON

Post-training · 2026-04-09

Cites (14) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Skip-Connected Policy Optimization for Implicit Advantage
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• 2.1 Group-Based Policy Optimization
• 2.2 Fine-Grained Reward/Advantage
• 2.3 Multi-Stage Pipeline Designs
• 3 Methodology
• 3.1 Monte Carlo Trajectory Sampling
• Ideal Token-Level Reward Estimation for Analysis.
• Upstream-Downstream Instantiation: A Cost-Effective Step Beyond.
• 3.2 Asymmetric Optimization Strategy
• Downstream Optimization: Group Relative Policy Optimization.
• Upstream Optimization: Single-stream Policy Optimization.
• 4 Experiments
• Training Setup.
• 4.1 Main Results
• Shortcut Validation.
• 4.2 Ablation Studies
• 4.3 Implicit Advantage Analysis
• Why External Evaluator?
• 5 Conclusion
• Limitations.
• Impact Statement
• Appendix A Why Fine-Grained Rewards Fail with Group-Relative Estimation
• A.1 Position-Dependent Variance
• A.2 Simulation Setup and Results
• A.3 Correctness Distribution Analysis
• A.4 How SKPO Structurally Partitions the Problem
• Appendix B Training Dynamics Analysis
• B.1 Performance Metrics
• B.2 Optimization Stability
• Entropy Analysis.
• Training Time.
• Appendix C Experimental Setup and Hyperparameters
• Dataset.
• Model Architectures.
• Optimization Hyperparameters.
• SKPO Configuration.
• Baseline Configurations.
• Appendix D Prompt for Implicit Advantage Analysis
• Why Instruction-Based Continuation?
• Instructions for reporting errors