iGRPO: Self-Feedback-Driven LLM Reasoning

arxiv:2602.09000

Original source · Paper JSON

Post-training · 2026-02-09

Cites (7) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• iGRPO: Self - Feedback–Driven LLM Reasoning
• Abstract
• 1 Introduction
• 2 Related Work
• RL for Reasoning.
• GRPO and Variants.
• LLM Self-Learning and Self-Improvement.
• 3 Methodology
• 3.1 Background: Group Relative Policy Optimization
• 3.2 Iterative Group Relative Policy Optimization
• 3.2.1 Motivation: From Static Examples to Dynamic Self-Conditioning
• 3.2.2 Algorithmic Framework
• Stage 1: Exploratory Draft Generation.
• Stage 2: Conditioned Refinement.
• 3.2.3 Theoretical Analysis: Bootstrapped Policy Improvement
• Definition 1 (Self-Conditioned Prompt Construction) .
• Proposition 3.1 (Progressive Conditioning Quality for Binary Rewards) .
• Proof.
• 3.2.4 Mathematical Formulation
• Stage 1: Draft Selection.
• Stage 2: Conditioned Generation and Advantage Computation.
• Full Objective.
• Reward Function.
• 3.2.5 Computational Analysis
• Baseline GRPO Cost.
• iGRPO Rollout Cost.
• 4 Experiments
• 4.1 Setup
• 4.2 Results
• Controlled study with matched sampling budget.
• Generalist 8B model: the largest gains from self-feedback.
• Stronger 7B distilled reasoner: consistent gains concentrated on multi-step tasks.
• Math-specialized 7B model: improvements persist when the base is already strong.
• Scaling to 14B parameters: benefits persist on complex reasoning.
• Competitiveness against critique-style objectives.
• 4.3 Generalization to a Stronger Base and Harder Dataset
• 5 Ablation
• Beyond GRPO: self-feedback as a reusable refinement wrapper.
• Generative judge study.
• Entropy analysis.
• 6 Conclusion
• Appendix
• Appendix A Policy Gradient Derivation for iGRPO
• A.1 Two-stage sampling and the induced self-conditioned prompt distribution
• A.2 From the self-conditioned expected reward to a REINFORCE-style gradient
• A.3 Group-relative advantage used in iGRPO
• A.4 Off-policy sampling from π θ old \pi_{\theta_{\mathrm{old}}} and PPO-style clipping
• A.5 Including the per-token KL penalty
• A.6 Final iGRPO surrogate objective and resulting policy gradient
• Interpretation.
• Appendix B Scaling OpenMath-Nemotron-14B with iGRPO
• B.1 Analysis of Pass@N on AIME Benchmarks
• Appendix C Hyperparameter Setup
• Prompt:
• Appendix D Memory and Throughput Comparisons
• D.1 Setup
• D.2 Measurements
• Appendix E Additional Ablation Studies
• Training Dynamics and Response Length.
• Effect of KL Divergence Term.
• Effect of Number of Completions.
• Instructions for reporting errors