iGRPO: Self-Feedback-Driven LLM Reasoning — part 3/4
Cleaned-text code points [31796, 47078). Bibliography omitted here; source.md includes it.
For OpenMath-Nemotron-7B , the base model starts at 74.83 % 74.83\% average. GRPO yields a small change to 75.02 % 75.02\% , suggesting limited headroom from standard one-shot RL updates in this regime. iGRPO increases performance to 76.07 % 76.07\% , with the most notable gains on the harder benchmarks, including AIME24 (from 73.28 % 73.28\% to 74.79 % 74.79\% ) and AMC (from 95.00 % 95.00\% to 97.50 % 97.50\% ). When drafts are already high-quality, the value of iGRPO is less about discovering a viable solution mode and more about systematically reinforcing the most reliable completion patterns.
##### Scaling to 14B parameters: benefits persist on complex reasoning.
At the 14B scale, iGRPO continues to improve on GRPO across both model families. For DeepSeek-R1-Distill-Qwen-14B , the macro-average increases from 71.29 % 71.29\% (GRPO) to 73.02 % 73.02\% (iGRPO), with a large gain on AIME24 from 60.26 % 60.26\% to 64.06 % 64.06\% . For OpenMath-Nemotron-14B , the macro-average improves from 76.73 % 76.73\% (GRPO) to 78.00 % 78.00\% (iGRPO), including gains on AIME25 from 64.53 % 64.53\% to 65.57 % 65.57\% and on AIME24 from 74.79 % 74.79\% to 76.72 % 76.72\% . While margins naturally shrink as models become stronger, the improvement pattern remains stable: iGRPO helps larger models correct residual errors that survive strong pretraining and standard RL fine-tuning, especially on long-horizon competition benchmarks.
##### Competitiveness against critique-style objectives.
Self-Verification and Critique-GRPO are strong baselines, but they ask the model to allocate capacity to additional behaviors (verifying, critiquing, or producing auxiliary text) that are only indirectly optimized by the outcome reward. iGRPO keeps the training loop tightly aligned with the verifiable objective while still capturing the benefit of iteration: Stage 1 supplies a high-quality draft as an explicit conditioning signal, and Stage 2 directly optimizes a refinement policy that can surpass that draft. Empirically, this design yields consistent improvements across model families and scales, with the strongest gains on benchmarks that are sensitive to small long-horizon reasoning mistakes.
### 4.3 Generalization to a Stronger Base and Harder Dataset
All prior controlled comparisons were trained on the 7,500-problem MATH set. We
next evaluate iGRPO in a harder regime by both strengthening the initialization and shifting to a more challenging training distribution. We start from OpenReasoning-Nemotron-7B and train on AceReason-Math ( Chen et al., 2025b ) , keeping the iGRPO configuration identical to Section 4 ; Figure 2 summarizes the results. iGRPO improves performance across all benchmarks, with the strongest gains on AIME24/25 ( +1.52 / +1.78 ) and clear transfer beyond math to GPQA ( +1.84 ) and MMLU-Pro ( +0.91 ), yielding a +1.23 overall average improvement. This indicates iGRPO remains effective with a stronger base model and harder training data, and that iterative self-feedback improves broadly useful refinement behaviors rather than only math-specific patterns.
## 5 Ablation
##### Beyond GRPO: self-feedback as a reusable refinement wrapper.
Our two-stage procedure is not tied to the GRPO objective itself: it can be layered on top of other group-based PPO variants by using Stage 1 to select a high-reward draft and Stage 2 to perform the base optimizer’s update on self-conditioned prompts. Table 2 applies this wrapper to DAPO and GSPO under matched rollout budgets (same total sampled completions per prompt, same reward, and identical training and evaluation settings). In both cases, self-feedback yields a consistent +1.1 to +1.2 macro-average improvement, indicating that the gains primarily stem from the refinement interface rather than GRPO-specific details.
##### Generative judge study.
Because iGRPO only needs a scalar reward to (i) rank Stage 1 drafts and (ii) compute Stage 2 group-normalized advantages, we can swap the binary outcome checker (Eq. 6 ) for a generative judge. On DeepSeek-R1-Distill-Qwen-7B trained on MATH, GPT-5 scoring each solution in [ 0 , 1 ] [0,1] improves Pass@1 on all six benchmarks and increases the average from 69.87 to 70.81 ( +0.94 ; Table 3 ). The largest gains on AIME24/25 and Minerva are consistent with partial credit for near-miss traces, which lets them survive Stage 1 selection and be refined into correct answers in Stage 2.
##### Entropy analysis.
To probe how self-feedback alters learning dynamics, we track the per-token Shannon entropy of the policy during RL. For a decoding step t t with context h t h_{t} and vocabulary 𝒱 \mathcal{V} , we measure entropy in nats as ℋ ( π θ ( ⋅ ∣ h t ) ) = − ∑ w ∈ 𝒱 π θ ( w ∣ h t ) ln π θ ( w ∣ h t ) , \mathcal{H}\bigl(\pi_{\theta}(\cdot\mid h_{t})\bigr)=-\sum_{w\in\mathcal{V}}\pi_{\theta}(w\mid h_{t})\,\ln\pi_{\theta}(w\mid h_{t}), (7) and in practice compute it from the log-softmax of the logits (base e e ), then average over all valid completion tokens in the batch to obtain a single scalar per training step. On DeepSeek-R1-Distill-Qwen-7B trained on MATH, both methods start at 2.45 2.45 nats, but GRPO collapses rapidly ( 0.60 0.60 at 10 % 10\% , 0.42 0.42 by 30 % 30\% ) and then stays flat through the end. In contrast, iGRPO decays more gradually ( 0.80 0.80 at 15 % 15\% , 0.48 0.48 at 30 % 30\% ) and remains slightly higher through mid-training ( 0.46 0.46 at 60 % 60\% ) before converging near GRPO ( 0.44 0.44 vs. 0.42 0.42 at 100 % 100\% ). This suggests iGRPO delays premature mode collapse: conditioning on the best draft encourages refinement around a strong scaffold while preserving alternative continuations long enough to recover from near-miss reasoning traces. Since final entropies are close, the gains are better explained by sustained mid-training exploration rather than higher randomness at convergence.
## 6 Conclusion
We introduced Iterative Group Relative Policy Optimization (iGRPO) , a simple and effective extension of GRPO that injects an explicit self-feedback signal into outcome-driven RL for reasoning. iGRPO replaces single-shot optimization with a two-stage loop: Stage 1 samples multiple drafts and selects the highest-reward completion as feedback; Stage 2 conditions on this best draft and applies a standard GRPO-style update on refinements. This yields dynamic self-conditioning : the training context automatically improves as the policy improves, creating a bootstrapped refinement behavior that is absent from conventional group-based objectives. We also provided a clean theoretical characterization of the bootstrapping effect under binary rewards, showing that the expected quality of the selected draft increases monotonically with the policy’s success probability.
Empirically, iGRPO consistently improves verifiable math reasoning across model families and scales under matched rollout budgets. Across 7B, 8B, and 14B backbones, iGRPO outperforms vanilla GRPO as well as strong self-improvement baselines that rely on critique or verification behaviors (Table 1 ). In a harder generalization setting, training OpenReasoning-Nemotron-7B on AceReason-Math with iGRPO yields new best results on AIME24/AIME25 (85.62%/79.64%) and transfers gains beyond math to GPQA and MMLU-Pro (Figure 2 ). Our ablations further suggest that the benefit primarily comes from the refinement interface itself: the same two-stage wrapper improves other group-based PPO variants (Table 2 ), remains compatible with richer scalar rewards such as a generative judge (Table 3 ), and measurably alters learning dynamics by delaying premature entropy collapse (Figure 3 ).
## Appendix
## Appendix A Policy Gradient Derivation for iGRPO
In this sectin, we derive the iGRPO policy gradient update consistent with the methodology in Section 3.2 and the full objective in Eq. equation 5 . Throughout, π θ \pi_{\theta} is the trainable policy, π θ old \pi_{\theta_{\mathrm{old}}} is the frozen snapshot used for sampling (PPO/GRPO convention), π ref \pi_{\mathrm{ref}} is the reference policy, and R ϕ ( ⋅ ) R_{\phi}(\cdot) is a scalar (verifier) reward. Importantly, iGRPO uses two stages per optimization step, but gradients are applied only to Stage 2 tokens.
### A.1 Two-stage sampling and the induced self-conditioned prompt distribution
For each prompt q ∼ P ( Q ) q\sim P(Q) , iGRPO constructs an augmented prompt by first sampling Stage 1 drafts and selecting the best: d i \displaystyle d_{i} ∼ π θ old ( ⋅ ∣ q ) , i = 1 , … , N , \displaystyle\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q),\quad i=1,\ldots,N, (8) d ^ \displaystyle\hat{d} = arg max i ∈ { 1 , … , N } R ϕ ( d i ) , \displaystyle=\arg\max_{i\in\{1,\ldots,N\}}R_{\phi}(d_{i}), (9) q ′ \displaystyle q^{\prime} = Concat ( q , d ^ ) . \displaystyle=\mathrm{Concat}(q,\hat{d}). (10) This defines an implicit sampling distribution over augmented prompts q ′ q^{\prime} induced by π θ old \pi_{\theta_{\mathrm{old}}} and the arg max \arg\max selection. Within a single PPO-style update, q ′ q^{\prime} is treated as part of the sampled context (no differentiation through Stage 1 and no differentiation through the arg max \arg\max ). Across iterations, the distribution over q ′ q^{\prime} shifts because π θ old \pi_{\theta_{\mathrm{old}}} changes.
Given the augmented prompt q ′ q^{\prime} , Stage 2 samples a group of G G completions: o j ∼ π θ old ( ⋅ ∣ q ′ ) , j = 1 , … , G . o_{j}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q^{\prime}),\quad j=1,\ldots,G. (11) Let each completion be a token sequence o j = ( o j , 1 , … , o j , | o j | ) o_{j}=(o_{j,1},\ldots,o_{j,|o_{j}|}) with factorization π θ ( o j ∣ q ′ ) = ∏ t = 1 | o j | π θ ( o j , t ∣ q ′ , o j , < t ) . \pi_{\theta}(o_{j}\mid q^{\prime})\;=\;\prod_{t=1}^{|o_{j}|}\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (12)
### A.2 From the self-conditioned expected reward to a REINFORCE-style gradient
Fix an augmented prompt q ′ q^{\prime} . Consider the (conceptual) self-conditioned objective 𝒥 ( θ ∣ q ′ ) = 𝔼 o ∼ π θ ( ⋅ ∣ q ′ ) [ R ϕ ( o ) ] . \mathcal{J}(\theta\mid q^{\prime})\;=\;\mathbb{E}_{o\sim\pi_{\theta}(\cdot\mid q^{\prime})}\bigl[R_{\phi}(o)\bigr]. (13) By the score-function (REINFORCE) identity, ∇ θ 𝒥 ( θ ∣ q ′ ) \displaystyle\nabla_{\theta}\mathcal{J}(\theta\mid q^{\prime}) = 𝔼 o ∼ π θ ( ⋅ ∣ q ′ ) [ R ϕ ( o ) ∇ θ log π θ ( o ∣ q ′ ) ] \displaystyle=\mathbb{E}_{o\sim\pi_{\theta}(\cdot\mid q^{\prime})}\Bigl[R_{\phi}(o)\,\nabla_{\theta}\log\pi_{\theta}(o\mid q^{\prime})\Bigr] (14) = 𝔼 o ∼ π θ ( ⋅ ∣ q ′ ) [ R ϕ ( o ) ∑ t = 1 | o | ∇ θ log π θ ( o t ∣ q ′ , o < t ) ] , \displaystyle=\mathbb{E}_{o\sim\pi_{\theta}(\cdot\mid q^{\prime})}\Biggl[R_{\phi}(o)\,\sum_{t=1}^{|o|}\nabla_{\theta}\log\pi_{\theta}(o_{t}\mid q^{\prime},o_{<t})\Biggr], (15) where the second line uses Eq. equation 12 . Introducing any baseline b ( q ′ ) b(q^{\prime}) that does not depend on the sampled tokens preserves unbiasedness and yields an advantage-weighted gradient: ∇ θ 𝒥 ( θ ∣ q ′ ) = 𝔼 o ∼ π θ ( ⋅ ∣ q ′ ) [ ( R ϕ ( o ) − b ( q ′ ) ) ∑ t = 1 | o | ∇ θ log π θ ( o t ∣ q ′ , o < t ) ] . \nabla_{\theta}\mathcal{J}(\theta\mid q^{\prime})=\mathbb{E}_{o\sim\pi_{\theta}(\cdot\mid q^{\prime})}\Biggl[\bigl(R_{\phi}(o)-b(q^{\prime})\bigr)\sum_{t=1}^{|o|}\nabla_{\theta}\log\pi_{\theta}(o_{t}\mid q^{\prime},o_{<t})\Biggr]. (16)
### A.3 Group-relative advantage used in iGRPO
In iGRPO, the baseline is estimated per prompt by sampling a group of G G completions { o j } j = 1 G \{o_{j}\}_{j=1}^{G} and normalizing rewards within the group. Define R j = R ϕ ( o j ) , R ¯ = mean ( { R 1 , … , R G } ) , s R = std ( { R 1 , … , R G } ) , R_{j}\;=\;R_{\phi}(o_{j}),\qquad\overline{R}\;=\;\mathrm{mean}(\{R_{1},\ldots,R_{G}\}),\qquad s_{R}\;=\;\mathrm{std}(\{R_{1},\ldots,R_{G}\}), (17) and the iGRPO advantage A ^ j = R j − R ¯ s R , with the convention A ^ j = 0 if s R = 0 . \hat{A}_{j}\;=\;\frac{R_{j}-\overline{R}}{s_{R}},\qquad\text{with the convention }\hat{A}_{j}=0\text{ if }s_{R}=0. (18) As in GRPO, this advantage is a single scalar per completion, shared across all token indices. iGRPO additionally uses the token-averaged form (the 1 / | o j | 1/|o_{j}| factor) so that completions of different lengths contribute comparably: g on - policy ( θ ∣ q ′ ) ≐ 1 G ∑ j = 1 G A ^ j | o j | ∑ t = 1 | o j | ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) . g_{\mathrm{on\text{-}policy}}(\theta\mid q^{\prime})\;\doteq\;\frac{1}{G}\sum_{j=1}^{G}\frac{\hat{A}_{j}}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (19) Eq. equation 19 is the basic (conceptual) iGRPO policy-gradient estimator when sampling from π θ \pi_{\theta} . The remainder of the derivation follows the standard PPO/GRPO stabilization used in the main methodology: sampling from π θ old \pi_{\theta_{\mathrm{old}}} and using importance ratios with clipping and a KL penalty.
### A.4 Off-policy sampling from π θ old \pi_{\theta_{\mathrm{old}}} and PPO-style clipping
Within each iteration, iGRPO samples Stage 2 completions from π θ old \pi_{\theta_{\mathrm{old}}} for stability. To relate gradients under π θ old \pi_{\theta_{\mathrm{old}}} to the updated policy π θ \pi_{\theta} , define the per-token importance ratio r j , t ( θ ) = π θ ( o j , t ∣ q ′ , o j , < t ) π θ old ( o j , t ∣ q ′ , o j , < t ) . r_{j,t}(\theta)=\frac{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{j,t}\mid q^{\prime},o_{j,<t})}. (20) Following PPO/GRPO, iGRPO maximizes a clipped surrogate that replaces the on-policy factor by a clipped importance-weighted term: ℒ j , t clip ( θ ) = min ( r j , t ( θ ) A ^ j , clip ( r j , t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ j ) . \mathcal{L}^{\mathrm{clip}}_{j,t}(\theta)=\min\!\Bigl(r_{j,t}(\theta)\,\hat{A}_{j},\;\mathrm{clip}\bigl(r_{j,t}(\theta),\,1-\epsilon,\,1+\epsilon\bigr)\,\hat{A}_{j}\Bigr). (21) The per-token clipped objective in Eq. equation 21 yields the usual piecewise gradient behavior: when the ratio is clipped, the clipped branch is constant in r j , t ( θ ) r_{j,t}(\theta) and contributes zero gradient through that branch.
A convenient way to write the gradient is via the indicator of the unclipped branch. Let 𝕀 j , t ( θ ) = { 1 , A ^ j ≥ 0 and r j , t ( θ ) ≤ 1 + ϵ , 1 , A ^ j < 0 and r j , t ( θ ) ≥ 1 − ϵ , 0 , otherwise , \mathbb{I}_{j,t}(\theta)=\begin{cases}1,&\hat{A}_{j}\geq 0\text{ and }r_{j,t}(\theta)\leq 1+\epsilon,\\ 1,&\hat{A}_{j}<0\text{ and }r_{j,t}(\theta)\geq 1-\epsilon,\\ 0,&\text{otherwise},\end{cases} (22) which matches the standard PPO clipping rule. Then, using ∇ θ r j , t ( θ ) = r j , t ( θ ) ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) \nabla_{\theta}r_{j,t}(\theta)=r_{j,t}(\theta)\,\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}) , the gradient of the clipped surrogate is ∇ θ ℒ j , t clip ( θ ) = 𝕀 j , t ( θ ) A ^ j r j , t ( θ ) ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) . \nabla_{\theta}\mathcal{L}^{\mathrm{clip}}_{j,t}(\theta)=\mathbb{I}_{j,t}(\theta)\;\hat{A}_{j}\;r_{j,t}(\theta)\;\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (23)
### A.5 Including the per-token KL penalty