iGRPO: Self-Feedback-Driven LLM Reasoning — part 2/4
Cleaned-text code points [15815, 31796). Bibliography omitted here; source.md includes it.
Although the expressions above are written in terms of π θ \pi_{\theta} for clarity, iGRPO follows the PPO/GRPO convention for stable optimization. At each iteration, we take a snapshot π θ old \pi_{\theta_{\text{old}}} , sample both Stage 1 drafts and Stage 2 completions from π θ old \pi_{\theta_{\text{old}}} , and update θ \theta using importance ratios computed relative to π θ old \pi_{\theta_{\text{old}}} . Stage 1 is not differentiated through, but the distribution of selected drafts d ^ \hat{d} (and thus the self-conditioned prompts q ′ q^{\prime} ) shifts as θ \theta evolves across iterations. In addition, dynamic self-conditioning is used only during training. At inference time, we use the trained policy in the standard single-shot manner, generating directly from the original prompt q q without any draft generation, conditioning, or specialized selection scheme.
#### 3.2.2 Algorithmic Framework
iGRPO operates through two tightly coupled stages within each optimization step. Crucially, only Stage 2 outputs receive gradient updates, while Stage 1 serves as an adaptive exploration mechanism that shapes the optimization landscape.
##### Stage 1: Exploratory Draft Generation.
Given a prompt q q , we sample N N candidate drafts from the current policy snapshot: d i ∼ π θ old ( ⋅ ∣ q ) , i = 1 , … , N . d_{i}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q),\quad i=1,\ldots,N. Each draft is evaluated using a reward function R ϕ R_{\phi} , and we identify the highest-scoring draft: d ^ = arg max i ∈ { 1 , … , N } R ϕ ( d i ) . \hat{d}=\arg\max_{i\in\{1,\ldots,N\}}R_{\phi}(d_{i}). (2) This stage performs implicit curriculum generation : early in training, d ^ \hat{d} may be a weak solution, but as the policy improves, d ^ \hat{d} increasingly represents a high-quality attempt that approaches (but does not yet reach) optimal performance.
##### Stage 2: Conditioned Refinement.
We form an augmented prompt by appending the best draft immediately after the original prompt: q ′ = Concat ( q , d ^ ) . q^{\prime}=\text{Concat}(q,\hat{d}). (3) We then sample a group of G G completions, mirroring the standard GRPO sampling: o j ∼ π θ old ( ⋅ ∣ q ′ ) , j = 1 , … , G . o_{j}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q^{\prime}),\quad j=1,\ldots,G. These completions are scored, and GRPO-style advantage estimation and policy updates are applied exclusively to these Stage 2 outputs. In practice, the concatenation in Eq. 3 uses a fixed prompt template, which we provide in the supplementary materials.
#### 3.2.3 Theoretical Analysis: Bootstrapped Policy Improvement
###### Definition 1 (Self-Conditioned Prompt Construction) .
Given a prompt q q and policy π θ \pi_{\theta} , sample N N drafts d i ∼ π θ ( ⋅ ∣ q ) , i = 1 , … , N , d_{i}\sim\pi_{\theta}(\cdot\mid q),\quad i=1,\ldots,N, and select the best draft d ^ θ ( q ) = arg max i ∈ { 1 , … , N } R ϕ ( d i ) . \hat{d}_{\theta}(q)=\arg\max_{i\in\{1,\ldots,N\}}R_{\phi}(d_{i}). We then define the self-conditioned prompt by appending the best draft immediately after the original prompt: q θ ′ ( q ) = Concat ( q , d ^ θ ( q ) ) . q^{\prime}_{\theta}(q)=\text{Concat}\bigl(q,\hat{d}_{\theta}(q)\bigr).
Unlike static ICL where the conditioning is independent of θ \theta , iGRPO’s conditioning is policy-dependent : the constructed prompt q θ ′ ( q ) q^{\prime}_{\theta}(q) changes as θ \theta changes, since d ^ θ ( q ) \hat{d}_{\theta}(q) is generated by π θ \pi_{\theta} . This creates a coupled dynamical system.
###### Proposition 3.1 (Progressive Conditioning Quality for Binary Rewards) .
Assume the reward is binary, R ϕ ( o ) ∈ { 0 , 1 } R_{\phi}(o)\in\{0,1\} , and Stage 1 drafts { d i } i = 1 N \{d_{i}\}_{i=1}^{N} are sampled i.i.d. from π θ ( ⋅ | q ) \pi_{\theta}(\cdot|q) . Let V θ ( q ) = 𝔼 o ∼ π θ ( ⋅ | q ) [ R ϕ ( o ) ] V_{\theta}(q)=\mathbb{E}_{o\sim\pi_{\theta}(\cdot|q)}[R_{\phi}(o)] denote the expected reward under policy π θ \pi_{\theta} , which equals the success probability p θ ( q ) = Pr [ R ϕ ( o ) = 1 ] p_{\theta}(q)=\Pr[R_{\phi}(o)=1] in the binary case. Then the expected reward of the selected best draft d ^ θ ( q ) = arg max i R ϕ ( d i ) \hat{d}_{\theta}(q)=\arg\max_{i}R_{\phi}(d_{i}) satisfies 𝔼 [ R ϕ ( d ^ θ ( q ) ) ] = 1 − ( 1 − V θ ( q ) ) N , \mathbb{E}\bigl[R_{\phi}(\hat{d}_{\theta}(q))\bigr]=1-(1-V_{\theta}(q))^{N}, which is monotonically increasing in V θ ( q ) V_{\theta}(q) . Consequently, if optimization increases V θ ( q ) V_{\theta}(q) , then 𝔼 [ R ϕ ( d ^ θ ( q ) ) ] \mathbb{E}[R_{\phi}(\hat{d}_{\theta}(q))] also increases, improving the conditioning quality for subsequent iterations in expectation.
###### Proof.
For binary rewards, R ϕ ( d ^ θ ( q ) ) = 1 R_{\phi}(\hat{d}_{\theta}(q))=1 if and only if at least one of the N N sampled drafts achieves reward 1 1 . Under i.i.d. sampling, this occurs with probability 1 − Pr [ ∀ i , R ϕ ( d i ) = 0 ] = 1 − ( 1 − V θ ( q ) ) N . 1-\Pr\bigl[\forall i,\;R_{\phi}(d_{i})=0\bigr]=1-(1-V_{\theta}(q))^{N}. Since R ϕ ( d ^ θ ( q ) ) ∈ { 0 , 1 } R_{\phi}(\hat{d}_{\theta}(q))\in\{0,1\} , its expectation equals this probability. The function 1 − ( 1 − x ) N 1-(1-x)^{N} is increasing in x ∈ [ 0 , 1 ] x\in[0,1] , establishing monotonicity in V θ ( q ) V_{\theta}(q) . ∎
This proposition captures the bootstrapping effect : better policies generate better drafts, which provide more informative conditioning, which enables learning better policies. The model does not merely learn to copy the conditioning; it learns a refinement function that maps draft attempts to improved solutions.
#### 3.2.4 Mathematical Formulation
Building on the GRPO objective in Eq. 1 , we present the complete iGRPO formulation. Let π θ \pi_{\theta} denote the policy being optimized, with π θ old \pi_{\theta_{\text{old}}} representing the policy snapshot at the start of each iteration for importance sampling.
##### Stage 1: Draft Selection.
For each prompt q q , Stage 1 samples N N drafts and selects the best: { d 1 , … , d N } ∼ π θ old ( ⋅ ∣ q ) , d ^ = arg max i ∈ { 1 , … , N } R ϕ ( d i ) . \{d_{1},\ldots,d_{N}\}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q),\qquad\hat{d}=\arg\max_{i\in\{1,\ldots,N\}}R_{\phi}(d_{i}).
##### Stage 2: Conditioned Generation and Advantage Computation.
We form the augmented prompt by appending the selected draft to q q (Eq. 3 ) and sample G G completions: { o 1 , … , o G } ∼ π θ old ( ⋅ ∣ q ′ ) . \{o_{1},\ldots,o_{G}\}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q^{\prime}). Each completion o j o_{j} receives a reward R ϕ ( o j ) R_{\phi}(o_{j}) , and advantages are computed via group normalization as in standard GRPO: A ^ j = R ϕ ( o j ) − mean ( { R ϕ ( o 1 ) , … , R ϕ ( o G ) } ) std ( { R ϕ ( o 1 ) , … , R ϕ ( o G ) } ) . \hat{A}_{j}=\frac{R_{\phi}(o_{j})-\mathrm{mean}(\{R_{\phi}(o_{1}),\ldots,R_{\phi}(o_{G})\})}{\mathrm{std}(\{R_{\phi}(o_{1}),\ldots,R_{\phi}(o_{G})\})}. (4) If std ( { R ϕ ( o 1 ) , … , R ϕ ( o G ) } ) = 0 \mathrm{std}(\{R_{\phi}(o_{1}),\ldots,R_{\phi}(o_{G})\})=0 , we set the normalized advantages to 0 0 (equivalently, one may add a small constant δ \delta to the denominator). All tokens t t in o j o_{j} share the same advantage A ^ j , t = A ^ j \hat{A}_{j,t}=\hat{A}_{j} , consistent with the GRPO formulation.
##### Full Objective.
The complete iGRPO objective combines both stages: 𝒥 iGRPO ( θ ) \displaystyle\mathcal{J}_{\text{iGRPO}}(\theta) = 𝔼 [ q ∼ P ( Q ) ] 𝔼 [ { d i } i = 1 N ∼ π θ old ( ⋅ ∣ q ) ⏟ Stage 1 , d ^ = arg max i R ϕ ( d i ) , q ′ = Concat ( q , d ^ ) , { o j } j = 1 G ∼ π θ old ( ⋅ ∣ q ′ ) ⏟ Stage 2 ] \displaystyle=\mathbb{E}\Bigl[q\sim P(Q)\Bigr]\;\mathbb{E}\Bigl[\underbrace{\{d_{i}\}_{i=1}^{N}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)}_{\text{Stage 1}},\;\hat{d}=\arg\max_{i}R_{\phi}(d_{i}),\;q^{\prime}=\mathrm{Concat}(q,\hat{d}),\;\underbrace{\{o_{j}\}_{j=1}^{G}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q^{\prime})}_{\text{Stage 2}}\Bigr] × 1 G ∑ j = 1 G 1 | o j | ∑ t = 1 | o j | [ min ( r j , t ( θ ) A ^ j , clip ( r j , t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ j ) − β D ^ KL ( j , t ) ] , \displaystyle\quad\times\frac{1}{G}\sum_{j=1}^{G}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\Bigl[\min\!\Bigl(r_{j,t}(\theta)\,\hat{A}_{j},\;\mathrm{clip}\!\bigl(r_{j,t}(\theta),\,1-\epsilon,\,1+\epsilon\bigr)\,\hat{A}_{j}\Bigr)\;-\;\beta\,\widehat{D}_{\mathrm{KL}}^{(j,t)}\Bigr], (5) where the importance sampling ratio is computed with respect to the augmented prompt: r j , t ( θ ) = π θ ( o j , t ∣ q ′ , o j , < t ) π θ old ( o j , t ∣ q ′ , o j , < t ) , r_{j,t}(\theta)=\frac{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta_{\text{old}}}(o_{j,t}\mid q^{\prime},o_{j,<t})}, and we use the same non-negative per-token KL penalty as in Section 3.1 , computed under the augmented prompt: D ^ KL ( j , t ) = π ref ( o j , t ∣ q ′ , o j , < t ) π θ ( o j , t ∣ q ′ , o j , < t ) − log π ref ( o j , t ∣ q ′ , o j , < t ) π θ ( o j , t ∣ q ′ , o j , < t ) − 1 . \widehat{D}_{\mathrm{KL}}^{(j,t)}=\frac{\pi_{\text{ref}}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}-\log\frac{\pi_{\text{ref}}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}-1. The key structural difference from vanilla GRPO (Eq. 1 ) is that all Stage 2 completions are conditioned on the augmented prompt q ′ q^{\prime} , formed by appending the best Stage 1 draft immediately after the original prompt q q . This provides a self-feedback signal that encourages the policy to refine its solutions beyond its strongest initial attempt.
##### Reward Function.
Following standard practice for verifiable reasoning tasks, we employ a rule-based reward function: R ϕ ( o ) = 𝟏 [ extract ( o ) = a ] , R_{\phi}(o)=\mathbf{1}\left[\texttt{extract}(o)=a\right], (6) where extract ( ⋅ ) \texttt{extract}(\cdot) parses the final answer from the completion and a a is the ground-truth reference answer from the training instance ( q , a ) ∈ ℬ (q,a)\in\mathcal{B} . This binary reward, combined with group normalization (Eq. 4 ), provides sufficient signal for distinguishing solution quality within each prompt group. The complete iGRPO procedure is presented in Algorithm 1 .
#### 3.2.5 Computational Analysis
Although iGRPO uses two stages, its compute is primarily controlled by the total number of sampled completions per prompt. With a fixed sampling budget, iGRPO can be run at essentially the same dominant generation cost as GRPO.
Let C gen C_{\text{gen}} denote the cost of producing one sampled completion (including prompt encoding, autoregressive decoding, and reward evaluation). Let G GRPO G_{\text{GRPO}} denote the number of completions sampled per prompt in standard GRPO.
##### Baseline GRPO Cost.
Standard GRPO samples G GRPO G_{\text{GRPO}} completions per prompt, giving per-prompt rollout cost C GRPO ≈ G GRPO C gen . C_{\text{GRPO}}\;\approx\;G_{\text{GRPO}}\,C_{\text{gen}}.
##### iGRPO Rollout Cost.
iGRPO samples N N Stage 1 drafts and G G Stage 2 refinements per prompt (Section 3.2 ), so C iGRPO ≈ ( N + G ) C gen , C iGRPO C GRPO = N + G G GRPO . C_{\text{iGRPO}}\;\approx\;(N+G)\,C_{\text{gen}},\qquad\frac{C_{\text{iGRPO}}}{C_{\text{GRPO}}}\;=\;\frac{N+G}{G_{\text{GRPO}}}. In our experiments, we keep the sampling budget fixed by setting N + G = G GRPO , N+G\;=\;G_{\text{GRPO}}, so iGRPO redistributes the same number of rollouts across Stage 1 and Stage 2 rather than increasing them. For example, with G GRPO = 16 G_{\text{GRPO}}=16 in GRPO, we use N = 8 N=8 and G = 8 G=8 in iGRPO, yielding C iGRPO ≈ ( N + G ) C gen = G GRPO C gen ≈ C GRPO , C_{\text{iGRPO}}\;\approx\;(N+G)\,C_{\text{gen}}\;=\;G_{\text{GRPO}}\,C_{\text{gen}}\;\approx\;C_{\text{GRPO}}, i.e., the dominant generation cost is comparable to GRPO.
## 4 Experiments
### 4.1 Setup
Our training data includes two datasets: MATH ( Hendrycks et al., 2021 ) (7,500 step-by-step competition problems) and AceReason-Math ( Chen et al., 2025b ) (9,400 problems). All models are trained for one epoch, with a KL divergence loss coefficient of 0 and no entropy regularization. We use a learning rate of 1 × 10 − 6 1\times 10^{-6} with a cosine schedule, generating eight completions per prompt (halved in each stage for iGRPO). The maximum prompt length is 1,024 tokens across all datasets. For all experiments, the completion length is capped at 4,096 tokens and the batch size at 1024. We evaluate model performance on well-known mathematical benchmarks, including AIME24/AIME25, MATH500, AMC23, GSM8K, and Minerva Math. For all benchmarks we report Pass@1 accuracy. However, for AIME24/AIME25 the reported value is averaged over 64 runs to ensure robustness. For other benchmarks, an average of 8 runs are reported. For evaluations, we use NeMo-Skills framework 1 1 1 https://github.com/NVIDIA/NeMo-Skills with decoding parameters such as a temperature of 0.6, top-p of 0.95, and generation length of 65,000. Additional training and evaluation details are provided in the supplementary materials.
### 4.2 Results
Our goal in this section is to isolate the practical benefit of adding a self-feedback refinement stage to GRPO. Concretely, we ask whether the two-stage training signal in iGRPO translates into better verifiable mathematical reasoning, and whether those gains persist across model families and scales.
##### Controlled study with matched sampling budget.
Table 1 compares iGRPO against vanilla GRPO and two recent self-improvement baselines, Self-Verification ( Zhang et al., 2025a ) and Critique-GRPO ( Zhang et al., 2025b ) , across model families and parameter scales (7B, 8B, 14B). All methods share the same training protocol and the same rule-based reward (Eq. 6 ). To ensure a fair compute comparison, we keep the total sampling budget fixed at eight completions per prompt for every method; iGRPO redistributes this budget across stages by using Stage 1 drafts to select self-feedback and Stage 2 completions to perform the GRPO-style update. As a result, differences in Table 1 reflect the effect of conditioning on the best draft rather than the effect of more sampling.
##### Generalist 8B model: the largest gains from self-feedback.
For Nemotron-H-8B-Base-8K , vanilla GRPO raises the macro-average from 29.65 % 29.65\% to 41.08 % 41.08\% . Self-Verification and Critique-GRPO further improve the average to 42.86 % 42.86\% and 43.39 % 43.39\% . iGRPO performs best at 45.04 % 45.04\% , which is +3.96 points over GRPO and +1.65 points over the strongest self-improvement baseline. The improvements are most visible on the benchmarks that most strongly penalize near-miss reasoning, including AIME25 ( 9.17 % 9.17\% ) and Minerva ( 32.72 % 32.72\% ), and iGRPO also reaches the highest GSM8K accuracy ( 91.26 % 91.26\% ). This is a setting where avoiding additional self-judgment tasks matters: iGRPO does not require the model to generate critiques or verification rationales; it provides the best draft as a direct, high-signal scaffold and trains the policy to refine beyond it.
##### Stronger 7B distilled reasoner: consistent gains concentrated on multi-step tasks.
For DeepSeek-R1-Distill-Qwen-7B , the base model is already strong ( 61.93 % 61.93\% average), and GRPO improves it to 68.29 % 68.29\% . Self-Verification and Critique-GRPO reach 69.08 % 69.08\% and 69.14 % 69.14\% . iGRPO remains best overall at 69.87 % 69.87\% . The gains concentrate on multi-step benchmarks where a mostly-correct attempt can fail due to a small late error (e.g., AIME24 at 56.30 % 56.30\% and AMC at 95.00 % 95.00\% ). This pattern fits the iGRPO mechanism: Stage 1 increases the chance that a strong reasoning trajectory appears in the context, and Stage 2 learns to reliably “finish the job” under the same verifiable reward.
##### Math-specialized 7B model: improvements persist when the base is already strong.