iGRPO: Self-Feedback-Driven LLM Reasoning — part 1/4

Cleaned-text code points [0, 15815). Bibliography omitted here; source.md includes it.
# iGRPO: Self - Feedback–Driven LLM Reasoning

###### Abstract

Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions. Reinforcement Learning (RL) is a framework for aligning these models with task-specific rewards, improving overall quality and reliability. Group Relative Policy Optimization (GRPO) is an efficient, value-function-free alternative to Proximal Policy Optimization (PPO) that leverages group-relative reward normalization. We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts. In Stage 1, iGRPO samples multiple exploratory drafts and selects the highest-reward draft using the same scalar reward signal used for optimization. In Stage 2, it appends this best draft to the original prompt and applies a GRPO-style update on draft-conditioned refinements, training the policy to improve beyond its strongest prior attempt. Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models (e.g., Nemotron-H-8B-Base-8K and DeepSeek-R1 Distilled), validating its effectiveness on diverse reasoning benchmarks. Moreover, applying iGRPO to OpenReasoning-Nemotron-7B trained on AceReason-Math achieves new state-of-the-art results of 85.62% and 79.64% on AIME24 and AIME25, respectively. Ablations further show that the refinement wrapper generalizes beyond GRPO variants, benefits from a generative judge, and alters learning dynamics by delaying entropy collapse. These results underscore the potential of iterative, self-feedback-based RL for advancing verifiable mathematical reasoning.

\abscontent

## 1 Introduction

Reinforcement Learning (RL) has proven to be successful in improving reasoning capabilities of LLMs by optimizing against task-specific reward signals. Early successes in this direction include RL from Human Feedback (RLHF) for aligning LLMs with human intent, most notably in InstructGPT ( Ouyang et al., 2022 ) and ChatGPT ( Achiam et al., 2023 ) , which have demonstrated that incorporating preference-based rewards can dramatically improve both the usability and correctness of model outputs. Recently DeepSeek-R1 ( Guo et al., 2025 ) proposed a distinguishing feature which is the so-called zero configuration, wherein the RL process directly enhances the base language model. This breakthrough started several efforts which were targeted at replicating DeepSeek-R1’s methodology or refining its underlying RL mechanisms ( Zeng et al., 2025 ; Yu et al., 2025 ; Liu et al., 2025 ; Cui et al., 2025 ; Hu et al., 2025 ) .

Yet, in the realm of complex reasoning, RL algorithms typically do not incorporate any form of feedback or reflection on the model’s own outputs. Humans, by contrast, rarely solve nontrivial problems in a single pass: they often iterate on initial drafts, identify mistakes, and refine their solutions based on internal feedback ( Flower & Hayes, 1981 ; Simon, 2012 ; Braidotti, 2019 ; Flavell, 1979 ; Schön, 2017 ; Polya, 2014 ) . There is growing evidence that self-feedback mechanisms can bolster multi-step reasoning and the capacity to correct errors ( Madaan et al., 2023 ; Shinn et al., 2023 ) . However, existing RL frameworks do not capitalize on this iterative refinement process, leaving a critical gap between how humans naturally solve problems and how LLMs are typically trained to do so.

In this work, we propose to fill this gap with Iterative GRPO (iGRPO) which is a powerful extension of GRPO ( Shao et al., 2024 ) . As illustrated in Figure 1 , Our method operates in two stages. First, we draw multiple candidate completions from the model and compute their relative rewards via the group-based mechanism of GRPO. We then select the highest-scoring draft and this serves as the "first-draft" output of the model. We consider this highest-scoring response as a guide to improve the final output. Hence, it is provided as a self-feedback to the model. We feed it back to the model alongside the original prompt. By conditioning on this exemplar, the second stage encourages the model to refine and surpass its own best prior attempt. Notably, this design preserves the efficiency of GRPO while introducing only minimal extra overhead, as iGRPO still relies on the same set of group-based reward signals. In doing so, iGRPO offers a promising avenue for self-guided improvement, enabling LLMs to iteratively improve their reasoning capabilities.

We conduct a series of controlled experiments to compare iGRPO and GRPO under identical training conditions, using different base models trained on the Mathematics Aptitude Test of Heuristics (MATH) ( Hendrycks et al., 2021 ) dataset. Specifically, we evaluate DeepSeek-R1 Distilled ( Guo et al., 2025 ) and OpenMath-Nemotron ( Moshkov et al., 2025 ) on an extensive array of mathematical reasoning benchmarks, including AIME24 ( AI-MO, 2024a ) , AIME25 ( OpenCompass, 2025 ) , MATH500 ( Lightman et al., 2023 ) , AMC23 ( AI-MO, 2024b ) , GSM8K ( Cobbe et al., 2021 ) , and Minerva Math ( Lewkowycz et al., 2022 ) . For models with 7B and 14B parameters, iGRPO consistently outperforms standard GRPO. Moreover, by leveraging iGRPO algorithm with OpenReasoning-Nemotron-7B model ( NVIDIA, 2025 ) on the large-scale AceReason - Math ( Chen et al., 2025b ) dataset ( R1, 2024 ) , we push the state of the art on AIME24 and AIME25 to 85.62% and 79.64%, respectively. These findings underscore the effectiveness of incorporating a self-feedback stage into group-based RL optimization, particularly for complex mathematical reasoning tasks.

## 2 Related Work

##### RL for Reasoning.

Reinforcement learning (RL) has become an important tool for refining large language models on logical and analytical tasks ( Lambert et al., 2024 ) . Early self-improvement lines of work, such as STaR-style bootstrapping with verified outcomes and sampling-based selection (as discussed in ( Lambert et al., 2024 ) ), demonstrate that iteratively leveraging model-generated solutions can improve reasoning behavior. More recent systems scale outcome-driven training substantially ( Jaech et al., 2024 ) , and open-weight efforts such as DeepSeek-R1 report strong reasoning performance under similar training regimes ( Guo et al., 2025 ) . Beyond natural language tasks, RL on procedurally generated puzzles ( Xie et al., 2025 ) and settings with limited human demonstrations ( Wang et al., 2025 ) further highlight the breadth of RL as a mechanism for improving mathematical and logical reasoning.

##### GRPO and Variants.

There has also been rapid progress in refining and extending GRPO for large-scale LLM training. Dr. GRPO ( Liu et al., 2025 ) analyzes sources of bias in GRPO-style token-level objectives and proposes modifications such as removing divisions by sequence length and group-level standard deviation to better match unbiased policy gradients. DAPO ( Yu et al., 2025 ) targets long chain-of-thought training through dynamic sampling, decoupled clipping, and reward shaping designed to mitigate instability and reward noise. GSPO ( Zheng et al., 2025 ) instead operates at the sequence level, redefining importance ratios and applying sequence-level clipping to improve stability, especially in Mixture-of-Experts settings. Whereas these approaches primarily focus on stabilizing or correcting the underlying optimization objective, iGRPO is orthogonal: it introduces a two-stage mechanism that uses externally evaluated best drafts as additional training context, shaping the data distribution seen by the optimizer.

##### LLM Self-Learning and Self-Improvement.

LLM self-learning methods aim to improve a model by leveraging feedback signals produced by the model itself or by closely related agents. For example, SPIN and Self-Rewarding Language Models ( Chen et al., 2024 ; Yuan et al., 2024 ) use the model (or a closely coupled variant) as an internal evaluator to drive further learning. Other approaches incorporate self-play, verifier-based alignment, or proof-oriented training signals ( Kirchner et al., 2024 ; Ye et al., 2024 ) , though unreliable rewards can hinder complex reasoning ( Lambert et al., 2024 ) . More specialized self-play extensions, such as SPC ( Chen et al., 2025a ) and SPAG ( Cheng et al., 2024 ) , introduce curated tasks or adversarial scenarios to strengthen critique and robustness. Self-Verification ( Zhang et al., 2025a ) unifies problem-solving and generative verification within a single RL framework, enabling inference-time scaling by using the model’s own verification scores to reweight or aggregate sampled solutions. Critique-GRPO ( Zhang et al., 2025b ) augments GRPO with natural-language critiques by generating critique-conditioned refinements and optimizing over both initial answers and their refinements in an online RL loop. While these methods often blur the roles of reward provider and reward recipient within a single agent (or paired agents), iGRPO instead uses externally evaluated best-prior drafts as an in-context guide for subsequent generations during training. This maintains a clearer separation between the model’s generation process and the reward signal, while still leveraging the core self-improvement principle of learning from the model’s own outputs.

## 3 Methodology

### 3.1 Background: Group Relative Policy Optimization

GRPO is a value-function-free variant of proximal policy optimization that leverages group-based relative rewards for advantage estimation. Let π θ \pi_{\theta} denote the current policy and π θ old \pi_{\theta_{\text{old}}} the old policy. We begin with a pretrained language model ℳ \mathcal{M} and a set of training instances ℬ = { ( q , a ) } \mathcal{B}=\{(q,a)\} , where q q is a prompt (e.g., a math problem) and a a is a reference answer. Given a prompt q q , GRPO samples a group of G G candidate outputs { o 1 , o 2 , … , o G } \{o_{1},o_{2},\dots,o_{G}\} from π θ old \pi_{\theta_{\text{old}}} : o i ∼ π θ old ( ⋅ ∣ q ) for i = 1 , … , G . o_{i}\sim\pi_{\theta_{\text{old}}}(\cdot\mid q)\quad\text{for }i=1,\dots,G. A reward model then evaluates each sampled output o i o_{i} , resulting in scores { R 1 , … , R G } \{R_{1},\dots,R_{G}\} . GRPO normalizes these scores within the group to compute the advantage A ^ i , t \hat{A}_{i,t} at each token index t t of o i o_{i} : A ^ i , t = R i − mean ⁡ ( { R 1 , … , R G } ) std ⁡ ( { R 1 , … , R G } ) , \hat{A}_{i,t}\;=\;\frac{R_{i}\;-\;\mathrm{mean}(\{R_{1},\dots,R_{G}\})}{\mathrm{std}(\{R_{1},\dots,R_{G}\})}, where mean ⁡ ( ⋅ ) \mathrm{mean}(\cdot) and std ⁡ ( ⋅ ) \mathrm{std}(\cdot) denote the sample mean and sample standard deviation of the group’s reward scores. If std ⁡ ( { R 1 , … , R G } ) = 0 \mathrm{std}(\{R_{1},\dots,R_{G}\})=0 , we set the normalized advantages to 0 0 (equivalently, one may add a small constant δ \delta to the denominator); we apply the same convention in iGRPO. Note that all tokens t t in o i o_{i} share the same advantage A ^ i , t = A ^ i \hat{A}_{i,t}=\hat{A}_{i} , reflecting a single scalar reward for each sampled completion.

GRPO then updates the current policy π θ \pi_{\theta} by maximizing a clipped surrogate objective. Let r i , t  ( θ ) = π θ  ( o i , t ∣ q , o i , < t ) π θ old  ( o i , t ∣ q , o i , < t ) . r_{i,t}(\theta)\;=\;\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}.

Then the GRPO objective is: 𝒥 GRPO  ( θ ) = \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\; 𝔼 ⁡ [ q ∼ P ⁡ ( Q ) , { o i } i = 1 G ∼ π θ old ] \displaystyle\mathbb{E}\Bigl[q\sim P(Q),\;\{o_{i}\}_{i=1}^{G}\sim\pi_{\theta_{\text{old}}}\Bigr] × 1 G  ∑ i = 1 G 1 | o i |  ∑ t = 1 | o i | [ min ⁡ ( r i , t  ( θ )  A ^ i , t , clip ⁡ ( r i , t  ( θ ) , 1 − ϵ , 1 + ϵ )  A ^ i , t ) − β  D ^ KL ( i , t ) ] , \displaystyle\times\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_{i}|}\sum_{t=1}^{|o_{i}|}\Bigl[\min\!\Bigl(r_{i,t}(\theta)\,\hat{A}_{i,t},\;\mathrm{clip}\!\bigl(r_{i,t}(\theta),\;1-\epsilon,\;1+\epsilon\bigr)\,\hat{A}_{i,t}\Bigr)\;-\;\beta\,\widehat{D}_{\mathrm{KL}}^{(i,t)}\Bigr], (1) where ϵ \epsilon is the PPO clipping parameter, β \beta is a regularization coefficient on the KL divergence to a reference policy π ref \pi_{\text{ref}} , and | o i | |o_{i}| denotes the token length of completion o i o_{i} . We use the following non-negative per-token estimator D ^ KL ( i , t ) \widehat{D}_{\mathrm{KL}}^{(i,t)} as a practical KL penalty Schulman (2020) : D ^ KL ( i , t ) = π ref  ( o i , t ∣ q , o i , < t ) π θ  ( o i , t ∣ q , o i , < t ) − log ⁡ π ref  ( o i , t ∣ q , o i , < t ) π θ  ( o i , t ∣ q , o i , < t ) − 1 , \widehat{D}_{\mathrm{KL}}^{(i,t)}\;=\;\frac{\pi_{\text{ref}}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}\;-\;\log\!\frac{\pi_{\text{ref}}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}\;-\;1, which remains guaranteed to be non-negative. This estimator is unbiased for D KL ( π θ ∥ π ref ) D_{\mathrm{KL}}(\pi_{\theta}\,\|\,\pi_{\text{ref}}) when the expectation is taken under samples from π θ \pi_{\theta} , and it serves as a convenient sample-based penalty within PPO-style updates. By directly computing group-based advantages instead of estimating value functions, GRPO avoids the overhead of training a separate critic model, making it particularly appealing for large-scale language model fine-tuning.

### 3.2 Iterative Group Relative Policy Optimization

We introduce Iterative Group Relative Policy Optimization (iGRPO) , a reinforcement learning algorithm that implements bootstrapped policy improvement through dynamic self-conditioning. Unlike standard policy gradient methods that optimize single-shot generations, iGRPO establishes a closed-loop refinement process where the policy learns to systematically improve upon its own best attempts. The key insight is that by coupling exploration (Stage 1) with conditioned exploitation (Stage 2) within each optimization step, the model acquires a generalizable self-improvement capability that compounds in training.

#### 3.2.1 Motivation: From Static Examples to Dynamic Self-Conditioning

The standard GRPO objective (Eq. 1 ) treats each generation as independent, ignoring potentially valuable information from the model’s own generation process. In-context learning (ICL) addresses this by conditioning on fixed demonstrations: 𝒥 ICL ( θ ) = 𝔼 q ∼ P ⁡ ( Q ) [ 𝔼 o ∼ π θ ( ⋅ | q , e ) [ R ϕ ( o ) ] ] , \mathcal{J}_{\text{ICL}}(\theta)=\mathbb{E}_{q\sim P(Q)}\left[\mathbb{E}_{o\sim\pi_{\theta}(\cdot|q,e)}\left[R_{\phi}(o)\right]\right], where e e is a static example. However, ICL suffers from a fundamental limitation: the conditioning signal e e remains fixed throughout training and does not adapt to the evolving policy’s capabilities.

iGRPO introduces a different paradigm, dynamic self-conditioning , where the conditioning signal is generated by the policy itself and co-evolves with learning: 𝒥 iGRPO  ( θ ) \displaystyle\mathcal{J}_{\text{iGRPO}}(\theta) = 𝔼 q ∼ P ⁡ ( Q ) [ 𝔼 o ∼ π θ ( ⋅ ∣ q ′ θ ( q ) ) [ R ϕ ( o ) ] ] , \displaystyle=\mathbb{E}_{q\sim P(Q)}\!\left[\mathbb{E}_{o\sim\pi_{\theta}(\cdot\!\mid\!q^{\prime}_{\theta}(q))}\!\left[R_{\phi}(o)\right]\right], q θ ′  ( q ) \displaystyle q^{\prime}_{\theta}(q) = Concat ⁡ ( q , d ^ θ  ( q ) ) \displaystyle=\mathrm{Concat}\!\bigl(q,\hat{d}_{\theta}(q)\bigr) where d ^ θ  ( q ) \hat{d}_{\theta}(q) is the best draft generated by the current policy (defined formally below). This creates a bootstrapped learning dynamic: as π θ \pi_{\theta} improves, so does the quality of d ^ θ \hat{d}_{\theta} , which in turn provides increasingly informative conditioning for subsequent generations.