iGRPO: Self-Feedback-Driven LLM Reasoning — part 4/4
Cleaned-text code points [47078, 61926). Bibliography omitted here; source.md includes it.
As in Section 3.1 , iGRPO includes a per-token KL penalty to a reference policy π ref \pi_{\mathrm{ref}} using the non-negative estimator D ^ KL ( j , t ) = π ref ( o j , t ∣ q ′ , o j , < t ) π θ ( o j , t ∣ q ′ , o j , < t ) − log π ref ( o j , t ∣ q ′ , o j , < t ) π θ ( o j , t ∣ q ′ , o j , < t ) − 1 . \widehat{D}_{\mathrm{KL}}^{(j,t)}=\frac{\pi_{\mathrm{ref}}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}-\log\frac{\pi_{\mathrm{ref}}(o_{j,t}\mid q^{\prime},o_{j,<t})}{\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t})}-1. (24) Let ρ j , t ( θ ) = π ref ( o j , t ∣ q ′ , o j , < t ) / π θ ( o j , t ∣ q ′ , o j , < t ) \rho_{j,t}(\theta)=\pi_{\mathrm{ref}}(o_{j,t}\mid q^{\prime},o_{j,<t})/\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}) . Then D ^ KL ( j , t ) = ρ j , t − log ρ j , t − 1 \widehat{D}_{\mathrm{KL}}^{(j,t)}=\rho_{j,t}-\log\rho_{j,t}-1 , and its gradient has a simple form: ∇ θ D ^ KL ( j , t ) = − ( ρ j , t ( θ ) − 1 ) ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) . \nabla_{\theta}\widehat{D}_{\mathrm{KL}}^{(j,t)}=-\bigl(\rho_{j,t}(\theta)-1\bigr)\;\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (25) Therefore, the KL-regularized contribution in the objective, − β D ^ KL ( j , t ) -\beta\,\widehat{D}_{\mathrm{KL}}^{(j,t)} , contributes − β ∇ θ D ^ KL ( j , t ) = β ( ρ j , t ( θ ) − 1 ) ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) . -\beta\,\nabla_{\theta}\widehat{D}_{\mathrm{KL}}^{(j,t)}=\beta\bigl(\rho_{j,t}(\theta)-1\bigr)\;\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (26)
### A.6 Final iGRPO surrogate objective and resulting policy gradient
Putting the pieces together and reinstating the full two-stage sampling (where Stage 1 affects the distribution of q ′ q^{\prime} but is not differentiated through within an iteration), the iGRPO surrogate objective is: 𝒥 iGRPO ( θ ) \displaystyle\mathcal{J}_{\mathrm{iGRPO}}(\theta) = 𝔼 [ q ∼ P ( Q ) ] 𝔼 [ { d i } i = 1 N ∼ π θ old ( ⋅ ∣ q ) , d ^ = arg max i R ϕ ( d i ) , q ′ = Concat ( q , d ^ ) , \displaystyle=\mathbb{E}\Bigl[q\sim P(Q)\Bigr]\;\mathbb{E}\Bigl[\{d_{i}\}_{i=1}^{N}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q),\;\hat{d}=\arg\max_{i}R_{\phi}(d_{i}),\;q^{\prime}=\mathrm{Concat}(q,\hat{d}), { o j } j = 1 G ∼ π θ old ( ⋅ ∣ q ′ ) ] \displaystyle\hskip 78.00014pt\{o_{j}\}_{j=1}^{G}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q^{\prime})\Bigr] × 1 G ∑ j = 1 G 1 | o j | ∑ t = 1 | o j | [ ℒ j , t clip ( θ ) − β D ^ KL ( j , t ) ] , \displaystyle\quad\times\frac{1}{G}\sum_{j=1}^{G}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\Bigl[\mathcal{L}^{\mathrm{clip}}_{j,t}(\theta)-\beta\,\widehat{D}_{\mathrm{KL}}^{(j,t)}\Bigr], (27) where ℒ j , t clip ( θ ) \mathcal{L}^{\mathrm{clip}}_{j,t}(\theta) is defined in Eq. equation 21 , r j , t ( θ ) r_{j,t}(\theta) in Eq. equation 20 , and D ^ KL ( j , t ) \widehat{D}_{\mathrm{KL}}^{(j,t)} in Eq. equation 24 . This matches the structure given in Eq. equation 5 .
Differentiating Eq. equation 27 yields the iGRPO policy gradient: ∇ θ 𝒥 iGRPO ( θ ) = 𝔼 [ ⋯ ] 1 G ∑ j = 1 G 1 | o j | ∑ t = 1 | o j | [ ∇ θ ℒ j , t clip ( θ ) − β ∇ θ D ^ KL ( j , t ) ] , \displaystyle\nabla_{\theta}\mathcal{J}_{\mathrm{iGRPO}}(\theta)=\mathbb{E}\Bigl[\cdots\Bigr]\;\frac{1}{G}\sum_{j=1}^{G}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\Bigl[\nabla_{\theta}\mathcal{L}^{\mathrm{clip}}_{j,t}(\theta)-\beta\,\nabla_{\theta}\widehat{D}_{\mathrm{KL}}^{(j,t)}\Bigr], (28) with the explicit per-token forms from Eqs. equation 23 and equation 25 . Concretely, combining them gives ∇ θ 𝒥 iGRPO ( θ ) = 𝔼 [ ⋯ ] 1 G ∑ j = 1 G 1 | o j | ∑ t = 1 | o j | [ 𝕀 j , t ( θ ) A ^ j r j , t ( θ ) + β ( ρ j , t ( θ ) − 1 ) ] ∇ θ log π θ ( o j , t ∣ q ′ , o j , < t ) . \displaystyle\nabla_{\theta}\mathcal{J}_{\mathrm{iGRPO}}(\theta)=\mathbb{E}\Bigl[\cdots\Bigr]\;\frac{1}{G}\sum_{j=1}^{G}\frac{1}{|o_{j}|}\sum_{t=1}^{|o_{j}|}\Bigl[\mathbb{I}_{j,t}(\theta)\,\hat{A}_{j}\,r_{j,t}(\theta)+\beta\bigl(\rho_{j,t}(\theta)-1\bigr)\Bigr]\;\nabla_{\theta}\log\pi_{\theta}(o_{j,t}\mid q^{\prime},o_{j,<t}). (29)
##### Interpretation.
Eq. equation 29 makes the roles of the two stages explicit. Stage 1 defines the self-conditioned context q ′ q^{\prime} (and thus the learning problem presented to Stage 2) but does not contribute direct gradients within an iteration. Stage 2 contributes a standard GRPO/PPO-style token-level policy gradient, where each token is weighted by a group-normalized advantage A ^ j \hat{A}_{j} (shared across tokens of the completion), stabilized by importance-ratio clipping, and regularized by a KL penalty to π ref \pi_{\mathrm{ref}} .
## Appendix B Scaling OpenMath-Nemotron-14B with iGRPO
To further validate the effectiveness of iGRPO, we trained the OpenMath-Nemotron-14B with iGRPO on large scale dataset of OpenR1-Math-220k ( R1, 2024 ) which consists of 220,000 math problems with reasoning traces from DeepSeek-R1. For this study, we use 94,000 examples. As shown in Table S.1 , the reasoning performance of the model trained with iGRPO is significantly improved, achieving an impressive AIME25 score of 66.04%.
### B.1 Analysis of Pass@N on AIME Benchmarks
Figure S.1 shows how the accuracy of iOpenMath-Nemotron-14B evolves when increasing the number of attempts N N on the AIME24 and AIME25 benchmarks. As expected, we see consistent gains in performance as N N grows, indicating that the model can generate correct solutions among multiple sampled responses even if the top-1 guess is sometimes incorrect. Although our base SFT model already shows strong results at pass@1, the additional RL fine-tuning appears to capitalize on multi-sample scenarios. For example, on AIME25, the model improves from 66.04% (pass@1) to 86.67% at pass@8, demonstrating the effectiveness of producing multiple solutions for challenging competition problems.
Despite these gains, there are clear saturation points. AIME24 converges to its best score of 93.33% by N = 16 N=16 , with no further improvement at higher N N . On the contrary, the performance of AIME25 continues to improve even at high values of N N . While the performance seems to plateau briefly at 90.00% between N = 32 N=32 and N = 128 N=128 , it eventually increases to 96.67% at N = 256 N=256 . This contrast suggests that certain problem distributions, especially in AIME25, may benefit from a larger number of sampled attempts, whereas others, such as those in AIME24, can be adequately solved with fewer solution attempts.
## Appendix C Hyperparameter Setup
We conduct ablation studies on both 7B and 14B model variants, spanning core architectures such as DeepSeek-R1-Distill-Qwen and OpenMath-Nemotron. All models are trained for one epoch across two datasets: (1) the MATH dataset ( Hendrycks et al., 2021 ) of 7,500 step-by-step problems, and (2) AceReason - Math ( Chen et al., 2025b ) dataset (9400 problems). Our training uses a KL divergence loss coefficient of 0. We set a cosine learning rate schedule (minimum rate of 0.1) with a base learning rate of 1 × 10 − 6 1\times 10^{-6} . We use 8 rollouts for all experiments.
Table S.2 lists the concrete hyperparameters for training 7B iGRPO models. Notably, we run on 2 nodes with 8 × \times NVIDIA A100 GPUs each, and one of these nodes is fully allocated to vLLM for generation. We keep a global batch size of 128, with a per-device batch size of 16 and a gradient accumulation step size of 8. For the 14B models, we scale out to 5 nodes of 8 × \times NVIDIA A100 GPUs each (again, one node reserved for vLLM) and reduce the per-device batch size to 4 (maintaining the same global batch size of 128). In both 7B and 14B setups, we continue to use bfloat16 precision and the FlashAttention-2 kernel. The temperature is set to 0.7 for generation, and we apply two reward functions (accuracy and format) each with weight 1.0. This configuration provides a balanced trade-off between training stability, throughput, and alignment with complex mathematical reasoning tasks.
##### Prompt:
We use the following prompt for training model with iGRPO.
## Appendix D Memory and Throughput Comparisons
### D.1 Setup
To evaluate the resource utilization of our training setup, we replicate the exact environment and conditions under which our 7B models are typically trained. Specifically, we use DeepSeek-R1-Distill-Qwen-7B as our base model, which serves as a representative checkpoint for measuring throughput and memory consumption when training on the MATH dataset. Our training configuration employs a per-device batch size of 16, along with a global gradient accumulation step of 8, allowing us to effectively simulate heavier loads without exceeding GPU memory constraints. Additionally, we use a maximum completion length of 2048 tokens to benchmark model performance. We run experiments on two nodes, each equipped with 8 × NVIDIA A100 GPUs. One node is dedicated to vllm generation, ensuring that inference or generation processes do not interfere with the primary training workload, while the other node is reserved exclusively for model training.
We measure peak memory usage by periodically querying the GPU memory allocator for the maximum memory allocation that has occurred since the start of training. Specifically, at the beginning of training, we reset the peak memory statistics, and then after each iteration, we retrieve the current peak memory usage in bytes. We convert this value to gigabytes for readability and log it alongside other training metrics. To measure throughput, we track the total number of samples processed over time. We calculate this by multiplying the current global step by both the per-device batch size and the number of devices used in data parallelism. Dividing this product by the elapsed training time in seconds yields the throughput, expressed as samples processed per second. This real-time monitoring of memory and throughput allows us to evaluate hardware utilization efficiency, identify possible bottlenecks, and compare different training configurations in a consistent and quantifiable manner.
### D.2 Measurements
Table S.3 presents measured GPU usage and training throughput under iGRPO vs GRPO. Despite the two-stage nature of iGRPO, its peak memory usage of 54.9349 GB closely matches GRPO’s 54.9286 GB, a difference of roughly 0.0063 GB, which is practically negligible. This matches our theoretical expectation that the self-feedback mechanism adds minimal overhead, validating the feasibility of integrating iterative refinements even under constrained resource budgets.
Regarding throughput, iGRPO processes 0.34 samples/s compared to GRPO’s 0.41 samples/s, reflecting a mild slowdown tied to the additional round of generation. Crucially, this is neither an order-of-magnitude nor a large factor reduction. Instead, it shows that iGRPO’s second-stage refinement imposes only a modest computational cost. In summary, these measurements confirm our claims that iGRPO can be implemented with little additional overhead, supporting it as a practical strategy for enhancing mathematical reasoning performance without compromising resource efficiency.
Beyond instantaneous throughput, we also report full training cost measured in total GPU hours. Under the same compute budget and eight generations per prompt, GRPO requires 83.3 GPU hours while iGRPO uses 94.1 GPU hours, which corresponds to roughly a 13% increase in wall-clock training time. This overhead arises from the sequential Stage 1 plus Stage 2 decoding but does not demand more GPUs or additional memory capacity, since peak usage remains essentially unchanged. Given that this modest time increase delivers several-point gains on AIME24 and AIME25 and enables our 7B models to reach state-of-the-art performance, we view the tradeoff between 13% extra training time and substantially higher reasoning accuracy as a favorable and practical value proposition in real deployments.
## Appendix E Additional Ablation Studies
##### Training Dynamics and Response Length.
As shown in Fig. S.2 , we compare average rewards for iGRPO and GRPO at multiple checkpoints, observing that iGRPO consistently maintains a higher reward throughout training. The iterative refinement in iGRPO ultimately yields a superior reward trajectory. In addition, we measure the response length over training steps and find that both methods exhibit nearly identical lengths, with GRPO producing slightly longer outputs on average. Notably, iGRPO’s two-stage process does not manifest in lengthy completions but instead appears to refine solutions within a similar token budget. This indicates that the gains from iterative refinement arise more from improved response quality than from verbosity.
##### Effect of KL Divergence Term.
We vary the coefficient β ∈ { 0 , 0.0001 , 0.001 , 0.01 } \beta\in\{0,0.0001,0.001,0.01\} to examine how tightly the policy is regularized against the reference model. As shown in Table S.4 , while β = 0.0001 \beta=0.0001 achieves the highest overall score (70.23%), the difference among all settings is relatively small. The KL term, in principle, balances exploration with adherence to the current policy. However, given the marginal gains observed, setting β = 0 \beta=0 offers a simpler training pipeline without sacrificing significant performance. Hence, we use β = 0 \beta=0 to reduce overhead and maintain efficiency.
##### Effect of Number of Completions.
We study how the total number of completions in iGRPO affects performance, allocating 4 , 8 , 16 , 4,8,16, or 32 32 completions evenly across the two stages. As shown in Table S.4 , increasing from 4 4 to 8 8 completions gives a clear improvement, while gains beyond 8 8 are modest. Larger budgets also increase training time and inference latency for minimal returns.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .