Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 6/7

Cleaned-text code points [78321, 93861). Bibliography omitted here; source.md includes it.

Standard Policy Gradient methods maximize 𝔼 ​ [ log ⁡ π θ ​ ( y ) ] \mathbb{E}[\log\pi_{\theta}(y)] . However, for refined responses where π θ ​ ( y ref ) ≪ 1 \pi_{\theta}(y_{\text{ref}})\ll 1 , standard gradients can be unstable or vanish. We employ the shaping function ρ t ​ ( θ ) = π t π t + γ \rho_{t}(\theta)=\frac{\pi_{t}}{\pi_{t}+\gamma} .

###### Lemma F.13 (Gradient Rescaling) .

Let π t = π θ ​ ( a t | s t ) \pi_{t}=\pi_{\theta}(a_{t}|s_{t}) . The gradient of the shaped objective with respect to the policy parameters θ \theta is a re-weighted version of the standard policy gradient: ∇ θ ρ t ​ ( θ ) = [ γ ​ π t ( π t + γ ) 2 ] ⏟ Ψ ⁡ ( π t ) ⋅ ∇ θ ​ log ​ π t . \nabla_{\theta}\rho_{t}(\theta)=\underbrace{\left[\frac{\gamma\pi_{t}}{(\pi_{t}+\gamma)^{2}}\right]}_{\Psi(\pi_{t})}\cdot\nabla_{\theta}\log\pi_{t}. (23)

###### Proof.

Applying the quotient rule to ρ t \rho_{t} with respect to π t \pi_{t} : ∂ ρ t ∂ π t = 1 ⋅ ( π t + γ ) − π t ⋅ 1 ( π t + γ ) 2 = γ ( π t + γ ) 2 . \frac{\partial\rho_{t}}{\partial\pi_{t}}=\frac{1\cdot(\pi_{t}+\gamma)-\pi_{t}\cdot 1}{(\pi_{t}+\gamma)^{2}}=\frac{\gamma}{(\pi_{t}+\gamma)^{2}}. (24) By the chain rule, ∇ θ ρ t = ∂ ρ t ∂ π t ​ ∇ θ π t \nabla_{\theta}\rho_{t}=\frac{\partial\rho_{t}}{\partial\pi_{t}}\nabla_{\theta}\pi_{t} . Using the log-derivative identity ∇ θ π t = π t ​ ∇ θ ​ log ⁡ π t \nabla_{\theta}\pi_{t}=\pi_{t}\nabla_{\theta}\log\pi_{t} , we substitute to obtain: ∇ θ ρ t = γ ​ π t ( π t + γ ) 2 ​ ∇ θ ​ log ⁡ π t . \nabla_{\theta}\rho_{t}=\frac{\gamma\pi_{t}}{(\pi_{t}+\gamma)^{2}}\nabla_{\theta}\log\pi_{t}. (25) ∎

##### Analysis of the Modulation Term Ψ ⁡ ( π t ) \Psi(\pi_{t}) .

The coefficient Ψ ⁡ ( π t ) \Psi(\pi_{t}) acts as a dynamic gain controller, theoretically aligning with the ”hard negative mining” mechanism found in Focal Loss ( Lin et al., 2017 ) . 1. Suppression of Easy Tokens ( π t → 1 \pi_{t}\to 1 ): lim π t → 1 Ψ ⁡ ( π t ) ≈ γ ( 1 + γ ) 2 \lim_{\pi_{t}\to 1}\Psi(\pi_{t})\approx\frac{\gamma}{(1+\gamma)^{2}} . Since γ ≪ 1 \gamma\ll 1 , the gradient magnitude is significantly dampened. This prevents the model from overfitting to tokens it has already mastered, preserving the KL-divergence on trivial tokens.

2. Peak Efficiency in the Learning Zone: Solving d ​ Ψ d ​ π = 0 \frac{d\Psi}{d\pi}=0 yields a maximum at π t = γ \pi_{t}=\gamma . By setting γ ≈ 0.1 \gamma\approx 0.1 , we explicitly maximize gradients for ”correction tokens”—those that the current policy considers unlikely but plausible. This focuses the update on the reasoning gap between the initial and refined response.

#### F.6.2 The Insufficiency of Clipping for Off-Policy Data

Standard PPO/GRPO relies on clipping the probability ratio r t r_{t} to [ 1 − ϵ , 1 + ϵ ] [1-\epsilon,1+\epsilon] to enforce a trust region ( Schulman et al., 2017 ) . While effective for on-policy data, we prove that clipping is insufficient for off-policy refinements due to vector orthogonality.

###### Proposition F.14 (Gradient Orthogonality) .

Let ℳ old \mathcal{M}_{\text{old}} be the manifold of the current policy. The gradient update from refinements, Δ ​ θ ref \Delta\theta_{\text{ref}} , is often orthogonal to the update from initial responses, Δ ​ θ init \Delta\theta_{\text{init}} .

Reasoning: Δ ​ θ init \Delta\theta_{\text{init}} reinforces the current mode of the distribution. Conversely, Δ ​ θ ref \Delta\theta_{\text{ref}} attempts to shift probability mass to a disjoint region of the sample space ( y ref y_{\text{ref}} ). Optimizing Δ ​ θ ref \Delta\theta_{\text{ref}} inherently requires reducing the probability of y init y_{\text{init}} . Therefore: ⟨ ∇ θ 𝒥 init , ∇ θ 𝒥 ref ⟩ < 0 . \langle\nabla_{\theta}\mathcal{J}_{\text{init}},\nabla_{\theta}\mathcal{J}_{\text{ref}}\rangle<0. (26) Clipping bounds the magnitude of the step but does not correct this conflicting direction . Without regularization, the accumulation of Δ ​ θ ref \Delta\theta_{\text{ref}} updates drags the policy off the manifold ℳ old \mathcal{M}_{\text{old}} , leading to entropy collapse.

#### F.6.3 Manifold Preservation via The Anchor Hypothesis

To prevent this collapse, we derive the necessity of the sample prioritization ratio N init ≫ N ref N_{\text{init}}\gg N_{\text{ref}} .

###### Theorem F.15 (The Anchor Condition) .

Let 𝐅 \mathbf{F} be the Fisher Information Matrix of the policy. To ensure the updated policy π θ ′ \pi_{\theta^{\prime}} remains within a trust region D KL ( π θ ′ | | π θ ) ≤ δ D_{\text{KL}}(\pi_{\theta^{\prime}}||\pi_{\theta})\leq\delta , the influence of refined samples must be bounded by the curvature of the on-policy manifold.

###### Proof.

We approximate the KL divergence using the second-order Taylor expansion ( Amari et al., 2019 ; Kakade and Langford, 2002 ) : D KL ( π θ + Δ ​ θ | | π θ ) ≈ 1 2 Δ θ T 𝐅 Δ θ . D_{\text{KL}}(\pi_{\theta+\Delta\theta}||\pi_{\theta})\approx\frac{1}{2}\Delta\theta^{T}\mathbf{F}\Delta\theta. (27) The total update is a convex combination: Δ ​ θ ∝ ( 1 − λ ) ​ g init + λ ​ g ref \Delta\theta\propto(1-\lambda)g_{\text{init}}+\lambda g_{\text{ref}} , where λ = N ref / N \lambda=N_{\text{ref}}/N . Since g init g_{\text{init}} is clipped on-policy, its contribution to the divergence is bounded by design. The risk arises from g ref g_{\text{ref}} , which is high-variance and off-distribution. If λ \lambda is large, the term λ ​ g ref \lambda g_{\text{ref}} dominates. Since g ref g_{\text{ref}} is not aligned with the eigenvectors of 𝐅 \mathbf{F} (which represent the current policy’s geometry), the term Δ ​ θ T ​ 𝐅 ​ Δ ​ θ \Delta\theta^{T}\mathbf{F}\Delta\theta grows rapidly, signifying a departure from the trust region. By enforcing N init ≫ N ref N_{\text{init}}\gg N_{\text{ref}} (small λ \lambda ), the term ( 1 − λ ) ​ g init (1-\lambda)g_{\text{init}} acts as a regularization term (an ”Anchor”). It ensures that the primary update direction respects the local curvature 𝐅 \mathbf{F} , while g ref g_{\text{ref}} acts as a small perturbation vector that rotates the policy toward higher reward regions without shattering the manifold. ∎

## Appendix G Implementation Details

Datasets and Evaluation Metrics. We use randomly sampled subsets of 4k examples from a reorganized 45k subset ( Yan et al., 2025 ) of OpenR1-Math-220k ( Bakouch et al., 2025 ) as the training set (as described in Section 3 ). For validation, we use the curated validation set provided by ( Yan et al., 2025 ) . We evaluate the model on five well-established mathematical reasoning benchmarks: MATH-500 ( Hendrycks et al., 2021 ) , Minerva-Math ( Lewkowycz et al., 2022 ) , OlympiadBench ( He et al., 2024b ) , AIME 2024 ( Li et al., 2024 ) , AIME 2025 ( Li et al., 2024 ) , and AMC 2023 ( Li et al., 2024 ) . For broader analysis, we assess the model’s generalization ability on three scientific and general reasoning tasks: TheoremQA (Math, Physics, EE&CS, and Finance) ( Chen et al., 2023 ) , GPQA-Diamond (Physics, Chemistry, and Biology) ( Rein et al., 2024 ) , and MMLU-Pro (Business, Computer Science, Law, etc. ) ( Wang et al., 2024 ) . During evaluation, we use greedy decoding (temperature = 0) and report pass@1 over three runs.

Reward Design. During RL fine-tuning, we use model-based evaluation to generate critiques and rule-based evaluation to provide binary scalar rewards, as described in Section 3 .

Compared Methods. We compare Critique-GRPO against the following representative approaches, categorized into supervised learning and reinforcement learning-based finetuning. All differences are considered significant at p < 0.01 p<0.01 .

Supervised Learning-based Finetuning:

( i ) (\textup{i}) Supervised Finetuning (SFT) : Finetuning the base model on high-quality annotated training data using supervised learning.

( ii ) (\textup{ii}) Reward rAnked Finetuning (RAFT) ( Dong et al., 2023 ) : Finetuning on self-generated correct responses, sampled based on rule-based evaluation.

( iii ) (\textup{iii}) Refinement Finetuning (Refinement FT) ( Chen et al., 2024 ) : Finetuning on refined correct responses generated conditionally on the question, initial response, and CoT critiques.

( iv ) (\textup{iv}) Critique Finetuning (Critique FT) ( Wang et al., 2025 ) : Finetuning on annotated CoT critique data to train the model to critique a given query-response.

( v ) (\textup{v}) Critique-in-the-Loop Finetuning (CITL-FT) ( Xi et al., 2024 ) : Finetuning on mixed data consisting of self-generated correct responses and refined correct responses, conditioned on the question-initial response-CoT critique triplet.

Reinforcement Learning-based Finetuning:

( vi ) (\textup{vi}) R1-GRPO ( DeepSeek-AI et al., 2025 ) : Finetuning the base model on its own generations using the GRPO algorithm with binary scalar rewards.

( vii ) (\textup{vii}) R1-Dr.GRPO ( Liu et al., 2025a ) : Finetuning the base model on its own generations using the Dr.GRPO algorithm, which removes terms that cause biased optimization, with binary scalar rewards.

( viii ) (\textup{viii}) Critique-GRPO (Indicative Critique) : Fine-tuning the base model with Critique-GRPO by utilizing indicative critiques for refinements, as described in Appendix E .

( ix ) (\textup{ix}) Critique-GRPO (Critique with Ground Truth) : Fine-tuning the base model with Critique-GRPO by leveraging indicative critiques alongside ground-truth answers for refinements, as detailed in Appendix E .

Implementation Details. We conduct experiments using Qwen2.5-7B-Base, Qwen2.5-Math-7B-Base, and Qwen3-8B, with GPT-4o (which can be replaced by other reasoning-based reward models) serving as the reward model, as described in Section 3 . For supervised finetuning baselines, models are finetuned until convergence, and the best performance is reported. For RL-based approaches, models are finetuned for 400 steps, and the best performance is recorded. To ensure a fair comparison: In R1-GRPO, 8 responses (rollouts) are sampled per training prompt with a temperature of 1. In Critique-GRPO, 7 responses are sampled per prompt, along with one refined response from the refinement sets. we present detailed hyperparameters and training configurations in Table 13 .

All experiments are conducted on 40 NVIDIA A800 80G GPUs. To ensure consistency, we use only critiques generated by the reward model that align with rule-based evaluations; otherwise, the reward model is prompted to regenerate the critiques. Following prior works ( Liu et al., 2025a ) , for evaluation, we adopt greedy sampling (temperature set to 0) to generate responses and report pass@1 as the evaluation metric. When reporting pass@k, we uniformly set the temperature to 0.6 and the top-p value to 0.95.

Compared Methods in the Investigation of Math-Centric Backbone Models. We evaluate the efficacy of RL fine-tuning with Critique-GRPO on the math-centric backbone model, Qwen2.5-Math-7B-Base. Specifically, we compare its performance against three representative RL fine-tuning approaches based solely on numerical feedback:

( i ) (\textup{i}) SimpleRL-Zero ( Zeng et al., 2025 ) : an open-source reproduction of R1-GRPO.

( ii ) (\textup{ii}) PRIME-Zero ( Cui et al., 2025a ) : fine-tuning the base model using both outcome binary rewards and process binary rewards.

( iii ) (\textup{iii}) Oat-Zero ( Liu et al., 2025a ) : fine-tuning the base model with Dr.GRPO using outcome binary rewards.

## Appendix H Detailed Results and Analysis of Self-Critiquing Mechanisms

To explore the potential of Critique-GRPO in enabling an LLM’s self-improvement through self-critiquing, we prompt the model itself to serve as a reasoning-based reward model. Specifically, we investigate two types of self-critiquing: ( i ) (\textup{i}) Self-critique, where the model evaluates the correctness of its own responses using CoT critiques with ground truth answers as reference; and ( ii ) (\textup{ii}) Self-critique & self-evaluation ( Zhang et al., 2024 ) , where the model evaluates its responses using CoT critiques without any reference. These approaches result in Critique-GRPO (self-critique) and Critique-GRPO (self-critique & self-evaluation), respectively. Details of the prompts are provided in Appendix M . Table 14 shows the evaluation results on Qwen3-8B, and Figure 9 presents pass@k performance changes on AIME24 and AIME25 ( Li et al., 2024 ) .

Critique-GRPO enhances self-improvement through self-critiquing. Table 14 RL fine-tuning with Critique-GRPO (self-critique) significantly outperforms fine-tuning with GRPO using external numerical feedback (R1-GRPO) and supervised fine-tuning with expert demonstrations (SFT). On average, Critique-GRPO (self-critique) improves pass@1 by +4.5% and +12.0% compared to R1-GRPO and SFT, respectively. Additionally, the unsupervised approach—Critique-GRPO (self-critique & self-evaluation)—achieves an average pass@1 improvement of 2.3% over R1-GRPO, highlighting the potential of leveraging self-critique for self-improvement without any external supervision .

Self-critiquing aids effective exploration. Figure 9 highlights the consistently superior performance of Critique-GRPO (self-critique) across pass@k metrics, with k k ranging from 1 to 256, indicating genuine improvements. Notably, Critique-GRPO (self-critique) achieves remarkable gains over R1-GRPO for pass@k with k = 1 k=1 to 4 4 , yielding improvements of 10-16.7% on AIME24 (Figure 9(a) ).

## Appendix I Detailed Investigation of Policy Exploration During RL Finetuning

To investigate policy exploration, we analyze two primary aspects of our RL-finetuned models: ( i ) (\textup{i}) entropy dynamics during RL fine-tuning for self-improvement using compared RL-based finetuning approaches on Qwen2.5-7B-Base and Critique-GRPO (self-critique) on Qwen3-8B (Figure 10 ), and ( ii ) (\textup{ii}) changes in response length during fine-tuning (Figure 11 ).

Learning from natural language feedback helps sustain exploration. As shown in Figure 10(a) , the policy entropy of Critique-GRPO generally remains higher than that of R1-GRPO and R1-Dr.GRPO, suggesting more consistent exploration. The peaks in Critique-GRPO’s entropy dynamics (before step 200) likely occur when its self-generated refinements deviate significantly from the initial sampled responses, leading to increased entropy and potentially beneficial distributional shifts. The subsequent decrease in entropy indicates that the model quickly internalizes these refinements, reducing the distributional deviation. This dynamic aligns with the observation that rare actions with high advantage can increase policy entropy ( i.e., unfamiliar but correct responses with high rewards promote effective exploration ), whereas high-probability actions with high advantage tend to reduce entropy ( Cui et al., 2025b ) . In contrast, R1-GRPO exhibits entropy collapse , where policy entropy drops sharply at the start of training and continues to decline monotonically to near zero. R1-Dr.GRPO initially exhibits higher entropy (before step 50) but rapidly drops to comparable near-zero values with R1-GRPO after step 150. Combined with the results in Table 2 , the superior performance of Critique-GRPO over R1-Dr.GRPO and R1-GRPO highlights the importance of maintaining a certain level of entropy for better performance.

Learning through self-critiquing facilitates policy exploration. Figure 10(b) shows that Critique-GRPO (self-critique) avoids entropy collapse and maintains higher entropy than R1-GRPO. This finding aligns with the observation that increased exploration improves performance.