Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 2/7

Cleaned-text code points [15464, 31323). Bibliography omitted here; source.md includes it.

Step 2: Critique-Guided Self-Refinement. To optimize computational efficiency, we initiate refinement only when the initial response set { y ( i ) } i = 1 n \{y^{(i)}\}_{i=1}^{n} contains zero correct solutions. We generate refined responses via in-context learning conditioned on the question-response-critique triplet ( q , y ( i ) , c ( i ) ) (q,y^{(i)},c^{(i)}) and a refinement instruction I refine I_{\text{refine}} (detailed in Appendix M and Appendix E ): y refined ( i ) ∼ π old ( ⋅ ∣ I refine , q , y ( i ) , c ( i ) ) , y_{\text{refined}}^{(i)}\sim\pi_{\text{old}}(\cdot\mid I_{\text{refine}},q,y^{(i)},c^{(i)}), where c ( i ) ∈ { c CoT ( i ) , c GT ( i ) , c I ( i ) } c^{(i)}\in\{c_{\text{CoT}}^{(i)},c_{\text{GT}}^{(i)},c_{\text{I}}^{(i)}\} . These refinements are evaluated by the reward system to obtain scores { R refine ( i ) } i = 1 n \{R_{\text{refine}}^{(i)}\}_{i=1}^{n} . To mitigate entropy explosion from significant distribution shifts induced by refinements, we sample a subset of k k refinements { y refined ( i ′ ) } i ′ = 1 k \{y_{\text{refined}}^{(i^{\prime})}\}_{i^{\prime}=1}^{k} from the full set, prioritizing correct solutions; if no correct refinements exist, we sample randomly. This subset is combined with the initial responses to form the final training group: { y ( i ) } i = 1 n ∪ { y refined ( i ′ ) } i ′ = 1 k \{y^{(i)}\}_{i=1}^{n}\cup\{y_{\text{refined}}^{(i^{\prime})}\}_{i^{\prime}=1}^{k} .

We justify our critique mechanism’s sample efficiency via the Transfer Eluder Dimension framework ( Xu et al., 2025 ) . By characterizing critiques as Reward-Informative Feedback ( Cheng et al., 2023 ) , we show they distinguish hypotheses more efficiently than scalar rewards.

###### Proposition 4.1 (Complexity Reduction via Critique-Guided Exploration) .

Consider a reasoning problem where the goal is to construct a hidden optimal solution sequence of L L steps, a ∗ = ( s 1 ∗ , … , s L ∗ ) a^{*}=(s^{*}_{1},\dots,s^{*}_{L}) , with each s i ∈ 𝒮 s_{i}\in\mathcal{S} . The action set is 𝒜 = ⋃ k = 1 L 𝒮 k \mathcal{A}=\bigcup_{k=1}^{L}\mathcal{S}^{k} . The hypothesis space is ℱ \mathcal{F} .

Standard Generation (Reward-Only): Binary rewards only indicate if a final state is correct ( R = 1 R=1 ) or not ( R = 0 R=0 ). Since incorrect sequences provide identical zero-information, the agent must effectively enumerate the action space 𝒜 \mathcal{A} . The Eluder dimension ( Russo and Van Roy, 2013 ) scales exponentially: dim E ( ℱ ) ≈ O ⁡ ( | 𝒮 | L ) \dim_{E}(\mathcal{F})\approx O(|\mathcal{S}|^{L}) .

Critique-Guided Refinement (Informative Feedback): (1) Indicative Feedback ( c I c_{\text{I}} , c GT c_{\text{GT}} ): While the worst-case complexity remains O ⁡ ( | 𝒮 | L ) O(|\mathcal{S}|^{L}) , the critique acts as a pruning signal. By conditioning on the failure and critique, the policy restricts its search to a subspace 𝒜 c ⊂ 𝒜 \mathcal{A}_{c}\subset\mathcal{A} consistent with the critique. (2) Constructive Feedback ( c CoT c_{\text{CoT}} ): If the critique localizes the first error step t t , the problem decomposes into L L independent sub-problems of size | 𝒮 | |\mathcal{S}| , reducing complexity to linear O ⁡ ( | 𝒮 | ​ L ) O(|\mathcal{S}|L) . If the critique provides the correction suggestion for the first error s t ∗ s_{t}^{*} , complexity becomes independent of the search space O ⁡ ( L ) O(L) .

Consequently, for a fixed computational budget M M where | 𝒮 | ​ L ≪ M ≪ | 𝒮 | L |\mathcal{S}|L\ll M\ll|\mathcal{S}|^{L} , critique-guided exploration yields a higher probability of success: P ⁡ ( a ∗ ∈ { y refined ( j ) } j = 1 M ) ≫ P ⁡ ( a ∗ ∈ { y ( i ) } i = 1 M ) P(a^{*}\in\{y_{\text{refined}}^{(j)}\}_{j=1}^{M})\gg P(a^{*}\in\{y^{(i)}\}_{i=1}^{M}) . (See Appendix F.4 for the full analysis).

Step 3: Online Policy Optimization. Finally, we fine-tune the model on the mixed set of initial and refined responses using scalar rewards. The training objective is: 𝒥 Critique-GRPO ​ ( θ ) = 𝒥 init ​ ( θ ) + 𝒥 refi ​ ( θ ) , \mathcal{J}_{\text{Critique-GRPO}}(\theta)=\mathcal{J}_{\text{init}}(\theta)+\mathcal{J}_{\text{refi}}(\theta), (2) The objective for initial responses follows the standard GRPO formulation: 𝒥 init ( θ ) = 𝔼 q ∼ Q , { y ( i ) } i = 1 n ∼ π old ( ⋅ ∣ q ) [ 1 n ∑ i = 1 n ∑ t = 1 | y ( i ) | \displaystyle\mathcal{J}_{\text{init}}(\theta)=\mathbb{E}_{q\sim Q,\{y^{(i)}\}_{i=1}^{n}\sim\pi_{\text{old}}(\cdot\mid q)}\bigg[\frac{1}{n}\sum_{i=1}^{n}\sum_{t=1}^{|y^{(i)}|} (3) min ( r t ( i ) ( θ ) A ^ t ( i ) , clip ( r t ( i ) ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ( i ) ) ] \displaystyle\min\big(r_{t}^{(i)}(\theta)\hat{A}_{t}^{(i)},\text{clip}(r_{t}^{(i)}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}^{(i)}\big)\bigg] The objective for refined responses are presented as follows: 𝒥 refi ( θ ) = 𝔼 q ∼ Q , { y refined ( i ′ ) } i ′ = 1 k ∼ π old ( ⋅ ∣ q ) [ 1 k ∑ i ′ = 1 k ∑ t = 1 | y refined ( i ′ ) | \displaystyle\mathcal{J}_{\text{refi}}(\theta)=\mathbb{E}_{q\sim Q,\{y_{\text{refined}}^{(i^{\prime})}\}_{i^{\prime}=1}^{k}\sim\pi_{\text{old}}(\cdot\mid q)}\bigg[\frac{1}{k}\sum_{i^{\prime}=1}^{k}\sum_{t=1}^{|y_{\text{refined}}^{(i^{\prime})}|} (4) min ( ρ t ( i ′ ) ( θ ) A ^ t ( i ′ ) , clip ( ρ t ( i ′ ) ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ( i ′ ) ) ] \displaystyle\min\big(\rho_{t}^{(i^{\prime})}(\theta)\hat{A}_{t}^{(i^{\prime})},\text{clip}(\rho_{t}^{(i^{\prime})}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}^{(i^{\prime})}\big)\bigg] The advantages A ^ t ( i ) \hat{A}_{t}^{(i)} and A ^ t ( i ′ ) \hat{A}_{t}^{(i^{\prime})} are computed using the group mean of rewards from both the initial and refined sets to ensure a unified baseline: A ^ t ( i / i ′ ) = R ( i / i ′ ) − mean ​ ( { R ( i ) } i = 1 n ∪ { R refined ( i ′ ) } i ′ = 1 k ) \hat{A}_{t}^{(i/i^{\prime})}=R^{(i/i^{\prime})}-\text{mean}\big(\{R^{(i)}\}_{i=1}^{n}\cup\{R_{\text{refined}}^{(i^{\prime})}\}_{i^{\prime}=1}^{k}\big) . For initial responses, r t ( i ) ​ ( θ ) r_{t}^{(i)}(\theta) is the standard importance sampling ratio. For refined responses, we employ a shaping function to define the policy ratio ρ t ( i ′ ) ​ ( θ ) \rho_{t}^{(i^{\prime})}(\theta) ( Yan et al., 2025 ) : ρ t ( i ′ ) ​ ( θ ) = π θ ​ ( y refined , t ( i ′ ) ∣ q , y refined , < t ( i ′ ) ) π θ ​ ( y refined , t ( i ′ ) ∣ q , y refined , < t ( i ′ ) ) + γ \rho_{t}^{(i^{\prime})}(\theta)=\frac{\pi_{\theta}(y_{\text{refined},t}^{(i^{\prime})}\mid q,y_{\text{refined},<t}^{(i^{\prime})})}{\pi_{\theta}(y_{\text{refined},t}^{(i^{\prime})}\mid q,y_{\text{refined},<t}^{(i^{\prime})})+\gamma} ( 0 < γ < 1 0<\gamma<1 ). The shaping term γ \gamma reweights gradients to assign higher importance to tokens in the refined responses that are currently low-probability under π θ \pi_{\theta} , facilitating efficient learning from valid but unfamiliar refinements. We remove the KL-divergence penalty to enable substantial policy updates towards these refinements. Detailed analyses are provided in Appendix F .

## 5 Experiments

### 5.1 Experimental Setup

Datasets and Evaluation Metrics. We use randomly sampled subsets of 4k examples from a reorganized 46k subset ( Yan et al., 2025 ) of OpenR1-Math-220k ( Bakouch et al., 2025 ) as the training set. For validation, we utilize the curated validation set provided by ( Yan et al., 2025 ) . The model is evaluated on five established mathematical reasoning benchmarks: MATH-500 ( Hendrycks et al., 2021 ) , Minerva-Math, OlympiadBench ( He et al., 2024b ) , AIME 2024/2025 ( Li et al., 2024 ) , and AMC 2023 ( Li et al., 2024 ) . To analyze out-of-distribution generalization, we further evaluate the model on three reasoning tasks spanning scientific and general domains: TheoremQA ( Chen et al., 2023 ) , GPQA-Diamond, and MMLU-Pro ( Wang et al., 2024 ) . For evaluation, we use greedy decoding (temperature = 0) and report pass@1 over three runs.

Compared Methods. We compare Critique-GRPO against the following approaches. All differences are considered significant at p < 0.01 p<0.01 . During RL fine-tuning, we utilize binary scalar rewards (+1 for correct responses and 0 for incorrect ones). 1. Supervised Learning-based Finetuning: ( i ) (\textup{i}) Supervised Fine-tuning (SFT) : Finetuning on high-quality annotated training data. ( ii ) (\textup{ii}) Reward Ranked Finetuning (RAFT) ( Dong et al., 2023 ) : Finetuning on correct initial responses. ( iii ) (\textup{iii}) Refinement Finetuning (Refinement FT) ( Chen et al., 2024 ) : Finetuning on correct refinements generated conditionally on CoT critiques. ( iv ) (\textup{iv}) Critique Finetuning (Critique FT) ( Wang et al., 2025 ) : Finetuning on CoT critiques. ( v ) (\textup{v}) Critique-in-the-Loop Finetuning (CITL-FT) ( Xi et al., 2024 ) : Fine-tuning on both correct initial responses and their refinements generated conditionally on CoT critiques. 2. Reinforcement Learning-based Finetuning: ( vi ) (\textup{vi}) R1-GRPO ( DeepSeek-AI et al., 2025 ) : Fine-tuning on initial responses using the GRPO algorithm with binary scalar rewards. ( vii ) (\textup{vii}) R1-Dr.GRPO ( Liu et al., 2025a ) : Fine-tuning on initial responses using the Dr.GRPO algorithm, which removes optimization bias terms, with binary scalar rewards. We implement Critique-GRPO using asynchronous rollouts via the VERL framework ( Sheng et al., 2024 ) for computational efficiency. Implementation details are in Appendix G .

### 5.2 Main Results

Table 2 presents our evaluation results, revealing three key findings:

Natural language feedback enhances online RL policy optimization. Critique-GRPO consistently outperforms both supervised and RL-based fine-tuning methods on Qwen2.5-7B-Base and Qwen3-8B across all tasks. Compared to R1-GRPO and R1-Dr.GRPO, Critique-GRPO improves average pass@1 by +4.4 points (42.66% → \rightarrow 47.08%) on Qwen2.5-7B-Base and +3.8 points (64.46% → \rightarrow 68.26%) on Qwen3-8B. See Appendix J for qualitative analysis.

Online self-refinement outperforms offline approaches. Critique-GRPO (CoT critique) surpasses Refinement FT by +11.9 points (47.08% vs. 35.21%) and +8.81 points (68.26% vs. 59.45%) in average pass@1 on Qwen2.5-7B-Base and Qwen3-8B, respectively. It also exceeds CITL-FT by +11.4 points (47.08% vs. 35.66%) on Qwen2.5-7B-Base and +12.4 points (68.26% vs. 55.84%) on Qwen3-8B.

Richer critiques yield superior refinements and policy optimization. Critique-GRPO with CoT critiques consistently outperforms its indicative critique variant, achieving average pass@1 improvements of +1.8–2.4% on Qwen2.5-7B-Base and +2.0–2.3% on Qwen3-8B. This advantage arises from CoT critiques’ ability to guide more effective refinements, as demonstrated in Section 3 . See Appendices B and J for computational cost and qualitative analyses.

### 5.3 Investigation on Math-Centric Backbone Models

We validate Critique-GRPO on the math-centric backbone model Qwen2.5-Math-7B-Base, comparing it against three RL fine-tuning approaches that use only numerical feedback: SimpleRL-Zero ( Zeng et al., 2025 ) , PRIME-Zero ( Cui et al., 2025a ) , and Oat-Zero ( Liu et al., 2025a ) (see Appendix G for details).

Natural language feedback overcomes performance plateaus from numerical-only approaches. Table 3 shows that Critique-GRPO achieves a 21.6% average pass@1 improvement over Qwen2.5-Math-7B-Base using only 4k RL training prompts, substantially outperforming numerical-feedback approaches that require 46k prompts.

Verbal credit assignment surpasses numerical credit assignment. Despite using only 4k prompts, Critique-GRPO (guided by outcome rewards and language critiques) consistently outperforms PRIME-Zero (which uses 46k prompts with dense intermediate rewards) by large margins.

### 5.4 Self-Improvement via Self-Critiquing

We explore Critique-GRPO’s capacity for LLM self-improvement through self-critiquing, yielding Critique-GRPO (self-critique). See Appendix H for details.

Critique-GRPO enables self-improvement via self-critiquing. Table 4 shows that Critique-GRPO (self-critique) improves average pass@1 by 4.5% over R1-GRPO and 12.0% over SFT.

Self-critiquing enhances exploration. Figures 3 and 9(b) demonstrate that Critique-GRPO (self-critique) consistently outperforms R1-GRPO and SFT across pass@k metrics on AIME24 and AIME25 for k ∈ [ 1,256 ] k\in[1,256] , indicating genuine improvements. See Appendix H for additional results.

### 5.5 Exploration with Varying Models

We validate Critique-GRPO’s robustness by (i) testing on different model architectures and scales, including Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 ) and Qwen3-32B ( Yang et al., 2025a ) (Appendix D ), and (ii) evaluating compatibility with alternative critique models: Llama3.1-405B ( Grattafiori et al., 2024 ) and DeepCritic-7B-RL1.5-PRM800K ( Yang et al., 2025b ) .

Critique-GRPO generalizes across model architectures and critique models. Table 5 (upper section) shows that Critique-GRPO consistently outperforms GRPO on all eight reasoning tasks with Llama-3.2-3B-Instruct and Qwen3-32B (Appendix D ), achieving average gains of 5.1 and 4.0 points, respectively, confirming robustness across architectures and scales. The lower section demonstrates compatibility with both proprietary and fine-tuned critique models, yielding average improvements of 5.6–6.8 points over GRPO, underscoring Critique-GRPO’s versatility.

### 5.6 Investigation of Policy Exploration

To investigate whether models learn from valuable explorations in refinements, we analyze the entropy dynamics of Qwen2.5-7B-Base in Figure 4 .

Learning from natural language feedback sustains exploration. Critique-GRPO maintains higher policy entropy than R1-GRPO and R1-Dr.GRPO, indicating more consistent exploration. Early entropy peaks (before step 200) occur when self-generated refinements deviate significantly from initial responses, increasing entropy and enabling beneficial distributional shifts. The subsequent entropy decrease reflects rapid internalization of these refinements. This pattern aligns with prior findings that rare, high-advantage actions increase policy entropy ( i.e., unfamiliar but correct responses promote effective exploration), while common, high-advantage actions reduce it ( Cui et al., 2025b ) . Together with Table 2 , Critique-GRPO’s superior performance demonstrates that maintaining higher entropy enhances policy optimization. This observation is consistent with theoretical analysis in Section 4 and Xu et al. (2025) , showing that critiques enable valuable explorations unreachable through standard generation. We provide a comprehensive analysis in Appendix I .

### 5.7 Fine-Grained Ablation Studies

We ablate Critique-GRPO’s objective modifications from GRPO: ( i ) (\textup{i}) removing KL regularization; ( ii ) (\textup{ii}) adding natural language feedback (fine-tuning on initial generations and one random refinement); ( iii ) (\textup{iii}) selecting higher-quality refinements; ( iv ) (\textup{iv}) applying policy shaping via token-level probability ratios.

Looser optimization and language feedback enable effective learning. Table 6 shows cumulative gains: removing KL (+1.5%), adding language feedback (+0.6% to 43.26%), quality-based selection (+0.7% to 43.95%), and policy shaping (+3.1% to 47.08%). The substantial gain from policy shaping demonstrates that amplifying rare but valuable reasoning patterns—those with low initial probability but high success rates—is crucial for RL fine-tuning.

### 5.8 Exploration of Weak-to-Strong Generalization

We investigate the potential of weak-to-strong generalization ( Burns et al., 2023 ) using Critique-GRPO, where a strong model learns from refinements generated by a weaker teacher model. Specifically, we use Qwen3-8B-Base ( Yang et al., 2025a ) as the weaker teacher to generate refinements based on indicative critiques with the ground truth answers, guiding the improvement of Qwen3-8B.