Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 4/7

Cleaned-text code points [46860, 62667). Bibliography omitted here; source.md includes it.

Models exhibit persistent failures on a subset of problems despite trial-and-error fine-tuning. We evaluated the best-performing RL-finetuned Qwen2.5-7B-Base and Qwen3-8B models on 4k training prompts. As shown in the left panel of Table 12 , these models consistently failed on approximately 29% and 3.75% of problems, respectively, with pass@4 = 0. This occurred despite undergoing trial-and-error fine-tuning, where correct responses are rewarded, and incorrect responses are penalized. While the reasoning model (Qwen3-8B) exhibits evident spontaneous self-reflection and significantly better performance with fewer persistent failures, it still struggles with certain problems when relying solely on numerical feedback.

A likely cause of these performance plateaus and persistent failures is the sparse informational content of numerical feedback. Scalar rewards often fail to convey why a response is correct or incorrect or how to improve multi-step reasoning. Furthermore, the limited effectiveness of spontaneous self-reflection exacerbates these challenges. Together, these limitations highlight the necessity of richer feedback mechanisms to enable more effective learning.

### E.2 Promise of Learning from Natural Language Feedback

To move beyond the limitations of purely numerical reward signals, we explore the potential of leveraging natural language feedback to help models identify errors and refine their responses. Specifically, we examine three types of critiques: • Indicative Critique : A heuristic-based critique that merely indicates the binary correctness of the generated solution, without an expert trajectory .

• Indicative Critique with Ground Truth : A heuristic-based critique providing both binary correctness and the ground truth answer, without an expert trajectory .

• CoT Critique : A model-generated critique offering a step-by-step evaluation to justify the assessment, concluding with a binary correctness label, without an expert trajectory .

Examples of these critique types are presented below. We summarize our method for leveraging textual critiques to guide response refinement in Algorithm 1 . A detailed description is provided in Section E.3 , and an illustrative example of refinement using CoT critique is available in Appendix N .

##### Setup.

As described in the previous section, we evaluate the best-performing RL-finetuned Qwen2.5-7B-Base and Qwen3-8B models, generating four responses per question for a set of 4k prompts. Persistently failed question-response pairs are identified as those with pass@4 = 0. For each such pair, we prompt GPT-4o ( Hurst et al., 2024 ) to generate a CoT critique and then construct the two corresponding heuristic-based critiques. Examples of these critique types are shown below. Subsequently, we prompt the models to refine these failed responses.

Evaluation Metrics. To assess the efficacy of the critique and refinement process, we define the following metrics: ( i ) (\textup{i}) Valid Critiques Generated : The percentage of initially erroneous responses for which the critique model ( π ϕ \pi_{\phi} ) successfully generated a well-formed and usable critique. This accounts for potential failures in the critique generation process itself. ( ii ) (\textup{ii}) Successful Refinements : The percentage of initially erroneous responses (that received a valid critique) for which the LLM ( π θ \pi_{\theta} ) subsequently generated a correct refined response. ( iii ) (\textup{iii}) Critiqued Questions : The percentage of unique questions (all of whose initial k k responses were erroneous) for which at least one of their erroneous responses received a valid critique. ( iv ) (\textup{iv}) Questions Successfully Refined : The percentage of unique questions (all of whose initial k k responses were erroneous) for which at least one of their erroneous responses was successfully corrected through the refinement process. We evaluate the correctness of the refinements using the rule-based method described earlier.

Results. Deliberate critique is more effective than spontaneous self-reflection for self-correction. Incorporating all three types of critiques yields non-zero valid refinements and correctly refined questions. This suggests that critiques can enable both models to successfully correct some persistently failed responses that cannot be resolved through spontaneous self-reflection, as discussed in Section 3.1 .

CoT Critiques facilitate effective model refinement. Table 12 shows that refinement guided by CoT critiques achieves the highest valid refinement rate (36.47% and 10.63%) and the largest percentage of successfully refined questions (55.37% and 20.00%) on Qwen2.5-7B-Base and Qwen3-8B, respectively. This performance significantly surpasses refinement based on indicative critiques or critiques with ground truth, even though the CoT critique generation process produces valid critiques for only 60.06% and 50.17% of erroneous responses. The effectiveness of CoT critiques can be attributed to their richness: by providing a step-by-step evaluation of the reasoning ( potentially along with targeted guidance on the correct solution approach).

Binary correctness signals alone can provide refinement benefits. Refinement with indicative critiques with/without ground truth also yields some successful refinements, albeit at a substantially lower rate (approximately 2%-4% valid refinements). This suggests that even simply indicating the correctness of a response can provide a minimal benefit, indicating some promise in leveraging natural language feedback to augment learning from numerical signals. However, the lack of a substantial difference between indicative critiques and critiques with ground truth suggests that providing only the ground-truth answer, without any explanation or reasoning, provides little additional guidance to the model.

### E.3 Leveraging Textual Critiques for Refining LLM Responses

We describe the process for leveraging these textual critiques to guide the refinement of LLM-generated responses:

1. Initial Response Sampling : Given an LLM π θ \pi_{\theta} parameterized by θ \theta and a set of questions { q } \{q\} , we sample multiple initial responses for each question { y 0 ( i ) } i = 1 k ∼ π θ ( ⋅ ∣ q ) \{y_{0}^{(i)}\}_{i=1}^{k}\sim\pi_{\theta}(\cdot\mid q) , where k k is the number of samples.

2. Response Evaluation and Critique Generation : We use an evaluation function Eval ​ ( q , y 0 ) \text{Eval}(q,y_{0}) to assess the correctness of each response y 0 y_{0} . The function outputs 1 if y 0 y_{0} is correct and 0 otherwise. Specifically, we adopt a model-based evaluation with a reasoning-based reward model π R ​ M \pi_{RM} . The reasoning-based reward model generates a CoT critique c CoT ( i ) ∼ π R ​ M ( ⋅ ∣ I c , q , y 0 ( i ) ) c_{\text{CoT}}^{(i)}\sim\pi_{RM}(\cdot\mid I_{c},q,y_{0}^{(i)}) , where I c I_{c} is a predefined instruction (detailed in Appendix M ). Based on the binary correctness label within c CoT ( i ) c_{\text{CoT}}^{(i)} , we construct the corresponding heuristic-based critiques: an indicative critique c I ( i ) c_{\text{I}}^{(i)} (containing only the correctness label) and a critique with ground truth c GT ( i ) c_{\text{GT}}^{(i)} (correctness label plus the known ground truth answer for q q ).

To focus on the model’s ability to learn from critiques for initially incorrect solutions and to control for spontaneous self-correction, we identify persistently failed questions. A question q q is classified as persistently failed if all k k of its initial responses { y 0 ( i ) } i = 1 k \{y_{0}^{(i)}\}_{i=1}^{k} are deemed incorrect based on the labels from their respective CoT critiques. For each such incorrect response y 0 ( j ) y_{0}^{(j)} from a persistently failed question, we form a triplet ( q , y 0 ( j ) , c ( j ) ) (q,y_{0}^{(j)},c^{(j)}) , where c ( j ) c^{(j)} is one of the three critique types: c CoT ( j ) c_{\text{CoT}}^{(j)} , c GT ( j ) c_{\text{GT}}^{(j)} , or c I ( j ) c_{\text{I}}^{(j)} .

3. Self-Refinement Generation : For each selected triplet ( q , y 0 ( j ) , c ( j ) ) (q,y_{0}^{(j)},c^{(j)}) corresponding to an initial incorrect response, we prompt the original LLM π θ \pi_{\theta} to generate a refined response y refined ( j ) ∼ π θ ( ⋅ ∣ I refine , q , y 0 ( j ) , c ( j ) ) y_{\text{refined}}^{(j)}\sim\pi_{\theta}(\cdot\mid I_{\text{refine}},q,y_{0}^{(j)},c^{(j)}) . This generation is conditioned on a specific refinement instruction I refine I_{\text{refine}} (detailed in Appendix M ), the original question q q , the initial failed response y 0 ( j ) y_{0}^{(j)} , and its associated critique c ( j ) c^{(j)} .

The full process is summarized in Algorithm 1 . An example illustrating the self-refinement process, including the application of a CoT critique, is provided in Appendix N .

### E.4 Investigation on Qwen3-8B-Base

We identify 50 previously unsolved problems from the Minerva-Math dataset and 15 from the GPQA-Diamond dataset for Qwen3-8B-Base. Figure 7 shows the average contribution of reasoning behaviors to successful completions of previously failed questions by Qwen3-8B-Base. Notably, self-reflection behaviors contribute minimally to successful problem-solving.

Table 12 reveals that the best-performing RL-finetuned Qwen3-8B-Base persistently failed on 17.18% of training problems. In addition, all three types of critiques facilitate the LLM’s self-refinements.

## Appendix F More Details about Critique-GRPO

### F.1 The Critique-GRPO Algorithm

The Critique-GRPO algorithm is summarized in Algorithm 2 .

### F.2 The shaping Function in Critique-GRPO

We adopt a shaping function f ⁡ ( x ) = x / ( x + γ ) f(x)=x/(x+\gamma) ( Yan et al., 2025 ) ( 0 < γ < 1 0<\gamma<1 ), depicted in Figure 8 , to reweight gradients and emphasize low-probability tokens in refined responses. As illustrated in Figure 8 , this function is bounded between ( 0 , 1 ) (0,1) , where x x represents the token probability of the policy. When γ \gamma is small ( i.e., 0.1), the function significantly amplifies low probabilities, with this amplification decreasing as x x increases. Larger γ \gamma values (0.9) produce less pronounced scaling effects. The black dashed diagonal line indicates no shaping ( i.e., f ⁡ ( x ) = x f(x)=x ). We set γ = 0.1 \gamma=0.1 to optimize learning from unfamiliar yet correct refinements while strongly penalizing unfamiliar incorrect ones.

### F.3 Ratio of Responses per Prompt in Critique-GRPO

We experiment with the ratio of initial responses to refinements per prompt, ranging from 1:1 to 7:1. Our results indicate that a 7:1 ratio achieves both stable training and optimal performance. Lower ratios lead to performance degradation due to sudden increases in entropy loss, caused by distribution shifts introduced by the refinements, during later training stages.

### F.4 Theoretical Analysis of Sample Efficiency

In this section, we provide a theoretical justification for the superior sample efficiency of critique-guided refinement compared to standard reward-based learning. We utilize the Transfer Eluder Dimension ( Xu et al., 2025 ) framework to quantify the information gain provided by language feedback. We first formalize our critique mechanism as Reward-Informative Feedback , establishing that it allows the policy to distinguish hypotheses significantly more efficiently than scalar rewards alone. We then apply this framework to reasoning tasks to demonstrate an exponential reduction in sample complexity.

##### Preliminaries: Complexity Measures and Verifiers.

To characterize the complexity of the hypothesis space ℋ \mathcal{H} , we first introduce the Eluder Dimension ( Russo and Van Roy, 2013 ) , a standard measure in reinforcement learning that quantifies how many observations are required to reduce uncertainty about a reward function.

###### Definition F.1 (Eluder Dimension ( Russo and Van Roy, 2013 ) ) .

An action a ∈ 𝒜 a\in\mathcal{A} is ϵ \epsilon -dependent on actions { a 1 , … , a n } ⊂ 𝒜 \{a_{1},\dots,a_{n}\}\subset\mathcal{A} with respect to a reward class ℛ \mathcal{R} if any pair of reward functions r , r ′ ∈ ℛ r,r^{\prime}\in\mathcal{R} satisfying ∑ i = 1 n ( r ⁡ ( a i ) − r ′ ​ ( a i ) ) 2 ≤ ϵ 2 \sum_{i=1}^{n}(r(a_{i})-r^{\prime}(a_{i}))^{2}\leq\epsilon^{2} also satisfies | r ⁡ ( a ) − r ′ ​ ( a ) | ≤ ϵ |r(a)-r^{\prime}(a)|\leq\epsilon . The ϵ \epsilon -eluder dimension, denoted dim E ( ℛ , ϵ ) \dim_{E}(\mathcal{R},\epsilon) , is the length of the longest sequence in 𝒜 \mathcal{A} such that every element is ϵ ′ \epsilon^{\prime} -independent of its predecessors for some ϵ ′ ≥ ϵ \epsilon^{\prime}\geq\epsilon .

In Language Learning from Feedback (LLF) ( Cheng et al., 2023 ) , agents leverage feedback o o ( e.g., critiques) beyond scalar rewards. To quantify the information in o o , we assume the existence of a verifier .

###### Assumption F.2 (Verifier ( Xu et al., 2025 ) ) .

The verifier defines a loss ℓ : 𝒜 × 𝒪 × ℋ → [ 0 , 1 ] \ell:\mathcal{A}\times\mathcal{O}\times\mathcal{H}\to[0,1] measuring the alignment between a hypothesis η \eta and feedback o o on action a a . Consistency yields ℓ ⁡ ( a , o , η ) = 0 \ell(a,o,\eta)=0 , while inconsistency incurs a non-zero penalty.

To accommodate potential noise, we assume feedback is unbiased: each hypothesis minimizes the expected verifier loss under its induced distribution. Defining the expected minimum loss as ℓ η min ​ ( a ) := min η ′ ⁡ 𝔼 o ∼ f η ​ ( a ) ​ [ ℓ ⁡ ( a , o , η ′ ) ] \ell^{\min}_{\eta}(a):=\min_{\eta^{\prime}}\mathbb{E}_{o\sim f_{\eta}(a)}[\ell(a,o,\eta^{\prime})] , we utilize this to define the Transfer Eluder Dimension :

###### Definition F.3 (Transfer Eluder Dimension ( Xu et al., 2025 ) ) .

An action a a is ϵ \epsilon - transfer dependent on actions { a 1 , … , a n } ⊂ 𝒜 \{a_{1},\dots,a_{n}\}\subset\mathcal{A} with respect to ℋ \mathcal{H} and verifier ℓ \ell if any pair η , η ′ ∈ ℋ \eta,\eta^{\prime}\in\mathcal{H} satisfying: ∑ i = 1 n ( 𝔼 o ∼ f η ′ ​ ( a i ) ​ [ ℓ ⁡ ( a i , o , η ) ] − ℓ η ′ min ​ ( a i ) ) ≤ ϵ 2 \sum_{i=1}^{n}\left(\mathbb{E}_{o\sim f_{\eta^{\prime}}(a_{i})}[\ell(a_{i},o,\eta)]-\ell^{\min}_{\eta^{\prime}}(a_{i})\right)\leq\epsilon^{2} (7) also satisfies | r η ​ ( a ) − r η ′ ​ ( a ) | ≤ ϵ |r_{\eta}(a)-r_{\eta^{\prime}}(a)|\leq\epsilon . The ϵ \epsilon -transfer eluder dimension, dim T ​ E ( ℋ , ℓ , ϵ ) \dim_{TE}(\mathcal{H},\ell,\epsilon) , is the length of the longest sequence of transfer-independent actions.

##### Critique as Reward-Informative Feedback.

Intuitively, the Transfer Eluder Dimension measures how effectively feedback reduces uncertainty about the reward. For this framework to apply, we must establish that our critique mechanism is at least as useful as the reward signal. We rely on the concept of Reward-Informative Feedback .

###### Definition F.4 (Reward-Informative Feedback) .

A feedback function f η f_{\eta} is reward-informative of r η r_{\eta} with respect to a verifier ℓ \ell if there exists a constant C F > 0 C_{F}>0 such that for all η ′ ∈ ℋ \eta^{\prime}\in\mathcal{H} and a ∈ 𝒜 a\in\mathcal{A} : | r η ​ ( a ) − r η ′ ​ ( a ) | 2 ≤ C F ​ 𝔼 o ∼ f η ​ ( a ) ​ [ ℓ ⁡ ( a , o , η ′ ) − ℓ η min ​ ( a ) ] . |r_{\eta}(a)-r_{\eta^{\prime}}(a)|^{2}\leq C_{F}\mathbb{E}_{o\sim f_{\eta}(a)}[\ell(a,o,\eta^{\prime})-\ell^{\min}_{\eta}(a)]. (8)

This condition implies that if two hypotheses differ in reward, the verifier can distinguish them via feedback. Our employed critiques (both indicative and constructive) satisfy this condition by providing strictly more information than a binary reward r ∈ { 0 , 1 } r\in\{0,1\} , enabling the rejection of incorrect hypotheses without needing to observe the final scalar reward.

###### Proposition F.5 (Generalization Bound) .