Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 1/7

Cleaned-text code points [0, 15464). Bibliography omitted here; source.md includes it.

# Advancing LLM Reasoning with Natural Language and Numerical Feedback

###### Abstract

Recent advances in reinforcement learning (RL) using numerical rewards have significantly enhanced the complex reasoning capabilities of large language models (LLMs). However, we identify three fundamental limitations of purely numerical feedback: performance plateaus, ineffective spontaneous self-reflection, and persistent failures. We show that plateaued RL models can successfully refine failed solutions when given natural language critiques. Motivated by this, we propose Critique-GRPO, an online RL framework that integrates both natural language and numerical feedback for policy optimization. This approach enables LLMs to learn simultaneously from initial responses and critique-guided refinements, effectively internalizing the exploration benefits of both stages. Extensive experiments show that Critique-GRPO outperforms all compared supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of approximately +15.0-21.6% on various Qwen models and +7.3% on Llama-3.2-3B-Instruct across eight challenging reasoning tasks. Notably, Critique-GRPO facilitates effective self-improvement through self-critiquing, achieving substantial gains over GRPO, e.g., a +16.7% Pass@1 improvement on AIME 2024. The code and models are released at: https://github.com/zhangxy-2019/critique-GRPO

###### Keywords:

Xiaoying Zhang ∗,†,‡,1,4 Chao Yang ‡,4 Helen Meng ‡,1

## 1 Introduction

RL has been a key driver of recent advancements in improving the reasoning capabilities of LLMs ( Yang et al., 2025a ; DeepSeek-AI et al., 2025 ; OpenAI et al., 2024 ; OpenAI, 2025 ) . Specifically, online RL with numerical feedback ( e.g., scalar rewards)—exemplified by the R1-Zero paradigm ( DeepSeek-AI et al., 2025 ) —enables LLMs to learn through trial-and-error ( Silver and Sutton, 2025 ) , yielding substantial improvements.

Despite its success, we identify three key limitations of RL with solely numerical feedback (Section 3 ): ( i ) (\textup{i}) Performance Plateaus: Performance stagnates despite 8 × 8\times training data scaling or extended training. ( ii ) (\textup{ii}) Ineffective Spontaneous Self-Reflection: Spontaneous “Aha moments” ( DeepSeek-AI et al., 2025 ) rarely improve problem-solving success. ( iii ) (\textup{iii}) Persistent Failures: Models consistently fail on specific training problems despite extensive iterations. While dense intermediate rewards ( Cui et al., 2025a ; Feng et al., 2025 ) can partially mitigate plateaus and persistent failures through fine-grained credit assignment ( Sutton, 1984 ) , they remain insufficient for rectifying ineffective self-reflection. Crucially, numerical feedback inherently lacks the expressivity to explain why a response fails or how to correct it. These limitations highlight the need for richer feedback mechanisms to enable effective RL scaling ( Karpathy, 2025 ) .

Natural Language Feedback (NLF) ( e.g., textual critiques) offers a promising solution by providing explicit guidance ( Saunders et al., 2022 ) . However, existing methods primarily utilize supervised fine-tuning (SFT) to imitate static, pre-collected critiques or critique-guided refinements ( Wang et al., 2025 ; Chen et al., 2024 ; Xi et al., 2024 ) . While effective for narrow policy alignment, these offline approaches lack the capacity for active exploration and real-time adaptation, which are essential for robust generalization. Crucially, the integration of expressive critiques into online RL loops—to augment purely numerical feedback—remains unexplored. This work addresses that gap by asking: Can we incorporate critiques into an online RL framework to enable LLMs to learn simultaneously from both natural language and numerical feedback?

To investigate this, we first show that critiques enable even plateaued RL models to correct persistent failures (Section 3 ). This efficacy extends from simple indicators to detailed chain-of-thought (CoT) evaluations ( Wang et al., 2025 ) . This suggests that verbal credit assignment from critiques enables models to access high-quality refinement trajectories via in-context learning ( Liu et al., 2022 ) —trajectories unreachable via standard exploration with scalar rewards alone. This aligns with theoretical work showing that granular diagnostic signals in informative language feedback can exponentially accelerate learning by reducing search-space complexity ( Xu et al., 2025 ) . Building on this insight, we propose Critique-GRPO, a novel online RL framework that synergizes numerical and natural language feedback. As depicted in Figure 1 , Critique-GRPO employs a dual mechanism: ( i ) (\textup{i}) learning from standard generation , where the model generates initial responses via standard exploration; and ( ii ) (\textup{ii}) learning from critique-guided refinement , where the model performs self-refinement via in-context learning based on critiques from model- or rule-based reward systems. This integrates targeted natural language feedback through high-quality refinements while preserving standard exploration. To mitigate entropy explosion and performance degradation arising from significant distribution shifts induced by refinements, we prioritize initial responses and incorporate refinements selectively. Additionally, a shaping function ( Yan et al., 2025 ) reinforces successful, yet unfamiliar refinements while penalizing incorrect attempts.

We evaluate Critique-GRPO using five different models across eight challenging reasoning tasks. Extensive results demonstrate that Critique-GRPO significantly outperforms supervised and RL-based fine-tuning methods, achieving average Pass@1 improvements of +15.0-21.6% on Qwen models ( Qwen et al., 2025 ; Yang et al., 2025a ) and +7.3% on Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 ) . Notably, the framework is robust to diverse critique sources and exhibits strong self-improvement capabilities; for instance, employing self-critiques yields a +16.7% Pass@1 gain over GRPO on AIME 2024.

In summary, our contributions are three-fold: ( i ) (\textup{i}) We identify three key limitations of RL relying solely on numerical feedback and highlight the potential of natural language feedback to address them. ( ii ) (\textup{ii}) We propose Critique-GRPO, an online RL framework that enables LLMs to learn from both initial responses and their refinements by integrating numerical and natural language feedback. ( iii ) (\textup{iii}) We validate the efficacy of Critique-GRPO through extensive experiments, demonstrating superior performance across eight challenging reasoning benchmarks.

## 2 Related Work

##### Enhancing LLM Reasoning with RL.

Reinforcement learning with numerical feedback has proven highly effective for improving the reasoning capabilities of LLMs ( OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ; Fatemi et al., 2025 ; Li et al., 2025 ; Liu et al., 2025a ; Yu et al., 2025 ) . However, numerical feedback is inherently limited in expressivity; sparse scalar rewards provide little diagnostic information about where or why errors occur, making it difficult for models to identify and correct specific reasoning failures ( Xi et al., 2024 ; Gandhi et al., 2025 ) . To bridge this gap, recent work has combined online RL with expert demonstrations ( Yan et al., 2025 ; Lv et al., 2025 ; Lanchantin et al., 2025 ) ; however, these approaches are constrained by their reliance on curated, high-quality data. In contrast, Critique-GRPO integrates expressive natural language critiques directly into online RL loops, eliminating the dependency on expert demonstrations. Further discussions on learning from NLF are provided in Appendix A .

## 3 Limitations of RL and the Promise of Natural Language Guidance

### 3.1 Three Limitations of Learning with Solely Numerical Feedback

In this section, we identify three key limitations of RL fine-tuning that relies solely on numerical feedback: ( i ) (\textup{i}) performance plateaus , ( ii ) (\textup{ii}) ineffective spontaneous self-reflection , and ( iii ) (\textup{iii}) persistent failures . We investigate these limitations using mathematical and STEM reasoning tasks on three models: Qwen2.5-7B-Base ( Qwen et al., 2025 ) (non-reasoning), Qwen3-8B ( Yang et al., 2025a ) (reasoning), and Qwen3-8B-Base. We summarize key findings for Qwen2.5-7B-Base below; complete analyses appear in Appendices E .

Performance plateaus despite scaling data and compute. As shown in Figure 5(a) (Appendix E ), Qwen2.5-7B-Base performance saturates after 120 steps; neither an 8-fold increase in training prompts (4k to 32k) nor extended training yields significant gains.

Spontaneous self-reflection contributes minimally to problem-solving. We analyze six cognitive behaviors emerging during RL fine-tuning: ( i ) (\textup{i}) planning behaviors (subgoal setting, summarization) and ( ii ) (\textup{ii}) self-reflection behaviors (verification, backtracking, backward chaining ( Gandhi et al., 2025 ) , anticipation). For problems that only the RL-finetuned model solves correctly, we measure each behavior’s contribution to success. Figure 6(a) (Appendix E ) shows self-reflection behaviors contribute minimally across both reasoning tasks.

Persistent failures on substantial training subsets. As shown in Table 1 (left panel), even the best-performing RL-finetuned Qwen2.5-7B-Base consistently fails on approximately 29% of training questions (Pass@4 = 0), indicating fundamental limitations of numerical-only feedback.

### 3.2 Promise of Learning from Natural Language Feedback

To investigate whether natural language feedback can overcome the identified scaling bottlenecks, we evaluate three critique variants: ( i ) (\textup{i}) Indicative Critique , which provides only a binary correctness signal; ( ii ) (\textup{ii}) Indicative Critique with Ground Truth (w/ GT) , which includes the final answer but excludes expert demonstrations ; ( iii ) (\textup{iii}) CoT Critique , a model-generated step-by-step evaluation that concludes with a correctness label, also without expert demonstrations . We apply these critiques to the subset of training questions where RL-finetuned models persistently fail (Pass@4 = 0) and prompt them to refine their initial responses. Our key findings, summarized below, demonstrate that even minimal language feedback can catalyze error correction where scalar rewards failed. Detailed analyses, refinement strategies, and examples are available in Appendix E .

Deliberate critiques outperform spontaneous self-reflection. All three critique types enable successful refinement of previously unsolvable problems, demonstrating that deliberate critiques succeed where spontaneous self-reflection fails.

CoT critiques enable substantial improvement. Table 1 shows that CoT critiques achieve the highest valid refinement rate (36.47%) and successfully refine 55.37% of persistently failed questions for Qwen2.5-7B-Base. The effectiveness of CoT critiques can be attributed to their richness, e.g., providing a step-by-step explanation.

## 4 Critique-GRPO

In this section, we introduce Critique-GRPO, an online RL framework built upon Group Relative Policy Optimization (GRPO) ( Shao et al., 2024a ) that enables LLMs to learn from both natural language and numerical feedback.

### 4.1 From GRPO to Critique-GRPO

GRPO is an online RL algorithm designed for efficient LLM fine-tuning. Unlike Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , it eliminates the need for a value function approximation by estimating advantages based on the relative performance of a group of sampled responses for the same query. For an LLM policy π θ \pi_{\theta} , GRPO operates as follows: For each query in the dataset q ∈ Q q\in Q , n n responses { y ( i ) } i = 1 n \{y^{(i)}\}_{i=1}^{n} are sampled from the old policy π old \pi_{\text{old}} and scored to obtain rewards { R ( i ) } i = 1 n \{R^{(i)}\}_{i=1}^{n} . The training objective is: 𝒥 GRPO ( θ ) = 𝔼 q ∼ Q , { y ( i ) } i = 1 n ∼ π old ( ⋅ ∣ q ) [ 1 n ∑ i = 1 n 1 | y ( i ) | ∑ t = 1 | y ( i ) | \displaystyle\mathcal{J}_{\text{GRPO}}(\theta)=\mathbb{E}_{q\sim Q,\{y^{(i)}\}_{i=1}^{n}\sim\pi_{\text{old}}(\cdot\mid q)}\bigg[\frac{1}{n}\sum_{i=1}^{n}\frac{1}{|y^{(i)}|}\sum_{t=1}^{|y^{(i)}|} (1) min ( r t ( i ) ( θ ) A ^ t ( i ) , clip ( r t ( i ) ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ( i ) ) ] \displaystyle\min\big(r_{t}^{(i)}(\theta)\hat{A}_{t}^{(i)},\text{clip}(r_{t}^{(i)}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}^{(i)}\big)\bigg] where r t ( i ) ​ ( θ ) = π θ ​ ( y t ( i ) ∣ q , y < t ( i ) ) π old ​ ( y t ( i ) ∣ q , y < t ( i ) ) r_{t}^{(i)}(\theta)=\frac{\pi_{\theta}(y_{t}^{(i)}\mid q,y_{<t}^{(i)})}{\pi_{\text{old}}(y_{t}^{(i)}\mid q,y_{<t}^{(i)})} is the probability ratio, comparing the current policy π θ \pi_{\theta} to the old policy π old \pi_{\text{old}} . The advantage A ^ t ( i ) = R ( i ) − mean ​ ( { R ( 1 ) , … , R ( n ) } ) std ​ ( { R ( 1 ) , … , R ( n ) } ) \hat{A}_{t}^{(i)}=\frac{R^{(i)}-\text{mean}(\{R^{(1)},\dots,R^{(n)}\})}{\text{std}(\{R^{(1)},\dots,R^{(n)}\})} for all tokens in a response is calculated by normalizing the rewards { R ( i ) } i = 1 n \{R^{(i)}\}_{i=1}^{n} using the group mean and standard deviation. The hyperparameter ϵ \epsilon sets the clipping range for the probability ratio, preventing overly large policy updates. For simplicity, we omit the KL divergence penalty in Equation 1 . Following Liu et al. (2025a) , we exclude the length-normalization factor 1 / | y ( i ) | 1/|y^{(i)}| and the reward standard deviation std ​ ( { R ( i ) } ) \text{std}(\{R^{(i)}\}) to avoid biased gradients.

### 4.2 Online RL with Critique-GRPO

We introduce Critique-GRPO, which optimizes π θ \pi_{\theta} using trajectories from both standard generation and critique-guided refinement. As illustrated in Figure 2 , the process consists of three steps:

Step 1: Initial Response Sampling. For each query q ∈ Q q\in Q , we sample n n initial responses { y ( i ) } i = 1 n \{y^{(i)}\}_{i=1}^{n} from the old policy π old ( ⋅ ∣ q ) \pi_{\text{old}}(\cdot\mid q) . A reward system evaluates these responses, generating critiques { c ( i ) } i = 1 n \{c^{(i)}\}_{i=1}^{n} and scalar rewards { R ( i ) } i = 1 n \{R^{(i)}\}_{i=1}^{n} , with positive rewards for correct responses and zero reward for incorrect ones: c ( i ) , R ( i ) ← Reward ​ ( q , y ( i ) ) , ∀ i c^{(i)},R^{(i)}\leftarrow\text{Reward}(q,y^{(i)}),\forall i . Critique-GRPO supports two types of reward systems: ( i ) (\textup{i}) Rule-based: Rewards are binary based on string matching with the ground truth: R ( i ) = is_equivalent ​ ( y ( i ) , y GT ) R^{(i)}=\text{is\_equivalent}(y^{(i)},y_{\text{GT}}) . Based on these evaluations, two indicative critiques are heuristically constructed: one without ground truth ( c I ( i ) c_{\text{I}}^{(i)} ) and one with ground truth ( c GT ( i ) c_{\text{GT}}^{(i)} ): c I ( i ) , c GT ( i ) ← R ( i ) c_{\text{I}}^{(i)},c_{\text{GT}}^{(i)}\leftarrow R^{(i)} . ( ii ) (\textup{ii}) Model-based: A reward model π R ​ M \pi_{RM} generates CoT critiques: c CoT ( i ) ∼ π R ​ M ( ⋅ ∣ I c , q , y ( i ) ) c_{\text{CoT}}^{(i)}\sim\pi_{RM}(\cdot\mid I_{c},q,y^{(i)}) , where I c I_{c} is the critique instruction. The binary correctness of the critique determines the scalar reward: R ( i ) ← c CoT ( i ) R^{(i)}\leftarrow c_{\text{CoT}}^{(i)} (detailed in Appendix E ).