Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 3/7

Cleaned-text code points [31323, 46860). Bibliography omitted here; source.md includes it.

Critique-GRPO enables effective weak-to-strong generalization. As shown in Table 7 , Critique-GRPO (weaker refinement via critique with ground truth) achieves a +12.3% average pass@1 improvement over Qwen3-8B and outperforms R1-GRPO (65.55% v.s. 63.75%). This demonstrates that refinements from a weaker model can significantly enhance the performance of a stronger model.

### 5.9 Online Joint Optimization vs. Sequential Baseline

To isolate the contribution of our online joint optimization, we compare Critique-GRPO against a sequential baseline, R1-GRPO + Refinement-SFT (CoT Critique) , executed in two decoupled stages: (1) running standard GRPO to convergence, and (2) critiquing/refining remaining failures followed by SFT on filtered refinements.

As shown in Table 8 , Critique-GRPO significantly outperforms this baseline across both ID math and OOD science & general benchmarks. For instance, Critique-GRPO advances the MATH-500 accuracy to 77.80% and markedly surges to 62.50% on AMC23. On heavy OOD reasoning tasks like GPQA-Diamond, the sequential baseline yields marginal gains ( 34.34 % 34.34\% v.s. 33.33 % 33.33\% ), whereas Critique-GRPO promotes it to 37.88%. These results strongly support the integration of online joint optimization via dual numeric and language feedback.

## 6 Conclusion

We identified three key challenges in RL approaches using only numerical feedback and proposed Critique-GRPO, an online RL framework that learns from both natural language and numerical feedback. Specifically, it enables learning from initial responses and critique-guided self-refinements simultaneously. Experiments across eight reasoning tasks demonstrate that Critique-GRPO consistently outperforms existing SFT and numerical-feedback-only RL methods, paving the way for scaling RL with diverse real-time feedback in real-world post-deployment scenarios.

## Acknowledgments

We sincerely thank Zichen Liu, Zhanhui Zhou, and the anonymous reviewers for their valuable feedback.

## Impact Statement

Throughout our research, we have adhered to the ICML Code of Ethics, prioritizing privacy, fairness, and individual well-being. All benchmark datasets used were strictly for research purposes and contained no personally identifiable information to safeguard privacy. Prompts were carefully designed to minimize bias or discriminatory language and reduce potential negative impacts. Additionally, modelgenerated responses were verified to ensure they were free from offensive content, misinformation, and personally identifiable information.

## Appendix

## Appendix A Additional Discussion on Related Work

##### Learning from Natural Language Feedback.

Natural Language Feedback (NLF) provides textual critiques with granular diagnostic signals that facilitate precise response refinement ( Saunders et al., 2022 ; Chen et al., 2024 ; Zhang et al., 2026 ) . Current literature typically integrates NLF by: (1) translating linguistic feedback into scalar reward signals for RL fine-tuning ( Kim et al., 2023 ; Whitehouse et al., 2025 ; Liu et al., 2025b ; Lightman et al., 2024 ; Ouyang et al., 2022 ; Casper et al., 2023 ; Rafailov et al., 2024 ) ; or (2) utilizing SFT to imitate static critiques or critique-guided refinements ( Wang et al., 2025 ; Xi et al., 2024 ; Chen et al., 2024 ) . However, these approaches are primarily offline and lack the capacity for active exploration, limiting their ability to adapt dynamically to idiosyncratic errors encountered during training. While emerging research in natural language RL ( Feng et al., 2024 ; Hong et al., 2025 ; Luo et al., 2025 ) has begun to explore linguistic feedback, it remains predominantly confined to offline settings. In contrast, Critique-GRPO integrates real-time NLF directly into the online RL loop. By enabling the model to learn from guided refinements during active exploration, our approach internalizes error-correction mechanisms more effectively, overcoming the performance plateaus inherent in systems relying solely on numerical rewards.

##### Enhancing LLM Reasoning with Reinforcement Learning.

Reinforcement Learning (RL) has demonstrated significant effectiveness in improving the reasoning capabilities of Large Language Models (LLMs) ( OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ; Fatemi et al., 2025 ; Li et al., 2025 ; Feng et al., 2026 ) . This is commonly achieved by fine-tuning models on complex reasoning tasks to encourage diverse and robust reasoning behaviors ( Gandhi et al., 2025 ; Yue et al., 2025 ) . Recent advancements leverage RL with numerical feedback, such as assigning positive rewards ( e.g., +1) for correct responses and negative rewards ( e.g., -1) for incorrect ones ( OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ; Liu et al., 2025a ; Yu et al., 2025 ) . These methods often employ online policy optimization algorithms, including Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) , Group Relative Policy Optimization (GRPO) ( Shao et al., 2024a ) , REINFORCE ( Williams, 1992 ) , and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) ( Yu et al., 2025 ) . However, numerical feedback tends to be sparse, and models often struggle with tasks that exceed their current knowledge boundaries, limiting the potential for substantial improvement ( Xi et al., 2024 ; Gandhi et al., 2025 ) . To address this limitation, recent approaches have incorporated high-quality expert demonstrations alongside online exploration ( Yan et al., 2025 ) . In contrast, our approach enables models to refine their outputs through the integration of various textual feedback, which helps identify and address potential errors. This combination of textual feedback with online exploration for policy optimization enhances flexibility and scalability.

## Appendix B Analysis of Training Computational Cost

We utilize asynchronous rollouts to minimize the additional computational cost introduced by incorporating critiques and self-refinement in Critique-GRPO. Table 9 presents the training time required for 300 steps on a single NVIDIA A800 GPU.

The results show that the additional cost is limited to only 5 and 19 GPU-hours for Qwen2.5-7B-Base and Qwen3-8B, respectively, representing approximately a 2% and 6.5% increase over GRPO. This overhead is marginal compared to the significant performance improvement, with average gains of +6 and +4.5 percentage points across 8 tasks, achieved using only 4K prompts. These findings highlight the substantial gains in data efficiency and model performance, demonstrating that the additional computational cost is highly justified.

## Appendix C Impact of Critique Length on Critique-GRPO

We investigate the impact of critique token length on the performance of Critique-GRPO by varying the level of detail in generated critiques. As shown in Table 10 , shorter CoT critiques (approximately 683 tokens) generally achieve strong performance. However, increasing the token length to around 2079 tokens by prompting the model to produce highly detailed critiques results in a slight performance drop (from 47.08 to 46.00). This decline is likely due to the increased difficulty of extracting critical information from longer contexts. Despite this, all results using Critique-GRPO consistently surpass those of GRPO, demonstrating the robustness of Critique-GRPO.

In addition, we evaluate the system against adversarial critiques to verify that improvements stem from valid guidance rather than spurious natural language signals. For example, by inverting feedback conclusions ( e.g., swapping “correct” and “incorrect”), we observe entropy explosion and subsequent performance degradation.

## Appendix D Scalability of Critique-GRPO to Large-Scale Models

We assess the scalability of Critique-GRPO on the large-scale Qwen3-32B model ( Yang et al., 2025a ) . As shown in Table 11 , Critique-GRPO consistently surpasses GRPO across all eight reasoning tasks.

## Appendix E Limitations of RL with Numerical Feedback and the Promise of Natural Language Guidance (Detailed Analyses)

### E.1 Three Limitations of Learning with Numerical Feedback

We investigate the limitations of fine-tuning with RL relying solely on numerical feedback from three key perspectives: ( i ) (\textup{i}) How best performance improves as the number of training examples increases. ( ii ) (\textup{ii}) How cognitive behaviors contribute to improving successful problem-solving. ( iii ) (\textup{iii}) The model’s ability to solve previously failed problems through trial-and-error.

##### Setup.

We conduct experiments on non-reasoning models, Qwen2.5-7B-Base ( Qwen et al., 2025 ) and Qwen3-8B-Base ( Yang et al., 2025a ) , and a reasoning model, Qwen3-8B ( Yang et al., 2025a ) , for mathematical reasoning tasks. Specifically, we fine-tune the models using GRPO ( Shao et al., 2024b ) with numerical feedback. 1 1 1 GRPO is used without loss of generality, as RL algorithms such as PPO and GRPO exhibit comparable performance.

Datasets and Evaluation Metrics. We utilize randomly sampled subsets of 4k, 8k, 16k, and 32k examples from a reorganized 46k subset ( Yan et al., 2025 ) of OpenR1-Math-220k ( Bakouch et al., 2025 ) . The prompts are sourced from NuminaMath 1.5 ( Li et al., 2024 ) , while the ground truth chain-of-thought (CoT) reasoning paths are generated by Deepseek-R1 ( DeepSeek-AI et al., 2025 ) . Unless otherwise specified, experiments primarily use 4k training prompts. For validation, we randomly sample 500 examples from the validation set curated by ( Yan et al., 2025 ) , which includes examples from Olympiad Bench ( He et al., 2024a ) , MATH ( Hendrycks et al., 2021 ) , Minerva-Math ( Lewkowycz et al., 2022 ) , AIME 2024 ( Li et al., 2024 ) , and AMC 2023 ( Li et al., 2024 ) . To provide a comprehensive evaluation, we assess performance on in-distribution (ID) tasks using Minerva-Math ( Lewkowycz et al., 2022 ) and on out-of-distribution (OOD) tasks using GPQA-Diamond, which covers physics, chemistry, and biology ( Rein et al., 2024 ) . For evaluation, we employ greedy decoding (temperature = 0) and report accuracy as pass@1, following prior work ( Zeng et al., 2025 ; Yang et al., 2025a ; Liu et al., 2025a ) . 2 2 2 The pass@k metric represents the percentage of problems where the model produces a correct solution within its first k k attempts.

Reward Design. We employ rule-based evaluation to provide numerical feedback (scalar rewards), using Math-Verify 3 3 3 https://github.com/huggingface/Math-Verify to validate the correctness of generated answers against ground truth during fine-tuning. Binary rewards are assigned as follows: + 1 +1 for correct final answers and 0 0 for incorrect ones. These rewards serve as a proxy for assessing the accuracy of generated responses.

Implementation Details. Our implementation leverages the VERL library ( Sheng et al., 2024 ) and samples four candidate responses per prompt during fine-tuning.

##### Results.

RL with solely numerical feedback frequently encounters performance plateaus. Figure 5 illustrates the RL fine-tuning dynamics of Qwen2.5-7B-Base and Qwen3-8B across varying numbers of training examples. On-policy RL frequently stagnates, as reflected in validation set accuracy: Qwen2.5-7B-Base reaches its highest performance at approximately 45-46% accuracy after 120 steps (Figure 5(a) ), while Qwen3-8B plateaus at 65-67% accuracy after 200 steps (Figure 5(b) ). Critically, peak accuracy shows negligible improvement despite an 8-fold increase in training prompts or extending the training duration to 420 steps. These results underscore a fundamental scaling bottleneck in scalar-reward RL, where increased compute and data fail to drive further reasoning gains.

Spontaneous self-reflection has limited impact on enhancing problem-solving success. Cognitive behaviors are widely recognized as key contributors to successful complex reasoning ( DeepSeek-AI et al., 2025 ; Gandhi et al., 2025 ) . In particular, increased self-reflection behaviors after RL fine-tuning, which mimic humans reflecting on past experiences and refining their approach to reach a solution (commonly referred to as the “Aha moment” ( DeepSeek-AI et al., 2025 ) ), have drawn significant attention. However, does spontaneous self-reflection play the most critical role in improved performance?

To address this question, we characterize six key cognitive behaviors that contribute to self-improving reasoning during RL fine-tuning:

• Subgoal Setting : Decomposing complex problems into smaller, manageable subtasks.

• Summarization : Summarizing the current state by identifying completed subtasks and determining the next steps in reasoning.

• Verification : Systematically checking intermediate results or computations to ensure correctness.

• Backtracking : Identifying errors or dead-ends in reasoning and revising previous methods or approaches.

• Backward Chaining : Reasoning from desired outcomes back to the initial inputs or steps required to achieve the result. This is particularly useful for multiple-choice questions with provided answer options ( Gandhi et al., 2025 ) .

• Anticipation : Anticipating potential inaccuracies or exploring alternative solutions to a problem.

We categorize the first two behaviors as planning behaviors and the remaining four as self-reflection behaviors . To analyze their contributions, we evaluate problems previously unsolved by the base model. For Qwen2.5-7B-Base, we identify 87 unsolved problems from the Minerva-Math dataset and 37 from the GPQA-Diamond dataset. For Qwen3-8B, we identify 33 unsolved problems from the Minerva-Math dataset and 15 from the GPQA-Diamond dataset. We then examine the cognitive behaviors exhibited in the correct responses generated by RL-finetuned models for these problems.

To detect these behaviors, we use GPT-4o ( Hurst et al., 2024 ) as an automatic judge with manually crafted prompts (Appendix M ). Each behavior b i b_{i} is scored as:

s i = { 1 if behavior ​ b i ​ is present, 0 otherwise. s_{i}=\begin{cases}1&\text{if behavior }b_{i}\text{ is present,}\\ 0&\text{otherwise.}\end{cases} (5)

The average success contribution of each behavior is computed as follows:

Contribution ​ ( b i ) = ∑ j = 1 N s i , j N , \text{Contribution}(b_{i})=\frac{\sum_{j=1}^{N}s_{i,j}}{N}, (6)

where N N is the total number of analyzed responses, and s i , j s_{i,j} indicates whether behavior b i b_{i} appears in response j j . Further details are provided in Appendix L .

Figure 6 presents the average success contribution of various behaviors, showing that planning behaviors are the primary contributors to successful problem-solving, while self-reflection behaviors contribute less in both the mathematical (Minerva-Math) and STEM (GPQA-Diamond) domains. In Figure 6(a) , self-reflection behaviors barely contribute in the mathematical domain for the non-reasoning model. Thanks to extensive training on expert demonstrations with diverse reasoning behaviors in the mathematical domain ( Yang et al., 2025a ) , the reasoning model shows that self-reflection behaviors make a noticeable contribution (Figure 6(b) ). Nevertheless, self-correction-related behaviors, such as backtracking, backward chaining, and anticipation, still contribute considerably less. These observations suggest the limited effectiveness of spontaneous self-reflection. This underscores the unreliability of relying on spontaneous self-reflection for self-correction to improve problem-solving success.