Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback — part 7/7

Cleaned-text code points [93861, 104060). Bibliography omitted here; source.md includes it.

Higher entropy does not always guarantee effective exploration. Unexpectedly, as shown in Figure 10(b) , Critique-GRPO (weaker refinement via critique with ground truth), shown in green, achieves higher entropy than Critique-GRPO (self-critique), shown in dark blue, yet performs worse (average pass@1: 65.55% v.s. 68.13%). This discrepancy may be due to refinements from weaker models causing larger distributional shifts compared to self-refinements, while also being of lower quality. This suggests that the quality of exploration signals is more critical than the extent of exploration (as reflected solely by entropy).

Critique-GRPO facilitates concise reasoning. In Figure 11 , Critique-GRPO achieves superior performance (Table 2 ) while minimally increasing response length on Qwen2.5-7B-Base (Figure 11(a) ). This efficiency likely stems from its critique mechanism, which enables precise error identification and refinement, reducing the need for verbose reasoning. Additionally, Critique-GRPO tends to reduce response length on Qwen3-8B (Figure 11(b) ). This trend can be attributed to the correction of Qwen3-8B’s tendency toward redundant and ineffective self-reflection, as discussed in Appendix J .

## Appendix J Qualitative Analysis

Fine-Grained Analysis. We conduct a fine-grained analysis of 100 generated responses on the Minerva-MATH dataset across four key dimensions: factuality, conciseness, correctness, and logicality, using the prompt in Appendix M . Figure 12 shows that fine-tuning with Critique-GRPO on Qwen2.5-7B-Base achieves the best performance across all four dimensions as well as in average performance. Additionally, the superior performance of RL fine-tuning with Critique-GRPO over R1-GRPO in terms of factuality and correctness indicates that CoT critiques help the model effectively identify errors and improve valid exploration. The inferior performance of SFT in logicality and conciseness may be attributed to the presence of redundant and sometimes illogical self-reflective reasoning behaviors in expert demonstrations, which could negatively impact user experience.

Case Study. We present a qualitative comparison between the responses generated by the base Qwen3-8B model and the RL-finetuned Qwen3-8B model using Critique-GRPO on the MATH-500 dataset. The given question is: Evaluate sin ⁡ ( arcsin ⁡ 0.4 + arcsin ⁡ 0.5 ) , sin ⁡ ( arcsin ⁡ 0.5 − arcsin ⁡ 0.4 ) \sin(\arcsin 0.4+\arcsin 0.5),\sin(\arcsin 0.5-\arcsin 0.4) .

The base Qwen3-8B model produces an incorrect response due to an incorrect formulation of the expression as sin ⁡ ( A + B ) ⋅ sin ⁡ ( A − B ) \sin(A+B)\cdot{\color[rgb]{1,0,0}\sin(A-B)} , as shown in the red square. In contrast, the RL-finetuned Qwen3-8B model using Critique-GRPO generates a correct response by correctly formulating the expression as sin ⁡ ( α + β ) ⋅ sin ⁡ ( β − α ) \sin(\alpha+\beta)\cdot{\color[rgb]{0,1,0}\sin(\beta-\alpha)} , as shown in the green square. Furthermore, the base Qwen3-8B model exhibits numerous redundant and ineffective self-reflection attempts ( e.g., “Wait, …” highlighted in blue), which fail to help the model identify the actual errors and answer the question correctly. This results in an excessively long response (over 6000 tokens). In contrast, the RL-finetuned Qwen3-8B model using Critique-GRPO demonstrates concise and effective reasoning. It remains on the correct path to solve the problem, exhibits valid self-reflection to validate the answer ( e.g., “Let me check with approximate values. …”), and ultimately generates the correct answer. Detailed responses are provided in Appendix O .

## Appendix K Limitations

While Critique-GRPO establishes a promising foundation for leveraging both natural language and numerical feedback, notable limitations remain.

Performance limitations due to failed refinements. Policy models sometimes fail to follow CoT critiques to refine their responses. We attribute this to the lack of deliberate training for self-refinement. An example of a failed refinement is provided in Appendix P . Future work could focus on improving the model’s refinement capabilities or training a specialized model dedicated to refinement tasks.

The role of critique detail in refinement quality. We currently utilize three types of critiques (see Section 3 ), with CoT critiques demonstrating the greatest benefits for refinement. This advantage likely stems from their detailed step-by-step evaluations and concise improvement suggestions, which help models identify and correct errors in initial responses. It follows that more detailed critiques could result in higher-quality refinements. For simplicity, we use GPT-4o as the reasoning-based reward model, not for expert knowledge distillation . Consequently, the generated CoT critiques do not include expert demonstrations. Future work may explore alternative reasoning-based reward models. One might assume that directly incorporating expert demonstrations into critiques would significantly improve performance. However, our experiments reveal otherwise. Upon analyzing the generated refinements, we observe that both pre-trained models ( e.g., Qwen2.5-7B-Base) and alignment-tuned models ( e.g., Qwen3-8B) tend to produce conclusive sentences and correct answers as refinements, rather than detailed step-by-step reasoning to derive the correct answer. This behavior limits the effectiveness of expert demonstrations as critiques.

Future work could investigate, in greater depth, which types of critiques provide the most significant benefits for refinement, particularly in reasoning-intensive tasks.

## Appendix L Analysis of Cognitive Behaviors

To systematically investigate this question, we characterize six key cognitive behaviors that contribute to self-improving reasoning during RL fine-tuning, as follows: • Subgoal Setting : Breaking down complex problems into smaller, manageable steps or subtasks. For example, “Step 1… Step 2…”

• Summarization : Summarizing the current state by identifying completed subtasks and determining what remains to be done. This helps guide the next steps in reasoning. For example, “Now we have obtained…, next, we need to…”

• Verification : Systematically checking intermediate results or computations to ensure correctness. For example, “Let’s verify this result by…”

• Backtracking : Identifying errors or dead-ends in reasoning and explicitly revising previous methods or approaches. For example, “This approach won’t work because…, let’s try another method…”

• Backward Chaining : Reasoning from desired outcomes back to initial inputs or steps required to achieve the result. This is particularly applicable to multiple-choice questions where answer options are provided. For example, “To get 24, I could do 24 ÷ 2 = 12…” ( Gandhi et al., 2025 )

• Anticipation : Anticipating potential inaccuracies or exhaustively considering multiple possibilities to solve a problem. For example, “Alternatively, this problem can be solved by…”

We analyze the reasoning (cognitive) behaviors using the prompts shown below.

When assessing the contributions of reasoning behaviors in Section 3 to successful problem-solving in RL fine-tuned models, we count each behavior appearing in the generated responses only once . For example, if the model produces multiple subgoals in a single response, the occurrence of “subgoal setting” is counted as one.

## Appendix M Prompts

##### Training Prompt.

The following training prompt is used during all RL fine-tuning experiments:

##### Prompt for Generating Chain-of-Thought Critique.

We adopt a prompt inspired by ( Wang et al., 2025 ) to enable GPT-4o ( Hurst et al., 2024 ) to generate CoT critiques. For quality control, we retained only those model-generated critiques whose evaluative conclusions (correct/incorrect) aligned with rule-based verification. When inconsistencies occurred, we prompted the critique model to regenerate the critiques.

##### Prompt for Generating Chain-of-Thought Critique with Internal Knowledge

The following prompt is designed to enable an LLM to leverage its internal knowledge and evaluate the correctness of its own generated responses through step-by-step CoT critiques.

##### Refinement Prompt.

The following refinement prompt is used to guide the model in improving its response by incorporating the critique.

Future work could explore designing prompts ( Zhang et al., 2023 ) to enable LLMs to generate high-quality CoT critiques.

##### Prompt for Qualitative Analysis.

We employ the following prompt to conduct qualitative analysis of the generated responses using GPT-4o.

## Appendix N An Example of Successful Refinement using a CoT Critique

The example below demonstrates a successful refinement using a CoT critique. This output was generated by the best-performing RL-finetuned Qwen3-8B model using GRPO algorithm in Section 3 , utilizing the refinement prompt detailed in Appendix N .

## Appendix O Responses Utilized in Qualitative Analysis

## Appendix P An Example of Failed Refinement

The following example demonstrates a failed refinement using a CoT critique, where RL-finetuned Qwen3-8B using Critique-GRPO fails to follow the critique to refine its responses.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .