Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback

arxiv:2506.03106

Original source · Paper JSON

Post-training · 2025-06-03

Cites (8) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• Advancing LLM Reasoning with Natural Language and Numerical Feedback
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• Enhancing LLM Reasoning with RL.
• 3 Limitations of RL and the Promise of Natural Language Guidance
• 3.1 Three Limitations of Learning with Solely Numerical Feedback
• 3.2 Promise of Learning from Natural Language Feedback
• 4 Critique-GRPO
• 4.1 From GRPO to Critique-GRPO
• 4.2 Online RL with Critique-GRPO
• Proposition 4.1 (Complexity Reduction via Critique-Guided Exploration) .
• 5 Experiments
• 5.1 Experimental Setup
• 5.2 Main Results
• 5.3 Investigation on Math-Centric Backbone Models
• 5.4 Self-Improvement via Self-Critiquing
• 5.5 Exploration with Varying Models
• 5.6 Investigation of Policy Exploration
• 5.7 Fine-Grained Ablation Studies
• 5.8 Exploration of Weak-to-Strong Generalization
• 5.9 Online Joint Optimization vs. Sequential Baseline
• 6 Conclusion
• Acknowledgments
• Impact Statement
• Appendix
• Appendix A Additional Discussion on Related Work
• Learning from Natural Language Feedback.
• Enhancing LLM Reasoning with Reinforcement Learning.
• Appendix B Analysis of Training Computational Cost
• Appendix C Impact of Critique Length on Critique-GRPO
• Appendix D Scalability of Critique-GRPO to Large-Scale Models
• Appendix E Limitations of RL with Numerical Feedback and the Promise of Natural Language Guidance (Detailed Analyses)
• E.1 Three Limitations of Learning with Numerical Feedback
• Setup.
• Results.
• E.2 Promise of Learning from Natural Language Feedback
• Setup.
• E.3 Leveraging Textual Critiques for Refining LLM Responses
• E.4 Investigation on Qwen3-8B-Base
• Appendix F More Details about Critique-GRPO
• F.1 The Critique-GRPO Algorithm
• F.2 The shaping Function in Critique-GRPO
• F.3 Ratio of Responses per Prompt in Critique-GRPO
• F.4 Theoretical Analysis of Sample Efficiency
• Preliminaries: Complexity Measures and Verifiers.
• Definition F.1 (Eluder Dimension ( Russo and Van Roy, 2013 ) ) .
• Assumption F.2 (Verifier ( Xu et al., 2025 ) ) .
• Definition F.3 (Transfer Eluder Dimension ( Xu et al., 2025 ) ) .
• Critique as Reward-Informative Feedback.
• Definition F.4 (Reward-Informative Feedback) .
• Proposition F.5 (Generalization Bound) .
• Complexity Reduction Analysis.
• Proposition F.6 (Sample Efficiency of Critique-Guided Refinement) .
• Proof.
• F.5 Theoretical Analysis of Convergence and Policy Improvement
• Preliminaries and Problem Setup.
• Definition F.7 (Sparse Reward Landscape) .
• Definition F.8 (GRPO Gradient Estimate) .
• F.5.1 Inefficiency of Standard Exploration
• Proposition F.9 (Signal-to-Noise Ratio in Standard GRPO) .
• F.5.2 Policy Improvement via Critique-Guided Refinement
• Lemma F.10 (Trajectory Quality Enhancement) .
• Theorem F.11 (Policy Improvement Lower Bound) .
• Proof.
• Theorem F.12 (Global Convergence via Policy Shaping) .
• Proof.
• F.6 Theoretical Analysis: Gradient Efficiency and Manifold Preservation
• F.6.1 Gradient Dynamics of the Shaping Function
• Lemma F.13 (Gradient Rescaling) .
• Proof.
• Analysis of the Modulation Term Ψ ⁡ ( π t ) \Psi(\pi_{t}) .
• F.6.2 The Insufficiency of Clipping for Off-Policy Data
• Proposition F.14 (Gradient Orthogonality) .
• F.6.3 Manifold Preservation via The Anchor Hypothesis
• Theorem F.15 (The Anchor Condition) .
• Proof.
• Appendix G Implementation Details
• Appendix H Detailed Results and Analysis of Self-Critiquing Mechanisms
• Appendix I Detailed Investigation of Policy Exploration During RL Finetuning
• Appendix J Qualitative Analysis
• Appendix K Limitations
• Appendix L Analysis of Cognitive Behaviors
• Appendix M Prompts
• Training Prompt.
• Prompt for Generating Chain-of-Thought Critique.
• Prompt for Generating Chain-of-Thought Critique with Internal Knowledge
• Refinement Prompt.
• Prompt for Qualitative Analysis.
• Appendix N An Example of Successful Refinement using a CoT Critique
• Appendix O Responses Utilized in Qualitative Analysis
• Appendix P An Example of Failed Refinement
• Instructions for reporting errors