Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization

arxiv:2508.07629

Original source · Paper JSON

Post-training · 2025-08-11

Cites (3) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization
• Abstract
• 1 Introduction
• 2 Preliminary
• 2.1 Proximal Policy Optimization (PPO)
• 2.2 Group Relative Policy Optimization (GRPO)
• 2.3 Raise the Ceiling: Clip-Higher
• 2.4 Token-Level Policy Gradient Loss
• 3 Method
• 3.1 long Chain-of-Thought Supervised Fine-tuning
• 3.2 Reinforcement Learning
• 3.2.1 Data curation
• 3.2.2 Gradient-Preserving Clipping Policy Optimization
• 3.2.3 RL Training with SFT Loss
• 3.2.4 Reward Design for Math and Code RL
• 4 Evaluation
• 4.1 Training Details
• 4.2 Benchmarks and Baselines
• 4.3 Evaluation Results
• 5 Analysis
• 5.1 SFT Analyses
• 5.1.1 Impact of data correctness
• 5.1.2 Ablation of teacher models
• 5.1.3 Data quality and diversity balance
• 5.2 RL Analyses
• 5.2.1 Effectiveness of Gradient-Preserving Clipping Policy Optimization
• 5.2.2 Impact of SFT loss
• 5.2.3 Soft Reward vs. Hard Reward
• 5.2.4 Improving Code RL via Test Case Filtering
• 5.2.5 Impact of Zero-Advantage Sample Filtering
• 6 Conclusions
• Appendix A Appendix
• A.1 CISPO Gradient Analysis
• A.2 The General Form of GPPO
• A.3 License for Scientific Artifacts
• Instructions for reporting errors