Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization — part 1/4

Cleaned-text code points [0, 15351). Bibliography omitted here; source.md includes it.

# Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization

###### Abstract

We present Klear-Reasoner, a model with long reasoning capabilities that demonstrates careful deliberation during problem solving, achieving outstanding performance across multiple benchmarks. Although there are already many excellent works related to inference models in the current community, there are still many problems with reproducing high-performance inference models due to incomplete disclosure of training details. This report provides an in-depth analysis of the reasoning model, covering the entire post-training workflow from data preparation and long Chain-of-Thought supervised fine-tuning (long CoT SFT) to reinforcement learning (RL), along with detailed ablation studies for each experimental component. For SFT data, our experiments show that a small number of high-quality data sources are more effective than a large number of diverse data sources, and that difficult samples can achieve better results without accuracy filtering. In addition, we investigate two key issues with current clipping mechanisms in RL: Clipping suppresses critical exploration signals and ignores suboptimal trajectories. To address these challenges, we propose G radient- P reserving clipping P olicy O ptimization (GPPO) that gently backpropagates gradients from clipped tokens. GPPO not only enhances the model’s exploration capacity but also improves its efficiency in learning from negative samples. Klear-Reasoner exhibits exceptional reasoning abilities in mathematics and programming, scoring 90.5% on AIME 2024, 83.2% on AIME 2025, 66.0% on LiveCodeBench V5 and 58.1% on LiveCodeBench V6 .

## 1 Introduction

The demonstrated success of OpenAI’s O1 series models ( Jaech et al., 2024 ) and DeepSeek-R1 ( DeepSeek-AI et al., 2025 ) highlights the significant potential of large-scale reinforcement learning for complex reasoning tasks. However, due to the incomplete disclosure of training details, many deep-seated issues remain regarding the reproduction of high-performance reasoning models. Although the community has produced numerous excellent reproductions of small models with 7–8B parameters, such as DeepScaleR ( Luo et al., 2025b , Luo et al., 2025a ) , Light-R1 ( Wen et al., 2025 ) , Ring-7B ( Team et al., 2025 ) , AReal-boba-RL-7B ( Fu et al., 2025 ) and AceReason-Nemotron-1.1-7B ( Liu et al., 2025 ) , their performance in mathematical and coding reasoning tasks still lags behind DeepSeek-R1-0528-Distill-8B ( DeepSeek-AI et al., 2025 ) and Qwen3-8B ( Yang et al., 2025 ) .

In this report, we conduct a detailed analytical study of Qwen3-8B-Base ( Yang et al., 2025 ) to advance its reasoning capabilities to state-of-the-art levels for models of comparable scale. We discover that for long Chain-of-Thought supervised fine-tuning (long CoT SFT), a compact set of high-quality data sources proves significantly more effective than larger, more diverse datasets, as high-quality examples ensure consistent learning of accurate reasoning patterns. Then, we find that simple SFT samples that are not filtered for correctness can easily interfere with the model, but difficult samples do not appear to harm the model’s performance even if they are not filtered. In fact, these difficult errors may even promote the model’s exploration and are beneficial to its performance.

For reinforcement learning, clipping importance sampling is a commonly used technique. The clipping mechanism primarily controls the magnitude of policy model updates to ensure training stability. Through in-depth analysis, we identify two issues with the current clipping mechanism ( Schulman et al., 2017 ) : • High-entropy token clipping. Among the tokens clipped beyond the upper threshold 1 + ϵ 1+\epsilon of importance sampling, there exist high-entropy tokens that often correspond to valuable exploratory behaviors at critical decision points. Directly clipping these tokens may lead to premature termination of exploration, adversely affecting the model’s post-convergence performance. Although DAPO ( Yu et al., 2025 ) proposes Clip-Higher to mitigate this issue by adjusting the upper importance sampling threshold to 1 + ϵ h 1+\epsilon_{h} , high-entropy tokens exceeding this threshold still face the same problem.

• Delayed convergence of negative samples. When the importance sampling ratio of suboptimal trajectories falls below 1 − ϵ 1-\epsilon , their gradients are forcibly truncated, preventing the model from updating based on these signals, which in turn slows down convergence.

To address the aforementioned two issues, we propose a G radient- P reserving clipping P olicy O ptimization (GPPO) that does not discard the gradients of any tokens. Even for truncated tokens, they are still included in the computational graph of backpropagation and participate in gradient computation. The gradients propagated back from these tokens by GPPO can be proven to be bounded and mild. This mechanism strikes a balance between maintaining training stability and preserving valuable gradient information. Our experimental results demonstrate that, compared to using Clip-Higher, GPPO achieves superior and more stable performance.

By integrating long CoT SFT with GPPO RL for math and coding tasks, we derive Klear-Reasoner-8B, a model that outperforms Qwen3-8B 1 1 1 https://huggingface.co/Qwen/Qwen3-8B and DeepSeek-R1-0528-Distill-8B 2 2 2 https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B across key reasoning benchmarks, as shown in Figure 1 . Klear-Reasoner-8B achieves 90.5% on AIME2024, 83.2% on AIME2025, 66.0% on LiveCodeBench V5, and 55.4% on LiveCodeBench V6. Comprehensive ablations validate the efficacy of each component in our training pipeline.

## 2 Preliminary

Before introducing GPPO, we first provide a brief overview of classical RL objectives and key features:

### 2.1 Proximal Policy Optimization (PPO)

PPO ( Schulman et al., 2017 ) is a policy gradient method that stabilizes training via a clipped surrogate objective, constraining policy updates to avoid excessive deviations. Its core objective combines reward maximization with policy constraints through:

ℒ PPO ​ ( θ ) = 𝔼 x ∼ 𝒟 ​ [ ∑ t = 1 T min ⁡ ( r t ​ ( θ ) ​ A ^ t , clip ​ ( r t ​ ( θ ) , 1 − ϵ , 1 + ϵ ) ​ A ^ t ) ] \displaystyle\mathcal{L}^{\text{PPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\sum_{t=1}^{T}\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\right)\right] (1)

Here, r t ​ ( θ ) = π θ ​ ( a t | s t ) π old ​ ( a t | s t ) r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\text{old}}(a_{t}|s_{t})} denotes the importance ratio, quantifying the policy change probability for action a t a_{t} in state s t s_{t} between the current policy π θ \pi_{\theta} and the previous policy π old \pi_{\text{old}} . The Generalized Advantage Estimate (GAE) A ^ t \hat{A}_{t} ( Schulman et al., 2016 ) reduces variance in advantage estimation through discounted temporal differences:

A ^ t = ∑ l = 0 T − t ( γ ​ λ ) l ​ δ t + l , δ t = r t + γ ​ V ​ ( s t + 1 ) − V ⁡ ( s t ) \displaystyle\hat{A}_{t}=\sum_{l=0}^{T-t}(\gamma\lambda)^{l}\delta_{t+l},\quad\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t}) (2)

where γ \gamma is the discount factor controlling future reward weighting, λ \lambda is the GAE parameter balancing bias-variance trade-offs, V V represents the value function, and r t r_{t} is the immediate reward. The clipping parameter ϵ \epsilon , typically 0.2, enforces a hard constraint on policy updates by limiting r t ​ ( θ ) r_{t}(\theta) to [ 1 − ϵ , 1 + ϵ ] [1-\epsilon,1+\epsilon] . The min ⁡ ( ) \min() operator implements a pessimistic bound : when advantages A ^ t \hat{A}_{t} are positive, it clips overly optimistic updates; for negative advantages, it does not suppress the update magnitude. This mechanism ensures stable policy improvement within a trust region.

### 2.2 Group Relative Policy Optimization (GRPO)

GRPO ( Shao et al., 2024 ) extends PPO by replacing value-based advantage estimation with group-normalized rewards, eliminating the need for a separate value network. For each prompt x x and its group of M M responses { y ( j ) } j = 1 M \{y^{(j)}\}_{j=1}^{M} , the objective constrains policy updates through:

ℒ GRPO ​ ( θ ) = 𝔼 x ∼ 𝒟 ​ [ 1 M ​ ∑ j = 1 M 1 | y ( j ) | ​ ∑ t = 1 T j min ⁡ ( r t ( j ) ​ ( θ ) ​ A ~ ( j ) , clip ​ ( r t ( j ) ​ ( θ ) , 1 − ϵ , 1 + ϵ ) ​ A ~ ( j ) ) ] . \displaystyle\mathcal{L}^{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{M}\sum_{j=1}^{M}\frac{1}{|y^{(j)}|}\sum_{t=1}^{T_{j}}\min\left(r_{t}^{(j)}(\theta)\tilde{A}^{(j)},\text{clip}\left(r_{t}^{(j)}(\theta),1-\epsilon,1+\epsilon\right)\tilde{A}^{(j)}\right)\right]. (3)

Here, r t ( j ) ​ ( θ ) = π θ ​ ( a t ( j ) | s t ( j ) ) π old ​ ( a t ( j ) | s t ( j ) ) r_{t}^{(j)}(\theta)=\frac{\pi_{\theta}(a_{t}^{(j)}|s_{t}^{(j)})}{\pi_{\text{old}}(a_{t}^{(j)}|s_{t}^{(j)})} remains the importance ratio per token t t in response j j , while A ~ ( j ) \tilde{A}^{(j)} represents the group-relative advantage computed across all responses: A ~ ( j ) = R ( j ) − μ R σ R , μ R = 1 M ​ ∑ j = 1 M R ( j ) , σ R = 1 M ​ ∑ j = 1 M ( R ( j ) − μ R ) 2 \displaystyle\tilde{A}^{(j)}=\frac{R^{(j)}-\mu_{R}}{\sigma_{R}},\quad\mu_{R}=\frac{1}{M}\sum_{j=1}^{M}R^{(j)},\quad\sigma_{R}=\sqrt{\frac{1}{M}\sum_{j=1}^{M}(R^{(j)}-\mu_{R})^{2}} (4) where R ( j ) R^{(j)} is the cumulative reward for response j j , μ R \mu_{R} is the group mean reward, and σ R \sigma_{R} is the reward standard deviation within the response group. This normalization eliminates advantage bias across prompts. The length normalization factor 1 | y ( j ) | \frac{1}{|y^{(j)}|} , with | y ( j ) | |y^{(j)}| denoting response length, ensures equitable contribution for variable-length trajectories.

### 2.3 Raise the Ceiling: Clip-Higher

Standard PPO/GRPO employ symmetric clipping bounds 1 ± ϵ 1\pm\epsilon , which can inadvertently suppress exploratory actions by equally constraining both advantageous and disadvantageous policy updates. To address this, DAPO ( Yu et al., 2025 ) introduces asymmetric clipping, defined as:

clip asym ​ ( r , ϵ l , ϵ h ) = { 1 − ϵ l if ​ r < 1 − ϵ l 1 + ϵ h if ​ r > 1 + ϵ h r otherwise \displaystyle\text{clip}_{\text{asym}}(r,\epsilon_{l},\epsilon_{h})=\begin{cases}1-\epsilon_{l}&\text{if }r<1-\epsilon_{l}\\ 1+\epsilon_{h}&\text{if }r>1+\epsilon_{h}\\ r&\text{otherwise}\end{cases} (5)

The modified objective function leverages these asymmetric bounds to balance exploration and conservatism:

ℒ clip-high ​ ( θ ) = 𝔼 x ∼ 𝒟 ​ [ 1 M ​ ∑ j = 1 M 1 | y ( j ) | ​ ∑ t = 1 T min ⁡ ( r t ​ ( θ ) ​ A ^ t , clip ​ ( r t ​ ( θ ) , 1 − ϵ l , 1 + ϵ h ) ​ A ^ t ) ] \displaystyle\mathcal{L}^{\text{clip-high}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{M}\sum_{j=1}^{M}\frac{1}{|y^{(j)}|}\sum_{t=1}^{T}\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-\epsilon_{l},1+\epsilon_{h})\hat{A}_{t}\right)\right] (6)

where typically ϵ h > ϵ l \epsilon_{h}>\epsilon_{l} (e.g., ϵ h = 0.28 \epsilon_{h}=0.28 , ϵ l = 0.2 \epsilon_{l}=0.2 ) to allow more aggressive exploration while maintaining conservative updates for unfavorable actions.

### 2.4 Token-Level Policy Gradient Loss

Recent work ( Yu et al., 2025 ) has demonstrated that applying policy gradient methods at the token level, rather than the sample level, can yield significant improvements in training stability and final model performance. The token-level variant of the GRPO objective equipped with clip-higher can be expressed as:

ℒ GRPO ​ ( θ ) = 𝔼 x ∼ 𝒟 ​ [ 1 ∑ j = 1 M T j ​ ∑ j = 1 M ∑ t = 1 T j min ⁡ ( r t ( j ) ​ ( θ ) ​ A ~ ( j ) , clip ​ ( r t ( j ) ​ ( θ ) , 1 − ϵ l , 1 + ϵ h ) ​ A ~ ( j ) ) ] \displaystyle\mathcal{L}^{\text{GRPO}}(\theta)=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\min\left(r_{t}^{(j)}(\theta)\tilde{A}^{(j)},\text{clip}\left(r_{t}^{(j)}(\theta),1-\epsilon_{l},1+\epsilon_{h}\right)\tilde{A}^{(j)}\right)\right] (7)

where ∑ j = 1 M T j \sum_{j=1}^{M}T_{j} serves as the normalization factor across all tokens. Token-level policy loss ensures that each token contributes equally to the gradient updates rather than weighting entire samples uniformly, which enables more balanced optimization across variable-length reasoning chains and leads to significantly more stable policy updates, particularly for long trajectories where traditional sample-level averaging might dilute critical learning signals. The token-level granularity preserves fine-grained training signals throughout extended reasoning processes while maintaining consistent gradient magnitudes regardless of sequence length.

## 3 Method

In this section, we present the details of the supervised fine-tuning and reinforcement learning processes used to train Klear-Reasoner.

### 3.1 long Chain-of-Thought Supervised Fine-tuning

For mathematical and coding tasks, we adopt a quality-centric data construction strategy inspired by OpenThoughts ( Guha et al., 2025 ) , prioritizing high-quality data over superficial diversity. This approach is reflected in four key design principles: • Our prompts are exclusively curated from high-quality sources: mathematics prompts primarily draw from OpenThoughts ( Guha et al., 2025 ) , NuminaMath ( LI et al., 2024 ) , and AceReason-Nemotron 1.1 ( Liu et al., 2025 ) , while coding prompts are gathered from OpenThoughts, OpenCodeReasoning ( Ahmad et al., 2025b ) , AceReason-Nemotron 1.1, TACO ( Li et al., 2023 ) , Apps ( Hendrycks et al., 2021 ) , and Codeforces ( Penedo et al., 2025 ) .

• To ensure data uniqueness and prevent contamination, strict deduplication protocols are implemented, including exact match removal of queries and filtering prompts with 9-gram overlap against test samples following ( Chen et al., 2025b , Liu et al., 2025 ) .

• The teacher model employed for response generation is Deepseek-R1-0528 3 3 3 https://huggingface.co/deepseek-ai/DeepSeek-R1-0528 , which produces up to 16 responses per prompt through sampling.

• After evaluating sample difficulty using Qwen3-8B ( Yang et al., 2025 ) , we retain all responses since most samples qualify as difficult, consistent with Section 5.1.1 insights.

Through this process, we construct a high-quality reasoning dataset containing 1.5 million math and coding samples.

To effectively leverage these high-quality reasoning data, we fine-tune the model using the standard SFT objective: ℒ LM ( θ ) = − 1 N ∑ j = 1 N ∑ t = 1 T j π θ ( a t | s t ) \displaystyle\mathcal{L}^{\text{LM}}(\theta)=-\frac{1}{N}\sum_{j=1}^{N}\sum_{t=1}^{T_{j}}\pi_{\theta}(a_{t}|s_{t}) (8)

where θ \theta denotes model parameters, N N is the total training sequences, T j T_{j} is the token count of sequence j j , a t a_{t} is the target token at position t t , and s t s_{t} is the contextual history. The term π θ ​ ( a t | s t ) \pi_{\theta}(a_{t}|s_{t}) represents the model’s probability of generating the true token a t a_{t} given context s t s_{t} . By minimizing the negative average probability across all tokens and sequences, this objective aligns the model’s output distribution with high-quality teacher demonstrations.

### 3.2 Reinforcement Learning

#### 3.2.1 Data curation