Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization — part 2/4
Cleaned-text code points [15351, 30325). Bibliography omitted here; source.md includes it.
We aim to filter high-quality reasoning data for RL. To this end, we have developed a data collection and validation pipeline that integrates prompts from various sources, including Skywork-OR1 ( He et al., 2025 ) , Acereason ( Chen et al., 2025b ) , NuminaMath ( LI et al., 2024 ) , and DeepScaleR ( Luo et al., 2025b , Luo et al., 2025a ) . A 9-gram filtering mechanism is employed to prevent overlap with common benchmark datasets ( Chen et al., 2025b , Liu et al., 2025 ) , and exact deduplication is applied to ensure the uniqueness of each sample. For code problems, we apply multiple filtering strategies to mitigate noisy or low-quality samples that could harm RL training. We first remove examples with fewer than 16 test cases, as these are more susceptible to false positives. For each retained prompt, we generate 16 completions using DeepSeek-R1-0120 4 4 4 https://huggingface.co/deepseek-ai/DeepSeek-R1 . Only those prompts for which DeepSeek-R1-0120 achieves a p a s s @ 16 pass@16 greater than 0.5 are kept, reducing noise introduced by faulty or insufficient test cases. For math problems, we focus on ensuring correctness and clarity, particularly because the raw data, often collected from the web and processed via OCR or parsers, can contain substantial noise (e.g., incorrect questions or answers). To address this, we generate 16 responses per prompt using DeepSeek-R1-0120 and retain only those for which a majority of completions pass a rule-based validator. Finally, after these rigorous filtering steps, we construct a high-quality RL dataset consisting of 88K math samples and 18K code samples.
#### 3.2.2 Gradient-Preserving Clipping Policy Optimization
In the classical PPO algorithm ( Schulman et al., 2015 , Schulman et al., 2017 ) without clipping, we can derive the gradient of the loss function as shown in Eq. 9 . Here, f ( a , s ) f(a,s) denotes the logits output by the policy network. b b iterates over all possible actions in the action space 𝒜 \mathcal{A} at state s t s_{t} .
∇ θ ℒ PPO \displaystyle\nabla_{\theta}\mathcal{L}^{\text{PPO}} = 𝔼 t [ r t ( θ ) ⋅ ϕ θ ( a t , s t ) ⋅ A ^ t ] , \displaystyle=\mathbb{E}_{t}\left[r_{t}(\theta)\cdot\phi_{\theta}(a_{t},s_{t})\cdot\hat{A}_{t}\right], (9) where ϕ θ ( a t , s t ) \displaystyle\text{where }\phi_{\theta}(a_{t},s_{t}) = ∂ f θ ( a t , s t ) ∂ θ − ∑ b ∈ 𝒜 π θ ( b | s t ) ⋅ ∂ f θ ( b , s t ) ∂ θ \displaystyle=\frac{\partial f_{\theta}(a_{t},s_{t})}{\partial\theta}-\sum_{b\in\mathcal{A}}\pi_{\theta}(b|s_{t})\cdot\frac{\partial f_{\theta}(b,s_{t})}{\partial\theta} However, the absence of clipping allows the importance sampling ratio r t ( θ ) r_{t}(\theta) to range from ( 0 , + ∞ ) (0,+\infty) . While this broad range enhances exploration by enabling significant policy updates in promising directions, such overly aggressive exploration may lead to training instability due to excessively large gradient updates. This can result in oscillating or divergent policy updates, ultimately hindering convergence to the optimal policy.
To mitigate the aforementioned issues, a common approach is to apply clipping to the importance sampling ratio, constraining its upper and lower bounds, as shown in Eq. 6 and Eq. 3 . Similarly, we can derive the gradient expression after applying the clipping operation, as presented in Eq. 10 .
∇ θ ℒ PPO-clip \displaystyle\nabla_{\theta}\mathcal{L}^{\text{PPO-clip}} = 𝔼 t [ 𝕀 t ⋅ r t ( θ ) ⋅ ϕ θ ( a t , s t ) ⋅ A ^ t ] \displaystyle=\mathbb{E}_{t}\left[\mathbb{I}_{t}\cdot r_{t}(\theta)\cdot\phi_{\theta}(a_{t},s_{t})\cdot\hat{A}_{t}\right] (10) where 𝕀 t \displaystyle\text{where}\quad\mathbb{I}_{t} = { 1 , if r t ( θ ) ⋅ A ^ t ≤ clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) ⋅ A ^ t 0 , otherwise \displaystyle=\begin{cases}1,&\text{if }r_{t}(\theta)\cdot\hat{A}_{t}\leq\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\cdot\hat{A}_{t}\\[3.00003pt] 0,&\text{otherwise}\end{cases}
From the above equation, we can infer the following: when A ^ t > 0 \hat{A}_{t}>0 , gradients are only propagated if the importance sampling ratio r t ( θ ) r_{t}(\theta) falls within the range ( 0 , 1 + ϵ ) (0,1+\epsilon) ; when A ^ t < 0 \hat{A}_{t}<0 , gradients are only present if r t ( θ ) r_{t}(\theta) lies within ( 1 − ϵ , + ∞ ) (1-\epsilon,+\infty) . Compared to the unclipped case, this clipping mechanism effectively restricts the range of gradient updates, capping the positive advantage updates to prevent over-encouragement, and flooring the negative advantage updates to avoid excessive punishment. Although introducing clip operations stabilized training to a certain extent, completely discarding the clipped token gradients also led to two key issues.
• High-entropy token clipping. A key side effect of clipping is the indiscriminate suppression of gradients for all tokens whose importance sampling ratios exceed predefined thresholds 1 + ϵ 1+\epsilon , regardless of whether those tokens may contain valuable exploratory behaviors. In our preliminary experiments, we observed a phenomenon similar to that reported in ( Yu et al., 2025 , Chen et al., 2025a ) : high-entropy tokens associated with reflective behaviors often lie at crucial decision branches. However, due to their elevated r t ( j ) ( θ ) r_{t}^{(j)}(\theta) values, these tokens are typically clipped out after the first on-policy update, severely hampering the model’s capacity to explore.
• Delayed convergence of negative samples. When the importance sampling ratio of suboptimal trajectories falls below 1 − ϵ 1-\epsilon , their gradients are forcibly clipped, preventing the model from updating based on these signals. As a result, the model must repeatedly sample similar suboptimal trajectories before it can correct its behavior, leading to slower convergence on negative examples and hindering timely policy adjustment.
Existing methods incorporating gradient signals from beyond clip boundaries, such as DAPO’s clip-higher operation that elevates the clip upper bound to preserve high-entropy tokens, fail to resolve the underlying issue and still inherently truncate gradient signals from certain samples. Fundamentally, the issue described above also stems from side effects induced by the clip operation. Therefore, the core motivation of our method is to propose a balanced approach that bridges unclipped and clipped methodologies. This approach aims to incorporate gradient signals from samples beyond clip boundaries while constraining these out-of-bound gradient signals within a defined range. Such constraint prevents them from exerting excessive influence on gradient updates, thereby safeguarding training stability.
To address the above challenges, we propose a G radient- P reserving clipping P olicy O ptimization (GPPO). Unlike traditional techniques that completely discard gradients outside the clipping range, our method introduces an innovative truncation mechanism that incorporates previously clipped tokens into the computational graph. This design ensures that valuable learning signals are retained while maintaining stable policy updates. Specifically, taking GRPO loss as an example, we modify Eq. 7 as follows:
ℒ GPPO ( θ ) \displaystyle\mathcal{L}^{\text{GPPO}}(\theta) = 𝔼 x ∼ 𝒟 [ 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j min ( δ A ~ ( j ) , clip ( δ , 1 − ϵ l sg ( δ ) δ , 1 + ϵ h sg ( δ ) δ ) A ~ ( j ) ) ] \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\min\left(\delta\tilde{A}^{(j)},\text{clip}\left(\delta,\frac{1-\epsilon_{l}}{\operatorname{sg}(\delta)}\delta,\frac{1+\epsilon_{h}}{\operatorname{sg}(\delta)}\delta\right)\tilde{A}^{(j)}\right)\right] (11) Here, δ = r t ( j ) ( θ ) \delta=r_{t}^{(j)}(\theta) is the importance sampling ratio, and sg ( ⋅ ) \operatorname{sg}(\cdot) denotes the stop-gradient operation. Notably, the term δ sg ( δ ) \frac{\delta}{\operatorname{sg}(\delta)} is always numerically equal to 1, so the forward computation remains unchanged . Since we decouple gradient propagation from the clipping constraint, the backward computation differs from the standard clipping approach, which directly discards gradients of samples falling outside the clipping bounds. The gradient expression of our proposed method is presented in Eq. 12 .
∇ θ ℒ GPPO ( θ ) \displaystyle\nabla_{\theta}\mathcal{L}^{\text{GPPO}}(\theta) = 𝔼 x ∼ 𝒟 [ 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j ℱ j , t ( θ ) ⋅ ϕ θ ( a j , t , s j , t ) ⋅ A ~ ( j ) ] , \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\mathcal{F}_{j,t}(\theta)\cdot\phi_{\theta}(a_{j,t},s_{j,t})\cdot\tilde{A}^{(j)}\right], (12) where ℱ j , t ( θ ) \displaystyle\text{where}\quad\mathcal{F}_{j,t}(\theta) = { 1 − ϵ l if δ < 1 − ϵ l and A ~ ( j ) < 0 , 1 + ϵ h if δ > 1 + ϵ h and A ~ ( j ) > 0 , δ OPEN otherwise (i.e., δ A ~ ( j ) ≤ clip ( δ , 1 − ϵ l , 1 + ϵ h ) ⋅ A ~ ( j ) ) . \displaystyle=\begin{cases}1-\epsilon_{l}&\text{if }\delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0,\\[5.0pt] 1+\epsilon_{h}&\text{if }\delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}>0,\\[5.0pt] \delta&\text{otherwise (i.e., }\delta\tilde{A}^{(j)}\leq\text{clip}(\delta,1-\epsilon_{l},1+\epsilon_{h})\cdot\tilde{A}^{(j)}).\end{cases}
By comparing Eq. 10 and Eq. 12 , we observe that our method avoids cases where the gradient becomes zero, meaning it fully leverages the gradient signals from all samples. Specifically, when the positive samples have a large importance sampling ratio δ > 1 + ϵ h \delta>1+\epsilon_{h} , it may correspond to high-entropy tokens. Our method preserves the gradients of these samples, thereby enhancing the model’s exploratory capability . Conversely, when the negative samples have δ < 1 − ϵ l \delta<1-\epsilon_{l} , the standard clipping mechanism discards its gradient, leading to delayed convergence . By retaining the gradient in these cases, our approach enables faster learning from negative samples, thus equipping the model with a stronger ability to perform rapid trial and error.
It is particularly noteworthy that although our method incorporates the gradient contributions from samples outside the clipping range, it still maintains stable policy updates. This is because for tokens clipped under the conditions δ < 1 − ϵ l and A ~ ( j ) < 0 \delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0 , their backpropagated gradients are constrained to 1 − ϵ l 1-\epsilon_{l} , while for those clipped when δ > 1 + ϵ h and A ~ ( j ) > 0 \delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}>0 , their gradients are constrained to 1 + ϵ h 1+\epsilon_{h} . In other words, for clipped tokens, we adjust their gradients in a controlled manner to ensure training stability, avoiding drastic gradient updates as described in Eq. 9 . In summary, our approach not only accelerates the model’s convergence when learning from negative samples but also facilitates exploration through high-entropy tokens, while maintaining training stability via moderate gradient backpropagation. A more general form of GPPO is shown in Appendix A.2 .
Additionally, concurrent work by MiniMax-M1 ( Chen et al., 2025a ) explores a related direction through their proposed CISPO method. To enable gradient backpropagation on all tokens, CISPO adopts a vanilla REINFORCE objective with a corrected distribution for RL training. While Appendix A.1 reveals gradient similarities between CISPO and our method GPPO, their update strategies diverge fundamentally. GPPO adopts the pessimistic update principle established in Section 2.1 : When the advantage A ^ t \hat{A}_{t} is positive, GPPO suppresses overly optimistic updates, while CISPO does not impose any restrictions; Conversely, when the advantage is negative, GPPO imposes no restriction while CISPO does. Crucially, this distinction stems from differing design philosophies. GPPO derives a more principled clipping approach through gradient analysis, while CISPO leans more toward heuristic design.
#### 3.2.3 RL Training with SFT Loss
Similar to Yue et al. (2025) , we calculate the language model (LM) loss for the positive examples generated in each rollout and incorporate this loss into the RL training process. First, we argue that the SFT loss can improve the utilization efficiency of positive examples. Additionally, the SFT loss on positive examples serves as an anchor during training, constraining the policy model’s output to maintain a reasonable distribution. We believe this helps mitigate reward hacking behavior and enhances training stability. Similarly, we employ a token-level loss, with the corresponding formula as follows: ℒ LM ( θ ) = − 1 ∑ j ∈ Φ T j ∑ j ∈ Φ ∑ t = 1 T j π θ ( a t | s t ) \displaystyle\mathcal{L}^{\text{LM}}(\theta)=-\frac{1}{{\textstyle\sum_{j\in\Phi}}T_{j}}\sum_{j\in\Phi}\sum_{t=1}^{T_{j}}\pi_{\theta}(a_{t}|s_{t}) (13) Let Φ \Phi denote the index set of correct samples from examples in rollout. The final ℒ LM ( θ ) \mathcal{L}^{\text{LM}}(\theta) is jointly trained with the GPPO loss, with the corresponding expression given by the following formula:
ℒ ( θ ) = ℒ GPPO ( θ ) + α ℒ LM ( θ ) \displaystyle\mathcal{L}^{(}\theta)=\mathcal{L}^{\text{GPPO}}(\theta)+\alpha\mathcal{L}^{\text{LM}}(\theta) (14) where α \alpha serves as a hyperparameter that balances the weighting between these two losses.
#### 3.2.4 Reward Design for Math and Code RL
Our reinforcement learning framework employs distinct reward schemes for mathematical reasoning and code generation tasks. For mathematical tasks, we use a binary reward system where solutions receive either positive or negative rewards based on final correctness. Notably, we penalize responses that fail to encapsulate their reasoning process within designated <think>...</think> tags.
In terms of code tasks, RL often struggles with sparse rewards, particularly in code generation where models may produce largely correct solutions that fail only on corner cases. Traditional approaches label such samples as entirely negative, disregarding their partial correctness. We argue that these partially correct solutions contain valuable learning signals. To overcome this challenge, we introduce a soft reward mechanism based on test case pass rates. For example, if generated code passes 4 out of 16 test cases, it receives a proportional reward of 0.25 (4/16). The soft p a s s @ k pass@k reward system provides granular feedback that mitigates sparse rewards, preserves learning signals from partially correct solutions, and encourages incremental quality improvement.
## 4 Evaluation
In this section, we present the final experimental results of Klear-Reasoner-8B. We first provide a detailed description of the training configuration, followed by an analysis of the training outcomes. Finally, we discuss the evaluation results.
### 4.1 Training Details