Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization — part 4/4
Cleaned-text code points [46188, 58208). Bibliography omitted here; source.md includes it.
Beyond improving average rewards, the soft reward also reduces reward variance, a crucial factor for stable policy optimization. In the hard reward setting, the binary nature of the reward leads to large fluctuations, particularly in the early stages when fully correct solutions are rare, thereby increasing the variance of policy gradient estimates and introducing noise into training. In contrast, the more continuous and structured feedback from soft reward stabilizes learning and guides the model toward better behaviors with more consistent gradients.
These improvements in both reward density and stability ultimately enhance downstream performance. As shown in Figure 3(a) , the model trained with soft reward achieves a LiveCodeBench V5 score of 61.0, outperforming the 59.2 achieved with hard reward. This 1.8-point gain confirms that addressing reward sparsity and variance not only improves learning dynamics, but also boosts generalization in code RL.
#### 5.2.4 Improving Code RL via Test Case Filtering
We observe that certain test cases in open-source code datasets contain flaws that can cause correct code to fail execution. To mitigate the negative impact of such problematic cases during code RL training, we adopt a data filtering strategy based on estimated p a s s @ 16 pass@16 scores. Applying this filtering to the DeepScaleR dataset and training for 100 RL steps, we find that models trained on filtered data consistently outperform those trained on unfiltered data in terms of final performance on LiveCodeBench V5, as illustrated in Figure 4 . Moreover, the filtered-data models exhibit a more stable and steady upward trend throughout training. These improvements indicate that test case filtering is more effective because it reduces false negatives where correct completions are unfairly penalized due to flawed test cases. This results in a more reliable reinforcement learning signal and enables more efficient policy learning.
#### 5.2.5 Impact of Zero-Advantage Sample Filtering
In GRPO, when all responses generated from a given prompt are either entirely correct or entirely incorrect, the advantage for that group evaluates to zero. This raises an important question about the role of such zero-advantage groups in reinforcement learning optimization. To investigate this empirically, we examine the effects of filtering out these zero-advantage groups during training.
The results, as illustrated in Figure 5(a) , reveal several findings. The filtered configuration demonstrates consistent and steady improvement on the AIME2024 benchmark throughout the RL training process. This stability suggests that the removal of zero-advantage samples helps maintain clear optimization signals. In contrast, while the unfiltered setting shows stable growth in reward values, its AIME2024 performance fluctuates significantly. This discrepancy indicates that while reward optimization may appear unaffected, the model’s generalization capability becomes less predictable when zero-advantage samples are included.
The underlying mechanism for this behavior can be explained through the lens of policy gradient optimization. Zero-advantage samples inherently produce vanishing policy gradients, providing no meaningful learning signal. More critically, their presence in large quantities effectively dilutes the contribution of non-zero advantage samples, creating ambiguity in the optimization direction. This not only slows down learning but may also introduce noise into the training process, potentially explaining the observed instability in generalization performance. The zero-advantage sample filtering approach appears to mitigate these issues by focusing the model’s learning on samples that provide clearer, more actionable feedback.
## 6 Conclusions
Klear-Reasoner advances reasoning capabilities in mathematics and programming by integrating long Chain-of-Thought supervised fine-tuning with Gradient-Preserving Clipping Policy Optimization. The quality-centric SFT strategy ensures consistent learning of accurate reasoning patterns, while GPPO addresses the limitations of traditional clipping by retaining gradient information from all tokens, enabling both stable policy updates and effective exploration. Complementary techniques, such as soft reward design, data filtering, and balanced SFT supervision, further enhance reinforcement learning efficiency. Evaluations on multiple challenging benchmarks show that Klear-Reasoner consistently matches or surpasses state-of-the-art models of comparable scale. These results demonstrate that principled data curation, targeted SFT, and carefully designed RL optimization can jointly deliver substantial improvements in long-form reasoning performance.
## Appendix A Appendix
### A.1 CISPO Gradient Analysis
To better understand the difference between GPPO and the concurrent work CISPO, we analyze the gradient of CISPO in this section. CISPO builds upon a vanilla REINFORCE objective by incorporating GRPO’s relative reward scaling within response groups, while also integrating DAPO’s asymmetric clipping strategy on the corrected distribution. The specific loss function takes the following form: ℒ CISPO ( θ ) \displaystyle\mathcal{L}^{\text{CISPO}}(\theta) = 𝔼 x ∼ 𝒟 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j δ A ~ ( j ) l o g ( π θ ( a t ( j ) | s t ( j ) ) . \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\delta\tilde{A}^{(j)}log(\pi_{\theta}(a_{t}^{(j)}|s_{t}^{(j)}). (15)
By expanding the gradient of the CISPO loss function, we obtain the following expression: ∇ θ ℒ CISPO ( θ ) \displaystyle\nabla_{\theta}\mathcal{L}^{\text{CISPO}}(\theta) = 𝔼 x ∼ 𝒟 [ 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j ℱ j , t ( θ ) ⋅ ϕ θ ( a j , t , s j , t ) ⋅ A ~ ( j ) ] , \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\mathcal{F}_{j,t}(\theta)\cdot\phi_{\theta}(a_{j,t},s_{j,t})\cdot\tilde{A}^{(j)}\right], (16) where ℱ j , t ( θ ) \displaystyle\text{where}\quad\mathcal{F}_{j,t}(\theta) = { 1 − ϵ l if δ < 1 − ϵ l and A ~ ( j ) < 0 , 1 + ϵ h if δ > 1 + ϵ h and A ~ ( j ) > 0 , 1 − ϵ l if δ < 1 − ϵ l and A ~ ( j ) > 0 , 1 + ϵ h if δ > 1 + ϵ h and A ~ ( j ) < 0 , δ otherwise . \displaystyle=\begin{cases}1-\epsilon_{l}&\text{if }\delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0,\\[5.0pt] 1+\epsilon_{h}&\text{if }\delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}>0,\\[5.0pt] 1-\epsilon_{l}&\text{if }\delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}>0,\\[5.0pt] 1+\epsilon_{h}&\text{if }\delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}<0,\\[5.0pt] \delta&\text{otherwise}.\end{cases} As shown in Eq. 12 , GPPO inherits PPO’s pessimistic update strategy compared to CISPO. Specifically, when δ < 1 − ϵ l and A ~ ( j ) > 0 \delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}>0 , the gradient term ℱ j , t ( θ ) = 1 − ϵ l \mathcal{F}_{j,t}(\theta)=1-\epsilon_{l} in CISPO but remains ℱ j , t ( θ ) = δ \mathcal{F}_{j,t}(\theta)=\delta in GPPO, where 0 < δ < ϵ l 0<\delta<\epsilon_{l} . This indicates that GPPO distrusts overly optimistic improvements and suppresses such cases through conservative updates, while CISPO applies more aggressive updates with a larger magnitude (i.e., 1 − ϵ l 1-\epsilon_{l} ). Conversely, when δ > 1 + ϵ l and A ~ ( j ) < 0 \delta>1+\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0 , CISPO’s gradient term ℱ j , t ( θ ) = 1 + ϵ l \mathcal{F}_{j,t}(\theta)=1+\epsilon_{l} , whereas GPPO retains ℱ j , t ( θ ) = δ \mathcal{F}_{j,t}(\theta)=\delta , where δ > ϵ h \delta>\epsilon_{h} . Here, GPPO fully trusts negative feedback without suppression, while CISPO applies milder updates (i.e., 1 + ϵ l 1+\epsilon_{l} ). GPPO’s PPO-inherited pessimistic update strategy ( Schulman et al., 2017 ) , which distrusts excessively optimistic improvements but fully accepts negative feedback, helps prevent policy collapse and enhances training stability.
### A.2 The General Form of GPPO
In this section, we further refine Eq. 11 to present a more generalized form of the loss function, which is formulated as follows: ℒ GPPO ( θ ) \displaystyle\mathcal{L}^{\text{GPPO}}(\theta) = 𝔼 x ∼ 𝒟 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j ℓ ( j ) , \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\ell^{(j)}, (17) where ℓ ( j ) \displaystyle\text{where}\quad\ell^{(j)} = { β 1 ⋅ 1 − ϵ l sg ( δ ) δ ⋅ A ~ ( j ) , if δ < 1 − ϵ l and A ~ ( j ) < 0 , β 2 ⋅ 1 + ϵ h sg ( δ ) δ ⋅ A ~ ( j ) , if δ > 1 + ϵ h and A ~ ( j ) > 0 , δ ⋅ A ~ ( j ) , OPEN otherwise (i.e., δ A ~ ( j ) ≤ clip ( δ , 1 − ϵ l , 1 + ϵ h ) ⋅ A ~ ( j ) ) . \displaystyle=\begin{cases}\beta_{1}\cdot\,\dfrac{1-\epsilon_{l}}{\operatorname{sg}(\delta)}\,\delta\,\cdot\tilde{A}^{(j)},&\text{if }\delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0,\\[8.0pt] \beta_{2}\cdot\,\dfrac{1+\epsilon_{h}}{\operatorname{sg}(\delta)}\,\delta\,\cdot\tilde{A}^{(j)},&\text{if }\delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}>0,\\[8.0pt] \delta\cdot\,\tilde{A}^{(j)},&\text{otherwise (i.e., }\delta\tilde{A}^{(j)}\leq\text{clip}(\delta,1-\epsilon_{l},1+\epsilon_{h})\cdot\tilde{A}^{(j)}).\end{cases} Here, β 1 \beta_{1} and β 2 \beta_{2} are two tunable hyperparameters that control the magnitude of the two clipped boundary losses. In additionl, they influence the scale of gradient backpropagation from the clipped boundaries during optimization. By further expanding the gradient of Eq. 17 , we obtain the following equation: ∇ θ ℒ GPPO ( θ ) \displaystyle\nabla_{\theta}\mathcal{L}^{\text{GPPO}}(\theta) = 𝔼 x ∼ 𝒟 [ 1 ∑ j = 1 M T j ∑ j = 1 M ∑ t = 1 T j ℱ j , t ( θ ) ⋅ ϕ θ ( a j , t , s j , t ) ⋅ A ~ ( j ) ] , \displaystyle=\mathbb{E}_{x\sim\mathcal{D}}\left[\frac{1}{\sum_{j=1}^{M}T_{j}}\sum_{j=1}^{M}\sum_{t=1}^{T_{j}}\mathcal{F}_{j,t}(\theta)\cdot\phi_{\theta}(a_{j,t},s_{j,t})\cdot\tilde{A}^{(j)}\right], (18) where ℱ j , t ( θ ) \displaystyle\text{where}\quad\mathcal{F}_{j,t}(\theta) = { β 1 ⋅ ( 1 − ϵ l ) if δ < 1 − ϵ l and A ~ ( j ) < 0 , β 2 ⋅ ( 1 + ϵ h ) if δ > 1 + ϵ h and A ~ ( j ) > 0 , δ OPEN otherwise (i.e., δ A ~ ( j ) ≤ clip ( δ , 1 − ϵ l , 1 + ϵ h ) ⋅ A ~ ( j ) ) . \displaystyle=\begin{cases}\beta_{1}\cdot(1-\epsilon_{l})&\text{if }\delta<1-\epsilon_{l}\text{ and }\tilde{A}^{(j)}<0,\\[5.0pt] \beta_{2}\cdot(1+\epsilon_{h})&\text{if }\delta>1+\epsilon_{h}\text{ and }\tilde{A}^{(j)}>0,\\[5.0pt] \delta&\text{otherwise (i.e., }\delta\tilde{A}^{(j)}\leq\text{clip}(\delta,1-\epsilon_{l},1+\epsilon_{h})\cdot\tilde{A}^{(j)}).\end{cases} The introduction of β 1 \beta_{1} and β 2 \beta_{2} allows Eq. 17 to achieve finer-grained gradient control compared to Eq. 11 .
### A.3 License for Scientific Artifacts
In this paper, we use the VERL 7 7 7 https://github.com/volcengine/verl framework for RL training, which was developed by the ByteDance team and is released under the Apache License 2.0.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .