Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization — part 3/4

Cleaned-text code points [30325, 46188). Bibliography omitted here; source.md includes it.

Here is a description of the training process for Klear-Reasoner-8B. Klear-Reasoner-8B is based on the Qwen3-8B-Base model, first fine-tuned with long CoT SFT and then trained using RL on math and coding tasks. For the training data, we collect math and coding problems from various sources and apply 9-gram filtering to remove data contaminated with evaluation sets, following ( Liu et al., 2025 ) . For the long CoT SFT data, we use DeepSeek-R1-0528 to generate responses for each question. Whether it is long CoT SFT or RL, we set the maximum training length to 32K. Following Guha et al., 2025 , during long CoT SFT, we adopt a maximum learning rate of 8 ​ e − 5 8e^{-5} with a cosine decay strategy, decreasing to a minimum of 4 ​ e − 5 4e^{-5} . We train for a total of 4 epochs to ensure model convergence. For RL, we employ a multi-stage training approach with a global batch size of 128 per step. Specifically, we adopt a two-stage training method, first training math RL and then training code RL. We use an off-policy training strategy, with a constant learning rate of 1 ​ e − 6 1e^{-6} and a mini-batch size of 16 for math RL, and a constant learning rate of 5 ​ e − 7 5e^{-7} with a mini-batch size of 32 for coding RL. For rollout, we sample 8 responses for each prompt. It is worth mentioning that we use the pass rate of test cases as a reward for code RL. The RL training is conducted jointly with SFT loss, as described in Section 3.2.3 , where we set α \alpha to 0.1. Furthermore, we utilize the GPPO proposed in Section 3.2.2 . We apply only the token-level policy gradient loss described in Section 2.4 and Clip-Higher with ϵ h = 0.28 \epsilon_{h}=0.28 as described in Section 2.3 , following ( He et al., 2025 ) , without using KL loss during training.

### 4.2 Benchmarks and Baselines

We evaluate the model on challenging benchmarks to assess its capabilities. For mathematical proficiency, we measure performance on AIME2024, AIME2025 as well as HMMT2025 Feb. For coding ability, we use LiveCodeBench V5 (covering August 2024 to February 2025) and LiveCodeBench V6 (spanning February 2025 to May 2025). We compare our model against several strong baselines, including OpenReasoning-Nemotron-7B ( Ahmad et al., 2025a ) , Deepseek-R1-0528-Distill-8B ( DeepSeek-AI et al., 2025 ) , Qwen3-8B ( Yang et al., 2025 ) , POLAR-4B-Preview ( An et al., 2025 ) , AceReason-Nemotron-1.1-7B ( Liu et al., 2025 ) and MiMo-7B-RL ( Xiaomi, 2025 ) , AReal-boba-RL-7B ( Fu et al., 2025 ) and Skywork-OR1 ( He et al., 2025 ) .

### 4.3 Evaluation Results

As shown in Table 1 , we present the evaluation results on math and code benchmarks. We have made the following results: • SFT Model Performance. Klear-Reasoner-8B-SFT model achieves performance comparable to Qwen3-8B. Notably, while Qwen3-8B is obtained through logits distillation from a larger model, Klear-Reasoner-8B-SFT is trained solely through data distillation (i.e., long CoT SFT), demonstrating the effectiveness of our long CoT SFT approach.

• RL Model Performance. Building upon SFT model, we further refine Klear-Reasoner-8B through RL fine-tuning for math and code tasks. With a 32K inference budget, Klear-Reasoner-8B already matches the performance of community SOTA models operating at 64K/96K inference budgets, achieving specific scores of 83.2% on AIME2024, 75.6% on AIME2025, 60.3% on HMMT2025, 61.6% on LiveCodeBench V5, and 53.1% on LiveCodeBench V6.

• More Reasoning Budget. When we expand the inference budget to 64K and adopt the YaRN ( Peng et al., 2024 ) method with a scaling factor of 2.5, Klear-Reasoner-8B achieves optimal performance, scoring 90.5% on AIME2024, 83.2% on AIME2025, 70.8% on HMMT2025, 66.0% on LiveCodeBench V5, and 58.1% on LiveCodeBench V6. It is worth noting that regardless of whether it is long CoT SFT or RL, our model is only trained on 32K length.

• RL Fine-Tuning Surpasses Data-Heavy SFT. OpenReasoning-Nemotron-7B is derived from Qwen2.5-7B using 5 million long CoT distillation samples from DeepSeek-R1-0528-Distill-8B. Although Klear-Reasoner-8B-SFT, trained with fewer data samples, underperforms OpenReasoning-Nemotron-7B ( Ahmad et al., 2025a ) , the RL-trained Klear-Reasoner-8B with a 64K inference budget still significantly surpasses OpenReasoning-Nemotron-7B. This validates that RL fine-tuning can effectively compensate for the disadvantage of limited training data and further boost model performance.

These results indicate that even when starting from a high-performance SFT model, a well-designed RL approach can further enhance model capabilities.

## 5 Analysis

In this section, we present a detailed analytical study of long-chain-of-thought SFT and RL methods. The SFT model used in these experiments differs from the one employed in the previously introduced Klear-Reasoner-8B-SFT, and is instead based on an earlier version of our SFT model (as reported in Table 3 ). Throughout all evaluations, we maintain strict control over individual variables to ensure valid and fair comparisons.

### 5.1 SFT Analyses

#### 5.1.1 Impact of data correctness

Understanding the correctness of CoT samples is essential for effective model training. Our study leverages the OpenR1-Math-220k dataset 5 5 5 https://huggingface.co/datasets/open-r1/OpenR1-Math-220k , which provides explicit correctness labels, and reveals that the impact of data correctness on performance is largely dependent on task difficulty, challenging the conventional belief that using only correct data is always optimal.

Initially, when comparing SFT performance on carefully curated, balanced subsets of the full dataset, we observe no statistically significant difference between training solely on correct CoT data and using a mixture of correct and incorrect responses. For example, as shown in the “Overall” column of Table 2 , the performance gap is minimal: AIME 2025 benchmark yields 32.29% for correct-only training versus 32.50% for mixed data.

The critical insight emerges when we examine the data through the lens of difficulty. By stratifying the dataset into easy and hard subsets based on model-annotated difficulty labels, a striking divergence become apparent. For easy tasks, training solely on correct data consistently yield superior results. For instance, on the AIME 2024 benchmark, using only correct data resulted in a 4.78% performance improvement compared to training with mixed responses. This suggests that when the underlying reasoning is relatively straightforward and aligns well with the model’s prior capabilities, exposure to incorrect paths acts primarily as detrimental noise, interfering with the model’s grasp of established, correct patterns.

In contrast, for hard tasks, incorporating incorrect examples alongside correct ones surprisingly improved performance. On the AIME 2024 Hard subset, training with mixed data achieved a 1.66% higher accuracy than using only correct samples. This counterintuitive finding indicates that in scenarios characterized by high uncertainty and weaker initial learning signals, incorrect examples serve a valuable purpose. They provide necessary contrast, helping the model more effectively discriminate between valid and invalid reasoning paths. This mechanism may enhance the model’s exploratory capacity within the solution space, preventing it from becoming trapped in suboptimal local reasoning patterns. This dynamic parallels the classic “exploration vs. exploitation” trade-off observed in reinforcement learning.

This difficulty-dependent effect directly inform our final data strategy. An analysis of the full training dataset shows that a substantial majority (88%) of samples are labeled as hard. Given this pronounced bias towards challenging examples and our empirical finding that mixed data benefits learning on such tasks, we deliberately choose not to filter CoT samples based on correctness during the training phase. The presence of incorrect reasoning, within a predominantly hard dataset, is leveraged as a beneficial learning signal rather than treated as noise to be eliminated.

#### 5.1.2 Ablation of teacher models

In this section, we conduct an ablation study to evaluate how the choice of teacher model affects the effectiveness of supervised fine-tuning. Specifically, we use a fixed set of prompts from the OpenThoughts3-1.2M dataset 6 6 6 https://huggingface.co/datasets/open-thoughts/OpenThoughts3-1.2M and generate responses using three different teacher models: QwQ-32B, DeepSeek-R1-0120, and DeepSeek-R1-0528.

From Table 3 , we observe a clear performance gap across the teacher variants. Distillation data generated by DeepSeek-R1-0528, the strongest teacher in our setup, leads to the highest student performance across benchmarks. In contrast, data from QwQ-32B yields the weakest results. These differences highlight the crucial role of teacher quality in distillation-based SFT.

Stronger teacher models tend to produce more accurate, coherent, and structured responses, which offer richer training signals. This in turn improves the student model’s ability to generalize. Weaker teachers, on the other hand, may generate ambiguous or incomplete answers, introducing noise and degrading downstream performance. This confirms that teacher quality is a bottleneck in distillation, and selecting a capable teacher is essential for effective knowledge transfer.

#### 5.1.3 Data quality and diversity balance

When selecting training data, a common trade-off arises between data quality and diversity. High-quality data often originates from a limited number of sources, which constrains its diversity. Conversely, incorporating data from more diverse sources can degrade overall data quality. In this section, we investigate how to balance quality and diversity in long-chain-of-thought datasets for mathematics and code reasoning tasks. We follow the data quality rankings provided by OpenThoughts ( Guha et al., 2025 ) . Experimental results for math and code tasks are presented in Table 4 and Table 5 , respectively.

From the results, we observe a clear trend: models fine-tuned on only the Top1 or Top2 high-quality sources achieve the best performance. As more data sources are added, performance consistently degrades. This pattern holds across both math and code domains, and aligns with findings reported in the OpenThoughts. We attribute this phenomenon to two primary factors:

• High-quality sources tend to encapsulate the most effective, internally consistent reasoning patterns necessary for solving complex tasks such as mathematics and code generation. With a fixed training budget, models benefit more from focusing on such core reasoning modes.

• The marginal utility of adding lower-quality sources is negative. These sources inevitably introduce noise, such as logical gaps, incorrect derivations, or inefficient problem-solving strategies, which competes for limited model capacity during training. Additionally, heterogeneous sources may exhibit conflicting reasoning styles, further disrupting learning and reducing training efficiency.

In summary, to achieve strong performance in long CoT reasoning, prioritizing data quality and internal consistency is more effective than simply maximizing surface-level diversity. Based on the above findings, our long CoT SFT data is primarily sourced from a few high-quality datesets rather than broad and diverse datasets. Specifically, the mathematical data is mainly derived from OpenThoughts, while the coding data primarily comes from OpenCodeReasoning, TACO, Apps, and Codeforces.

### 5.2 RL Analyses

#### 5.2.1 Effectiveness of Gradient-Preserving Clipping Policy Optimization

In this section, we examine the effects of GPPO on RL training. Figure 2(a) and 2(b) present the evaluation results on AIME2024 and LiveCodeBench V5. Compared to the traditional clipping method ( GRPO w/ Clip-Higher ), our approach achieves superior performance on both benchmarks. We attribute these gains to two key factors: • Better utilization of high-entropy tokens. High-entropy tokens often correspond to critical decision points in the policy. Traditional clipping methods may discard their gradients due to large importance sampling ratios, limiting the model’s exploration capability. Although DAPO’s clip-higher method alleviates this problem, it does not fundamentally solve it, as tokens exceeding the boundary are still directly clipped. In contrast, our method preserves the gradients of these tokens and constrains them within a bounded range, updating model parameters in a gentle manner to ensure the model maintains its exploration capability at all times;

• Accelerated convergence of negative samples. Unlike traditional clipping which cuts off gradients from suboptimal trajectories that fall below the lower clipping bound, our method effectively leverages these negative samples. This prevents the model from repeatedly sampling similar negative trajectories, thereby enhancing its ability for rapid trial and error.

We further observe that CISPO, a concurrent work, achieves notable improvements on AIME2024 over the traditional clipping method, reinforcing the effectiveness of backpropagating gradients from clipped tokens. However, when directly comparing GPPO with CISPO, GPPO not only delivers higher stability in benchmark evaluations but also converges more effectively on both AIME2024 and LiveCodeBench V5. We attribute this advantage to GPPO’s pessimistic update strategy. Due to the min ⁡ ( ⋅ ) \min(\cdot) operation in GPPO, when the advantage A ^ t \hat{A}_{t} is positive, GPPO suppresses overly optimistic updates, an effect that is absent in CISPO’s REINFORCE-based formulation. In contrast, when the advantage is negative, GPPO imposes no constraints on these updates, whereas CISPO limits them. This pessimistic update mechanism helps GPPO maintain clearer optimization signals and leads to more stable policy training.

It is worth noting that, as shown in Figure 2(c) , all three methods maintained stable training, with most gradient norms remaining between 0.05 and 0.08. This indicates that decoupling gradient flow from clipping constraints not only improves learning ability but also preserves robustness advantages.

#### 5.2.2 Impact of SFT loss

In this section, we aim to investigate the impact of different α \alpha values (the hyperparameter in Eq. 14 ). As shown in Table 6 , most cases with α > 0 \alpha>0 achieve better performance than α = 0 \alpha=0 , except for α = 0.05 \alpha=0.05 which shows slight degradation on AIME2024. These results demonstrate that incorporating moderate SFT supervision can effectively guide the policy to produce higher-quality responses, benefiting model performance. The optimal performance is achieved when α \alpha equals 0.1.

We posit that an appropriate α \alpha value enhances the utilization efficiency of positive examples during training. Simultaneously, SFT supervision acts as a policy regularizer, constraining the output distribution to remain within plausible bounds. However, excessive α \alpha disproportionately amplifies gradients from limited positive SFT samples, causing the policy to overfit these specific instances. This leads to localized overfitting and suppresses two critical capacities: negative learning from suboptimal trajectories, and autonomous exploration of novel solution spaces.

#### 5.2.3 Soft Reward vs. Hard Reward

The comparison between soft and hard reward strategies reveals clear advantages for the soft reward approach. As shown in Figures 3(a) and 3(b) , assigning rewards proportional to the test case pass rate consistently yields higher average rewards than the hard reward baseline. This dense supervision allows the model to learn from partially correct outputs, alleviating reward sparsity and providing richer training signals.