On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification — part 4/4

Cleaned-text code points [47002, 51633). Bibliography omitted here; source.md includes it.

Inspired by DeepSeek-R1 DeepSeek-AI et al. (2025) , several studies have attempted to train open-source models to reproduce its reasoning capabilities ( Hugging Face, 2025 ) . To this end, a high-quality dataset, OpenR1-Math-220k ( Hugging Face, 2025 ) , was constructed, where the prompts are drawn from NuminaMath 1.5 and the off-policy reasoning traces are generated by DeepSeek-R1. LUFFY ( Yan et al., 2025 ) further filtered out sequences longer than 8192 tokens as well as those verified incorrect by Math-Verify, resulting in about 45k prompts paired with off-policy reasoning traces. We adopt this dataset as the training corpus for SFT. All training details remain the same as previous experiments, except that the number of epochs is set to 3.

As shown in Table 8 , training on OpenR1-Math-220k consistently improves performance, and the use of higher-quality annotations yields additional gains. SFT on this dataset increases the average accuracy of Qwen2.5-Math-1.5B by +13.24 points compared to the base model, while DFT provides a further +9.03 gain, resulting in a total improvement of +22.27. These results suggest that DFT remains effective even when applied on top of high-quality training data, highlighting its potential as a general fine-tuning paradigm.

### A.6 Exploratory Experiment - PEFT Training Setting

To investigate whether DFT remains effective under parameter-efficient fine-tuning (PEFT) settings with limited compute, we apply DFT using LoRA adapters across two model families: LLaMA-3.2-3B and Qwen2.5-Math-1.5B. All training configurations remain identical to previous full-parameter experiments, except that LoRA is enabled with rank=8 and alpha=16.

As shown in Table 9 , DFT provides consistent improvements over both base and SFT baselines under LoRA-based PEFT. For Qwen2.5-Math-1.5B, DFT increases the average accuracy from 15.92 (base) and 16.87 (SFT) to 32.90. For LLaMA-3.2-3B, DFT achieves a gain of +3.44 over SFT (from 1.19 to 4.63). These results indicate that DFT can serve as an effective fine-tuning strategy in low-resource or compute-constrained settings, where full model updates are not practical.

### A.7 Training Hyper-Parameters Ablation

To assess the robustness and sensitivity of our approach (DFT) with respect to key training hyper-parameters, we conduct an ablation study focused on learning rate and batch size, using the Qwen2.5-Math-1.5B base model. This analysis aims to answer two central questions: (1) Is the performance gap between DFT and SFT due to a suboptimal hyperparameter configuration in SFT? (2) How sensitive are both methods to changes in learning rate and batch size?

We evaluate both DFT and SFT across four learning rates: 2e-4, 1e-4, 5e-5, and 1e-5. As shown in Figure 3 (left), both methods exhibit a certain degree of sensitivity to the learning rate. DFT consistently outperforms SFT under all configurations, suggesting that the performance gap cannot be attributed solely to suboptimal hyperparameter choices in SFT. For both methods, intermediate learning rates (1e-4 and 5e-5) yield the best results, while both lower (1e-5) and higher (2e-4) values lead to noticeable degradation.

We further assess the impact of batch size, sweeping values from 32 to 256. As shown in Figure 3 (right), both DFT and SFT exhibit relatively stable performance across the full range of batch sizes. While minor fluctuations are observed, there is no consistent trend indicating that larger or smaller batches significantly affect final accuracy. This suggests that batch size is not a dominant factor for either method in this setup, and that default values may suffice in practice.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .