On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification — part 2/4

Cleaned-text code points [15312, 31181). Bibliography omitted here; source.md includes it.

Due to the inherently sparse reward signal in the SFT setting, we identify the importance weight 1 / π θ 1/\pi_{\theta} as a key contributor to SFT’s generalization limitations compared to RL. When the model assigns low probability to the expert response, the resulting weight becomes excessively large, introducing an ill-posed reward landscape. This leads to disproportionately large gradients and training instability. The issue is compounded by the fact that the reward function r ( x , y ) = 𝟏 [ y = y ⋆ ] r(x,y)=\mathbf{1}[y=y^{\star}] is non-zero only for exact matches to the expert outputm causing optimization to overfit rare exact-match samples and weakening the model’s ability to generalize beyond the training data.

### 3.3 Proposed Method

#### Reward Rectification via Dynamic Reweighting.

To neutralize the skewed reward issue identified when viewing SFT under the RL objective, we dynamically reweight the reward by multiplying by a corrective inverse ratio given by the policy probability 1 / w 1/w . The resulting “dynamically fine-tuned” gradient is then

∇ θ ℒ SFT ( θ ) = − 𝔼 x ∼ 𝒟 x , y ∼ π θ ( ⋅ ∣ x ) [ sg ( 1 w ) ⋅ w ( y ∣ x ) ∇ θ log π θ ( y ∣ x ) r ( x , y ) ] . \nabla_{\theta}\mathcal{L}_{\mathrm{SFT}}(\theta)=-\mathbb{E}_{x\sim\mathcal{D}_{x},\;y\sim\pi_{\theta}(\cdot\mid x)}\bigl[\operatorname{sg}(\frac{1}{w})\cdot{\color[rgb]{0,0,1}w(y\mid x)}\,\nabla_{\theta}\log\pi_{\theta}(y\mid x)\,{\color[rgb]{0,0,1}r(x,y)}\bigr]. (7)

where sg ⁡ ( ⋅ ) \operatorname{sg}(\cdot) denotes the stop gradient operator, ensuring that gradients do not flow through the reward scaling term w w . To facilitate transition to later equations, we directly write 1 / w 1/w to be π θ ​ ( y ⋆ ∣ x ) \pi_{\theta}(y^{\star}\mid x) instead of π θ ​ ( y ∣ x ) \pi_{\theta}(y\mid x) because the indicator function in equation 5 or equation 6 would leave all cases where y ≠ y ⋆ y\neq y^{\star} is 0. Now since the gradient does not flow, the corrected SFT loss also becomes a simple reweighted loss, called Dynamic Fine-tuning (DFT).

ℒ DFT ​ ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒟 ​ [ − sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) ​ log ​ π θ ​ ( y ⋆ ∣ x ) ] . \mathcal{L}_{\text{DFT}}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\Bigl[-\operatorname{sg}\big(\pi_{\theta}(y^{\star}\mid x)\big)\log\pi_{\theta}(y^{\star}\mid x)\Bigr]. (8)

However, in practice, computing importance weights over the entire trajectory can induce numerical instability. A common treatment of this issue is to simply apply importance sampling at the token level, as was adopted in PPO ( Schulman et al., 2017 ) . This leads to the final DFT loss version:

ℒ DFT ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒟 [ − ∑ t = 1 | y ⋆ | sg ( π θ ( y t ⋆ ∣ y < t ⋆ , x ) ) log π θ ( y t ⋆ ∣ y < t ⋆ , x ) ] . \mathcal{L}_{\text{DFT}}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\Bigl[-\!\sum_{t=1}^{|y^{\star}|}\operatorname{sg}\big(\pi_{\theta}(y^{\star}_{t}\mid y^{\star}_{<t},x)\big)\log\pi_{\theta}(y^{\star}_{t}\mid y^{\star}_{<t},x)\Bigr]. (9)

Note that the reward of this corrected SFT (in RL form), i.e., DFT, now becomes 1 uniformly for all expert trajectory. This is akin to contemporary verification based reward approach RLVR ( DeepSeek-AI et al., 2025 ) that assigns uniform reward to all correct samples. Consequently, it avoids over-concentration on specific low-probability reference tokens, leading to more stable updates and improved generalization without introducing any additional sampling or reward models.

## 4 Experiments

We design four groups of experiments to comprehensively evaluate DFT. We first study the standard SFT setting on mathematical reasoning tasks to establish its core advantage over SFT (Section 4.1 ). We then extend to an offline RL setting, comparing DFT with representative offline and online RL methods (Section 4.2 ). To test cross-domain robustness, we further examine DFT on code generation benchmarks (Section 4.3 ) and its applicability to multi-modal reasoning math datasets (Section 4.4 ).

### 4.1 Main Experiment - Mathematical Reasoning Task

To examine whether DFT can outperform vanilla SFT across tasks, architectures, and scales, we use mathematical reasoning as a representative testbed.

#### Implementation details.

To efficiently manage computational resources, We andomly sample 100,000 instances from the the NuminaMath-CoT dataset ( LI et al., 2024 ) for training. We conduct experiments using multiple models, including Qwen2.5-Math-1.5B, Qwen2.5-Math-7B ( Qwen Team et al., 2024b ) , LLaMA-3.2-3B, LLaMA-3.1-8B ( Dubey et al., 2024 ) , and DeepSeekMath-7B ( Shao et al., 2024 ) . Our implementation builds upon the verl framework ( Sheng et al., 2025 ) , using recommended SFT hyper-parameters. Specifically, we employ the AdamW optimizer with learning rates of 5 × 10 − 5 5\times 10^{-5} for all models except the LLaMA-3.1-8B-Base, for which we adopt a lower learning rate of 2 × 10 − 5 2\times 10^{-5} . We set the mini-batch size to 256 and the maximum input length to 2048 tokens. The learning rate follows a cosine decay schedule with a warm-up ratio of 0.1. We evaluate on benchmarks including Math500 ( Hendrycks et al., 2021 ) , Minerva Math ( Lewkowycz et al., 2022 ) , Olympiad Bench ( He et al., 2024 ) , AIME 2024 ( American Institute of Mathematics, 2024 ) , and AMC 2023 ( Mathematical Association of America, 2023 ) through the official Qwen2.5-Math evaluation pipeline ( Qwen Team et al., 2024b ) . Each model uses the default chat template and Chain-of-Thought (CoT) prompting to stimulate step-by-step reasoning. All reported results represent average accuracy across 16 decoding runs, evaluated with a temperature of 1.0 and maximum generation length of 4096 tokens.

DFT consistently yields average performance improvements over base models compared to standard SFT across all benchmarks. Table 1 shows that, for Qwen2.5-Math-1.5B, DFT achieves an average gain of +15.66 points over the base model, which is over 5.9 × \times larger than the +2.09 point improvement from SFT. This pattern generalizes across other model families and sizes: LLaMA-3.2-3B benefits from a +3.46 point gain with DFT, exceeding the SFT gain (+2.05) by approximately 1.4 × \times ; LLaMA-3.1-8B achieves +10.02 from DFT, surpassing SFT’s +5.33 by 1.88 × \times ; DeepSeekMath-7B sees a +15.51 point improvement via DFT, which is 1.58 × \times larger than SFT’s +7.18; and Qwen2.5-Math-7B reaches a +15.90 point gain, nearly 3.8 × \times higher than the SFT improvement of +2.37.

DFT demonstrates generalization and robustness, especially on challenging benchmarks where standard SFT yields minimal or even negative impact. For instance, on Olympiad Bench, SFT degrades performance for Qwen2.5-Math-1.5B, dropping accuracy from 15.88 to 12.63, while DFT boosts it to 27.08, +11.20 point improvement over base model. On AIME24, SFT reduces accuracy for Qwen2.5-Math-7B by 4.20 points (from 6.68 to 2.48), whereas DFT improves performance to 8.56, achieving a +1.88 point gain over the base model despite the difficulty of the benchmark. A similar trend is observed on AMC23. SFT reduces the performance of Qwen2.5-Math-1.5B from 19.38 to 18.75, while DFT raises it to 38.13, a +18.75 point gain over base. For Qwen2.5-Math-7B, SFT yields only a marginal improvement (+1.86), whereas DFT achieves a +17.04 point gain. These results underscore that DFT not only scales more effectively across models of varying capacities, but also exhibits better resilience on difficult reasoning tasks where traditional SFT struggles.

DFT exhibits better learning efficiency and faster convergence characteristics. Figure 1 reveals clear differences in learning dynamics between DFT and standard SFT on Qwen2.5-Math-1.5B across all math reasoning benchmarks. Compared to SFT, our method demonstrates three distinct advantages: (1) Faster convergence, achieving peak performance within the first 120 training steps on most benchmarks; (2) Better early-stage performance, with DFT already outperforming best final accuracy of SFT within the first 10–20 steps; and (3) Higher sample efficiency, consistently requiring fewer updates to reach relatively optimal results. This accelerated convergence shows that the dynamic reweighting mechanism in DFT leads to more informative gradient updates, guiding the model toward high-quality solutions early in training. It also suggests that DFT helps avoid the optimization plateaus or noise-prone regions often encountered in standard SFT, thereby enabling more efficient acquisition of complex mathematical reasoning patterns.

We also report the results of parameter-efficient fine-tuning (PEFT) training setting ( Hu et al., 2022 ) and training on the OpenR1-Math dataset ( Hugging Face, 2025 ) with better quality in Appendix A.6 and Appendix A.5 , respectively. Comparison and Discussion with the concurrent method iw-SFT ( Qin and Springenberg, 2025 ) is provided in Appendix A.4 .

### 4.2 Exploratory Experiment - Offline RL Setting

Equation 7 shows that SFT suffers from reward sparsity, since in a constructed dataset each query x x has only a single reference answer y ⋆ y^{\star} . From the perspective of RL, RFT/RAFT ( Dong et al., 2023 ; Ahn et al., 2024 ) can be viewed as alleviating the sparse reward issue by effectively increasing reward density, thereby enhancing model performance. Motivated by this observation, we conduct an exploratory study applying DFT in an offline RL setting, where the reward sparsity problem is inherently less severe compared to standard SFT, to further validate the effectiveness.

#### Implementation details.

We sample responses for 100,000 math questions using a temperature of 1.0 and generate four responses per question from the base model itself. Correct responses are identified using math verify and retained as training data, resulting in approximately 140,000 examples. For DPO training, we construct 100,000 positive–negative preference pairs from the generated responses. We compare DFT with representative offline RL methods, including DPO ( Rafailov et al., 2023 ) and RFT ( Dong et al., 2023 ; Ahn et al., 2024 ) , as well as online RL methods PPO ( Schulman et al., 2017 ) and GRPO ( Shao et al., 2024 ) . For RFT and DFT, the training setup follows the configuration in Section 4.1 . For DPO, we use the ms-swift ( Zhao et al., 2024 ) with a learning rate of 1 × 10 − 6 1\times 10^{-6} , batch size of 128, and a warmup ratio of 0.05. For PPO and GRPO, training is performed using the verl ( Sheng et al., 2025 ) with a learning rate of 1 × 10 − 6 1\times 10^{-6} , batch size of 256, and a warmup ratio of 0.1. We set the number of response n = 4 n=4 for GRPO.

DFT demonstrates competitive performance in the offline RL setting, outperforming both offline and online RL baselines. Table 2 shows DFT achieves an average score of 35.43, exceeding the best offline method RFT by +11.46 points, and even outperforming the strongest online RL algorithm GRPO by +3.43 points. Specially, on Math500, DFT scores 64.71, slightly ahead of GRPO (62.86) and better than PPO (56.10) and RFT (48.23). The gains are also notable on more challenging benchmarks: on AMC23, DFT achieves 48.44, a +7.19 point margin over GRPO and a +17.66 point gain over RFT. Similarly, on Minerva Math, DFT reaches 25.16, outperforming GRPO by +6.23 points, PPO by +9.75, and all offline baseline methods.

These results highlight the strength of DFT as a simple yet effective fine-tuning strategy. Despite its lack of iterative reward modeling or environment interaction, it provides a stronger learning signal than both offline methods like DPO/RFT and online policy optimization algorithms like PPO/GRPO in certain scale train set. This suggests that DFT can serve as a more efficient and scalable alternative to traditional RL pipelines, particularly in domains where preference supervision is available but reward modeling or online response sampling is expensive or impractical.

### 4.3 Exploratory Experiment - Code Generation Task

#### Implementation details.

We adopt UltraFeedback ( Cui et al., 2024 ) as the training dataset. From this corpus, we sample 10,000 prompts and, for each prompt, select the response with the highest average score to perform supervised fine-tuning (SFT) ( Du and others, 2025 ) . Model performance is assessed on three widely used code generation benchmarks: HumanEval ( Chen et al., 2021 ) , HumanEval+ ( Liu et al., 2023 ) , and MultiPL-E ( Cassano et al., 2023 ) . Training is conducted for one epoch with a learning rate of 5 × 10 − 5 5\times 10^{-5} , a warm-up ratio of 0.05, and a batch size of 16.

Table 3 shows DFT achieves improvements in most cases compared to both base models and SFT. For Qwen2.5-3B, DFT raises HumanEval from 43.3 to 45.7 and HumanEval+ from 36.0 to 39.0, with the MultiPL-E average also increasing from 40.05 (base) and 39.10 (SFT) to 41.84. Similar trends are observed for Qwen2.5-Coder-3B, where DFT improves HumanEval to 56.7 and HumanEval+ to 50.0, outperforming both base and SFT. For Qwen2.5-Coder-7B, DFT reaches 67.7 on HumanEval, 59.8 on HumanEval+, and 62.3 average on MultiPL-E, surpassing SFT by +12.8, +11.0, and +4.7 points respectively. The overall trend demonstrates that DFT generally provides stronger performance across different models and languages.

### 4.4 Exploratory Experiment - Multi-Modal Reasoning

#### Implementation details.

We use the WeThink dataset ( Yang et al., 2025 ) for training. The model is fine-tuned using LLaMA-Factory ( Zheng et al., 2024 ) and evaluated with VLMEvalKit ( Duan et al., 2024 ) . We train the model for 1 epoch with a learning rate of 5e-5. To comprehensively assess reasoning capabilities, we adopt a suite of multi-modal reasoning benchmarks including MathVerse ( Zhang et al., 2024b ) , MathVision ( Wang et al., 2024 ) , and WeMath ( Qiao et al., 2024 ) for evaluation.

DFT achieves consistent improvements over base models and SFT across all multi-modal reasoning benchmarks. Table 4 shows, on MathVerse, DFT boosts Qwen2.5-VL-3B from 33.83 to 37.54 average accuracy, outperforming the SFT gain of only +1.83 by +3.71 points. Consistent improvements are observed across all major vision-related subcategories. On MathVision, DFT improves performance from 21.25 (base) to 22.30, exceeding SFT which fails to provide gains (21.02). On WeMath, SFT already yields a +19.23 point gain, but DFT pushes performance slightly further to 23.71, maintaining superiority over both base and SFT. These results indicate that DFT not only strengthens text-only reasoning but also extends effectively to multi-modal domains.

### 4.5 Limitations of DFT: A Case Study on Factual Knowledge

While DFT consistently outperforms SFT on reasoning-heavy tasks, it may not always be the better choice, particularly in factual knowledge domains. We conduct an exploratory experiment on the Natural Questions dataset ( Kwiatkowski et al., 2019 ) , which consists of real-user, open-domain factual queries grounded in Wikipedia articles.

In this setting, we find that SFT improves performance from 31.24% to 36.62%, while DFT unexpectedly reduces it to 30.14%. This result reveals an important limitation of DFT: because it reweights samples based on the model’s own confidence, it tends to reinforce the model’s existing beliefs. When the model lacks sufficient factual knowledge, such reinforcement may hinder effective learning instead of facilitating it.

This case suggests that DFT is most effective when the task aligns well with the model’s prior competence, such as logical reasoning or structured prediction. In contrast, when the objective is to absorb new factual information, especially in domains beyond the model’s current capabilities, SFT remains a more reliable and stable fine-tuning strategy.

### 4.6 An Empirical Comparison with Sentence-Level Weighting