On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification — part 3/4

Cleaned-text code points [31181, 47002). Bibliography omitted here; source.md includes it.

Our framework applies confidence-based weighting at the token level. While this design was primarily motivated by numerical stability, we also compared it against two sentence-level variants to better understand their behavior.

The first variant uses the full sequence probability to scale the loss. However, these values are extremely small in practice, making the loss nearly uninformative and producing a highly skewed weight distribution that is difficult to tune. To address this, we also evaluated a geometric-mean variant inspired by GSPO ( Zheng et al., 2025 ) , which rescales sentence probabilities to avoid numerical collapse. Although this version is more stable, it still provides a weak training signal and offers limited performance gains.

As shown in Table 5 , both sentence-level strategies lead to minimal changes over the base model, while our token-level formulation delivers substantial and consistent improvements, raising average accuracy from 15.92 to 31.58. These results demonstrate that token-level weighting provides a more reliable optimization signal and significantly stronger empirical performance.

### 4.7 Analysis of Probabilities

To understand how the model trained by DFT is different from SFT and other RL methods, we look into the token probability distribution of the model’s output over the training set in Figure 2 . SFT tends to uniformly increase token probabilities, shifting the entire distribution towards higher confidence, but mainly targeting the lower and lowest probability tokens. The highest probability token portion barely increases. In stark contrast, DFT exhibits a polarizing effect: it significantly boosts the probabilities of a subset of tokens while actively suppressing the probabilities of others. This leads to a bimodal distribution, with more tokens occupying both the highest and lowest probability bins. Other RL methods such as DPO, GPPO and PPO show the same trend as DFT, although the scale is much milder than it. We look into the words that belong to the lowest probability bin, and find that they are generally the conjunctive words or punctuations such as ‘the’, ‘let’, ‘,’, ‘.’ etc. These results suggest that for robust learning, models should not attempt to fit all tokens with uniform confidence. It may be beneficial to deprioritize fitting tokens that serve grammatical functions rather than carrying primary semantic content. This concept is analogous to human pedagogy, where students are taught to focus on substantive concepts rather than perfecting the usage of common connective words. Further analysis can be found in Appendix A.3 .

## 5 Conclusion

In this work, we revisit the well-known generalization gap between SFT and RL. We offer a theoretical perspective showing that the standard SFT gradient can be interpreted as a policy gradient with an ill-posed, implicitly defined reward inversely related to model confidence. This formulation helps explain the instability and limited generalization observed in SFT training. Motivated by this analysis, we introduce DFT, a simple yet effective method that dynamically reweights the SFT loss using the token probability. This one-line change improves gradient stability and leads to better generalization. Our empirical results show that DFT consistently improves over standard SFT across a range of models and challenging mathematical reasoning tasks. Beyond supervised settings, we adapt DFT to offline RL scenarios and find that it outperforms several established online and offline RL baselines, suggesting broader applicability. Overall, this work contributes both a refined understanding of SFT’s limitations and a lightweight, practical method that helps bridge the gap to more complex RL-based approaches.

#### Limitations.

While our experiments demonstrate the effectiveness of DFT on mathematical reasoning benchmarks and code generation tasks, the evaluation scope remains limited. We have not yet assessed its performance on broader task categories or with larger-scale LLM, which we leave for future exploration. Moreover, DFT can not offer universal benefits across all scenarios. In domains that primarily involve the acquisition of factual knowledge, conventional SFT still remains the most efficient approach. DFT may also not be an ideal choice for hard examples or domains underrepresented in the training data, since it assigns low initial probabilities to such samples, reducing their learning weight. Our aim is not to assert that DFT universally outperforms SFT, but rather to offer a new perspective on objective design by analyzing the distinction between RL and SFT. Besides, an important future direction is to explore non-uniform or quality-aware reward assignments for demonstrations.

## Acknowledgements

Supported by Jiangsu Province Carbon Peak Carbon Neutrality Science and Technology Innovation Special Fund Project (Grant No. BT2025029), National Natural Science Foundation of China (Grant No. 62576091), and Big Data Computing Center of Southeast University.

## Ethics Statement

This work adheres to the ICLR Code of Ethics. Our study does not involve human subjects, personally identifiable information, or proprietary data. All datasets used, including NuminaMath, OpenR1-Math, UltraFeedback, and WeThink, are publicly available and documented in the appendix. The proposed method is a simple training strategy that modifies gradient computation for improved generalization. It does not introduce any new capabilities that could cause harm, nor does it enable misuse beyond the standard capabilities of existing large language models. We are not aware of any potential risks related to bias, fairness, or security that arise specifically from the method proposed. Nonetheless, we acknowledge that like any fine-tuning strategy, DFT may inherit biases present in the underlying data or model, and future research may explore safeguards for these scenarios. No conflicts of interest, legal compliance issues, or sponsorship-related influences are present in this work.

## Reproducibility Statement

We have taken multiple steps to ensure the reproducibility of our work. All datasets used in our experiments are publicly available and properly cited in the main text and appendix. Training configurations, including model architectures, hyperparameters, optimizers, and evaluation settings, are described in detail in Section 4 and Appendi A.5 - A.6 . Theoretical claims, including the equivalence between SFT and policy gradient, are formally derived in Appendix A.2 . Experimental results include multiple model scales, tasks, and training settings to validate robustness. A complete implementation of our method is included in the supplementary material, along with scripts for reproducing all reported results. We will release the full source code and training logs upon publication to further support reproducibility.

## Appendix A Appendix

### A.1 Usage of LLM

We employ LLM primarily as writing assistants to refine and polish the manuscript. Their usage was limited to improving clarity, coherence, and presentation, while all conceptual and experimental contributions remain original.

### A.2 Detailed Derivation of Equation (5)

We start from the SFT gradient in Equation (2): ∇ θ ℒ SFT ​ ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒟 ​ [ − ∇ θ ​ log ​ π θ ​ ( y ⋆ ∣ x ) ] . \nabla_{\theta}\mathcal{L}_{\text{SFT}}(\theta)=\mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\big[-\nabla_{\theta}\log\pi_{\theta}(y^{\star}\mid x)\big]. (1)

For each query x x , the expectation over expert demonstrations ( x , y ⋆ ) (x,y^{\star}) can be written explicitly as a summation over all possible outputs y y : 𝔼 ( x , y ⋆ ) ∼ 𝒟 [ − ∇ θ log π θ ( y ⋆ ∣ x ) ] = 𝔼 x ∼ 𝒟 x ∑ y 𝟏 [ y = y ⋆ ] ] [ − ∇ θ log π θ ( y ∣ x ) ] . \mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\big[-\nabla_{\theta}\log\pi_{\theta}(y^{\star}\mid x)\big]=\mathbb{E}_{x\sim\mathcal{D}_{x}}\sum_{y}\mathbf{1}[y=y^{\star}]]\,\big[-\nabla_{\theta}\log\pi_{\theta}(y\mid x)\big]. (2)

We insert the model distribution π θ ​ ( y ∣ x ) \pi_{\theta}(y\mid x) , which allows us to express the summation in terms of importance weights: 𝔼 x ∼ 𝒟 x ​ ∑ y π θ ​ ( y ∣ x ) ⋅ 𝟏 [ y = y ⋆ ] ] π θ ​ ( y ∣ x ) ​ [ − ∇ θ ​ log ​ π θ ​ ( y ∣ x ) ] . \mathbb{E}_{x\sim\mathcal{D}_{x}}\sum_{y}\pi_{\theta}(y\mid x)\cdot\frac{\mathbf{1}[y=y^{\star}]]}{\pi_{\theta}(y\mid x)}\,\big[-\nabla_{\theta}\log\pi_{\theta}(y\mid x)\big]. (3)

Here, the term 𝟏 [ y = y ⋆ ] ] π θ ​ ( y ∣ x ) \tfrac{\mathbf{1}[y=y^{\star}]]}{\pi_{\theta}(y\mid x)} serves as an importance weight comparing the expert (Dirac delta) distribution with the model’s distribution.

The summation over y y can now be rewritten as an expectation under the policy distribution y ∼ π θ ( ⋅ ∣ x ) y\sim\pi_{\theta}(\cdot\mid x) : 𝔼 x ∼ 𝒟 x 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ 𝟏 [ y = y ⋆ ] ] π θ ​ ( y ∣ x ) ( − ∇ θ log π θ ( y ∣ x ) ) ] . \mathbb{E}_{x\sim\mathcal{D}_{x}}\,\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\frac{\mathbf{1}[y=y^{\star}]]}{\pi_{\theta}(y\mid x)}\big(-\nabla_{\theta}\log\pi_{\theta}(y\mid x)\big)\right]. (4)

Thus, we obtain Equation (5): 𝔼 ( x , y ⋆ ) ∼ 𝒟 [ − ∇ θ log π θ ( y ⋆ ∣ x ) ] = 𝔼 x ∼ 𝒟 x 𝔼 y ∼ π θ ( ⋅ ∣ x ) [ 𝟏 [ y = y ⋆ ] ] π θ ​ ( y ∣ x ) ( − ∇ θ log π θ ( y ∣ x ) ) ] . \mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\big[-\nabla_{\theta}\log\pi_{\theta}(y^{\star}\mid x)\big]=\mathbb{E}_{x\sim\mathcal{D}_{x}}\,\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\left[\frac{\mathbf{1}[y=y^{\star}]]}{\pi_{\theta}(y\mid x)}\big(-\nabla_{\theta}\log\pi_{\theta}(y\mid x)\big)\right]. (5)

This derivation shows that the SFT gradient can be expressed as an on-policy policy gradient with importance sampling, where the expert demonstration distribution is reweighted relative to the model distribution.

### A.3 Discussions and Insights

#### Gradient Analysis of DFT.

We now analyze the gradient induced by the DFT surrogate loss. Recall the sequence-level definition: ℒ DFT ​ ( θ ) = − sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) ​ log ​ π θ ​ ( y ⋆ ∣ x ) , \mathcal{L}_{\text{DFT}}(\theta)=-\,\operatorname{sg}\!\big(\pi_{\theta}(y^{\star}\mid x)\big)\;\log\pi_{\theta}(y^{\star}\mid x), (10) where sg ⁡ ( ⋅ ) \operatorname{sg}(\cdot) denotes the stop-gradient operator. Since the stop-gradient blocks backpropagation, the detached probability sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) \operatorname{sg}(\pi_{\theta}(y^{\star}\mid x)) is treated as a constant during differentiation. Consequently, the gradient becomes ∇ θ ℒ DFT \displaystyle\nabla_{\theta}\mathcal{L}_{\text{DFT}} = − sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) ​ 1 π θ ​ ( y ⋆ ∣ x ) ​ ∇ θ π θ ​ ( y ⋆ ∣ x ) \displaystyle=-\,\operatorname{sg}\!\big(\pi_{\theta}(y^{\star}\mid x)\big)\;\frac{1}{\pi_{\theta}(y^{\star}\mid x)}\,\nabla_{\theta}\pi_{\theta}(y^{\star}\mid x) (11) = − ( sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) π θ ​ ( y ⋆ ∣ x ) ) ​ ∇ θ π θ ​ ( y ⋆ ∣ x ) . \displaystyle=-\,\Bigl(\tfrac{\operatorname{sg}(\pi_{\theta}(y^{\star}\mid x))}{\pi_{\theta}(y^{\star}\mid x)}\Bigr)\,\nabla_{\theta}\pi_{\theta}(y^{\star}\mid x). (12)

Since sg ⁡ ( π θ ​ ( y ⋆ ∣ x ) ) \operatorname{sg}(\pi_{\theta}(y^{\star}\mid x)) equals π θ ​ ( y ⋆ ∣ x ) \pi_{\theta}(y^{\star}\mid x) in the forward pass, the prefactor is numerically equal to 1 1 . Therefore, ∇ θ ℒ DFT = − ∇ θ π θ ​ ( y ⋆ ∣ x ) . \nabla_{\theta}\mathcal{L}_{\text{DFT}}=-\,\nabla_{\theta}\pi_{\theta}(y^{\star}\mid x). (13) This shows that DFT is mathematically equivalent to directly maximizing the model probability of the target token, rather than its log-probability as in cross-entropy.

For cross-entropy, the loss is ℒ CE ​ ( θ ) = − log ⁡ π θ ​ ( y ⋆ ∣ x ) , \mathcal{L}_{\text{CE}}(\theta)=-\log\pi_{\theta}(y^{\star}\mid x), yielding gradient ∇ θ ℒ CE = − 1 π θ ​ ( y ⋆ ∣ x ) ​ ∇ θ π θ ​ ( y ⋆ ∣ x ) . \nabla_{\theta}\mathcal{L}_{\text{CE}}=-\tfrac{1}{\pi_{\theta}(y^{\star}\mid x)}\nabla_{\theta}\pi_{\theta}(y^{\star}\mid x). Thus both CE and DFT share the same gradient direction but differ in scaling: CE amplifies updates for low-probability targets (factor 1 / π 1/\pi ), while DFT applies a uniform factor 1 1 . As a result, DFT avoids the instability caused by excessively large gradients on unlikely expert tokens, providing more conservative and stable updates.

From the reinforcement learning perspective, the reward of DFT becomes uniformly 1 1 across all expert trajectories, equivalent to a verification-style objective that treats all correct references equally. From the optimization perspective, DFT trades off aggressive fitting of rare tokens for better stability and calibration. Practically, this explains why DFT often yields smoother training and stronger generalization, while maintaining alignment with the pre-training distribution.

#### Learning from Noisy Data.

DFT offers a simple yet effective approach, prompting us to reflect on why it might actually work. One intuitive explanation lies in its ability to learn from noisy data ( Freund, 2009 ) . Sasaki and Yamashina (2020) propose an imitation learning algorithm for learning from noisy demonstrations, based on the core idea of avoiding the fitting of data that is difficult to model, as such data is likely to originate from suboptimal behaviors, i.e., noise. Their method introduces a weighted behavioral cloning objective, where the weights are derived from a previously trained policy’s confidence in each action. Similarly, the weighting mechanism in DFT shares the same intuition, but instead of relying on a fixed old policy model to compute confidence scores, it uses a single policy model to perform confidence-based weighting on-the-fly during training.

### A.4 Comparision with Concurrent work iw-SFT

We include a concurrent method, Importance-Weighted SFT (iw-SFT) ( Qin and Springenberg, 2025 ) , for comparison. All training settings follow those reported in the original paper, except that we set the number of training epochs to 1.

As shown in Table 6 , DFT achieves higher average accuracy than iw-SFT on most model families: LLaMA-3.2-3B (+2.39), LLaMA-3.1-8B (+4.15), DeepSeekMath-7B (+3.34), and Qwen2.5-Math-1.5B (+1.30). Although iw-SFT outperforms our method on Qwen2.5-Math-7B (+2.45), this improvement is not consistent across datasets. In particular, for LLaMA-3.2-3B, iw-SFT underperforms standard SFT on Math500 (5.13 vs. 8.65) and AMC23 (2.03 vs. 3.13). Similarly, for LLaMA-3.1-8B, iw-SFT results in worse performance than SFT on Minerva Math (4.31 vs. 5.78) and AMC23 (7.34 vs. 8.28). In contrast, DFT consistently improves upon both the base model and SFT across nearly all datasets, including those where iw-SFT fails. These results underline better generalization ability of DFT in diverse mathematical reasoning scenarios. Moreover, iw-SFT incurs additional computational overhead by requiring a separate reference model to compute importance weights, whereas DFT dynamically derives its own weighting directly from the token probabilities of model, resulting in a more efficient training procedure.

We also compare against iw-SFT under the offline setting, as shown in Table 7 . While iw-SFT performs competitively on certain datasets, achieving 60.80 on Math500 and 44.21 on AMC23, its overall average performance (31.86) remains below that of our method by +3.57 points. Moreover, iw-SFT shows only modest improvements compared to its standard SFT counterpart, with an average score of 31.86 in the offline RL setting versus 30.28 with SFT (+1.58). In contrast, DFT achieves a larger gain of +4.76 (from 30.67 to 35.43). These results indicate that iw-SFT provides limited benefits from reward supervision under offline constraints, whereas DFT is able to more effectively incorporate such signals, leading to better generalization and higher task performance.

### A.5 Exploratory Experiment - OpenR1-Math Training Dataset