On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification — part 1/4

Cleaned-text code points [0, 15312). Bibliography omitted here; source.md includes it.
# On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

###### Abstract

In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, DFT achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT .

## 1 Introduction

Supervised Fine-Tuning (SFT), which adapts models to expert demonstrations, has become the standard post-training paradigm for Large Language Models (LLMs) ( Zhang et al., 2025c ; Zhang et al., 2024a ; Zhang et al., 2025b ; Zhang et al., 2025a ; Wang et al., 2025b ) . It enables efficient task adaptation and capability enhancement ( Chung et al., 2024 ; Zhang et al., 2024c ; Sanh et al., 2022 ; Ouyang et al., 2022 ; Chen et al., 2024 ; Chen et al., 2025a ; Fang et al., 2025 ) , and is popular for its ease of implementation and rapid acquisition of expert-like behaviors ( Wei et al., 2022 ; Zhou et al., 2023 ) . Despite these advantages, SFT often shows limited generalization compared to reinforcement learning (RL) ( Chu et al., 2024 ; Ouyang et al., 2022 ; Christiano et al., 2017 ; Bai et al., 2022 ; Huan et al., 2025 ; Swamy et al., 2025 ) . RL leverages explicit reward or verification signals to explore diverse strategies and thus generalizes better. However, RL requires substantial computation, careful hyperparameter tuning, and explicit reward signals—conditions often impractical in real-world settings ( Schulman et al., 2017 ; Ouyang et al., 2022 ; Sheng et al., 2025 ; Strubell et al., 2019 ; Liu and Yin, 2024 ; Winsta, 2025 ) . Moreover, RL can struggle to recover expert-like behaviors that SFT captures efficiently ( Mandlekar et al., 2022 ; Chen et al., 2025c ) .

To exploit the complementary strengths of both approaches, many hybrid methods combine SFT with RL ( Ouyang et al., 2022 ; Sheng et al., 2025 ; Rafailov et al., 2023 ; Liu et al., 2025 ; Qiu et al., 2025 ) . Yet a key question remains: can SFT itself be fundamentally improved? This is crucial, as SFT remains the only viable option when datasets contain only positive demonstrations, with no negative samples or reward model available.

In this work, we address this gap with a mathematical analysis of the connection between SFT and RL. We show that the gradient update in SFT can be interpreted as a form of policy gradient with a specific, implicitly defined reward under certain assumptions. Crucially, this reward is (i) sparse, and (ii) inversely proportional to the model’s probability of expert actions (see equation 6 ). As a result, when the model assigns low probability to expert actions, the gradient becomes excessively large, yielding an ill-posed reward structure and unstable optimization ( Pascanu et al., 2013 ; Yang et al., 2019 ) .

Building on this insight, we propose Dynamic Fine-Tuning (DFT), a principled fix. Our method rescales the SFT objective at each token by its probability, canceling the distortion introduced by inverse-probability weighting. This reframing turns the SFT gradient from a potentially unstable and biased estimator into a more stable, more uniformly weighted update rule that behaves closer to an RL-style.

Empirically, DFT delivers substantial improvements. On the Qwen-2.5-Math series ( Qwen Team et al., 2024a ) fine-tuned with NuminaMath-CoT ( LI et al., 2024 ) , DFT yields gains several times larger than standard SFT. More importantly, unlike SFT, which often degrades on challenging benchmarks such as OlympiadBench ( He et al., 2024 ) , AIME 2024 ( American Institute of Mathematics, 2024 ) , and AMC 2023 ( Mathematical Association of America, 2023 ) , our method consistently improves performance and generalization. These improvements hold across models, scales, and data sizes (Table 1 , Figure 1 ), and extend to code generation and multimodal reasoning (Tables 3 , 4 ) ( Zhao et al., 2025d ; Luo et al., 2025 ; Li et al., 2025a ; Li et al., 2025b ) ( Zhu et al., 2024b ; Zhu et al., 2024c ; Zhu et al., 2024a ; Zhu et al., 2020 ) .

We further test DFT in off-policy RL settings (Table 2 ), where dense rewards are available ( Levine et al., 2020 ) . Our method not only outperforms offline RL approaches such as DPO ( Rafailov et al., 2023 ) and RAFT ( Dong et al., 2023 ; Ahn et al., 2024 ) , but also achieves competitive or superior performance to online methods like GRPO and PPO on math tasks with Qwen2.5-Math-1.5B ( He et al., 2025 ; Tan et al., 2026 ; Zhao et al., 2025b ; Zhao et al., 2025a ) . Unlike these RL methods, DFT requires neither a reference model nor large batch sizes, making it a simpler and more resource-efficient alternative.

To understand its effect, we analyze token probability distributions after training (Figure 2 ). While traditional SFT uniformly pushes probabilities toward the training set, DFT selectively increases some while reducing others. In particular, the proportion of less strongly fitted tokens rises, suggesting improved regularization. We provide further discussion in Appendix A.3 .

The contributions of this work are theoretical and practical. On the theoretical side, we mathematically establish LLM SFT as a special RL in policy gradient space, pinpoint the underlying reasons for the limited generalization of SFT, and derive a method to improve it. On the experimental side, we show that such a simple solution, just one line of code, can enhance the performance and generalization capabilities of SFT across various tasks and models.

## 2 Related Work

The trade-off between supervised fine-tuning (SFT) and reinforcement learning (RL) is central to the alignment of large language models ( Song et al., 2024 ; Chai et al., 2024 ; Song et al., 2025c ; Song et al., 2025a ; Xu et al., 2025 ; Song et al., 2025b ; Zhu et al., 2025b ; Zhu et al., 2025c ; Zhu et al., 2025a ) . SFT is widely adopted due to its simplicity and efficiency in imitating expert demonstrations ( Chung et al., 2024 ; Zhou et al., 2023 ; Wei et al., 2022 ) , analogous to behavioral cloning in robotics ( Sammut, 2011 ; Mandlekar et al., 2022 ) . However, the literature consistently highlights its limitations, particularly the tendency to overfit and generalize poorly compared to RL, which leverages reward signals to discover more robust policies ( Ouyang et al., 2022 ; Christiano et al., 2017 ; Bai et al., 2022 ; Swamy et al., 2025 ; Zhang et al., 2025d ) . A recent systematic comparison by Chu et al. (2024) across textual and visual domains confirms this distinction, concisely summarized as “SFT memorizes while RL generalizes.” They further show that SFT remains indispensable as an initialization step, stabilizing output formatting prior to effective RL training. Nonetheless, RL faces significant practical hurdles, including computational expense, sensitivity to hyperparameters, and the requirement of an explicit reward function, all of which constrain its applicability ( Schulman et al., 2017 ; Strubell et al., 2019 ; Sheng et al., 2025 ) .

To combine the strengths of both paradigms, much recent work has pursued hybrid approaches. The most common strategy involves SFT pretraining followed by RL-based refinement with a learned reward model, as popularized by InstructGPT ( Ouyang et al., 2022 ) . More recent methods interleave SFT and RL updates to improve stability and performance ( Sheng et al., 2025 ; Liu et al., 2025 ; Qiu et al., 2025 ) . Other approaches, such as Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) , bypass reward modeling entirely by directly optimizing policies on preference data, thereby unifying imitation and reinforcement signals within a single loss function. Chen et al. (2025b) introduce Negative-aware Fine-Tuning (NFT), which models incorrect generations via an implicit negative policy, enabling self-improvement without explicit feedback. While powerful, these methods rely on reward signals, preference pairs, or negative samples. They enrich the training pipeline but do not fundamentally improve SFT in its native setting, where only positive demonstrations are available. Our work instead focuses on enhancing SFT itself without requiring external feedback.

A complementary line of theoretical research seeks to unify SFT and RL under a common formalism. Du and others (2025) reinterpret RLHF as a reward-weighted variant of SFT, preserving reliance on an explicit reward. Wang et al. (2025a) show that SFT can be cast as RL with an implicit reward, proposing adjustments such as smaller learning rates to manage the vanishing KL constraint. Abdolmaleki et al. (2025) analyze learning from both positive and negative feedback, studying how their balance affects convergence. Qin and Springenberg (2025) view SFT as a lower bound of RL and introduce importance weighting based on the data-generating policy. While these works establish connections between SFT and RL through weighting, they do not provide a precise mathematical equivalence between the SFT gradient and the offline policy gradient. Some methods approximate this connection in practice by reweighting training losses. For instance, MixCE ( Zhang et al., 2023 ) combines the forward and reverse KL divergences to form a unified objective, while GOLD ( Pang and He, 2021 ) adopts offline RL with demonstrations, introducing reliance on an unknown demonstration distribution π b \pi_{b} and a restrictive 1 / N 1/N assumption. Kantharaju and Sankar (2022) also provide a clear and insightful exposition of GOLD’s motivation and mechanics from an alternative perspective, offering useful intuition for understanding its underlying design. Zhao et al. (2025c) provide a promising method to combine RL and SFT during training. In contrast, our work offers a more formal perspective on this connection, highlighting the role of the inverse-probability weighting term in shaping the difference between SFT and RL-like updates. This perspective motivates a simple adjustment: multiplying the loss by the model’s token probability to neutralize the weighting.

Interestingly, our method modifies the standard cross-entropy (CE) loss in a way that inverts the weighting philosophy of the widely used Focal Loss ( Lin et al., 2017 ) . Specifically, our modified CE takes the form − p  log ⁡ ( p ) -p\log(p) , whereas focal loss is defined as − ( 1 − p ) γ  log ⁡ ( p ) -(1-p)^{\gamma}\log(p) . Focal Loss deliberately downweights well-classified samples to emphasize underrepresented or hard cases, whereas we deliberately downweight poorly classified samples to encourage generalization. This inversion reflects a fundamental shift in the LLM era: while underfitting was once a central challenge, overfitting and memorization now dominate, demanding a rethinking of objective design.

## 3 Method

### 3.1 Preliminaries

#### Supervised Fine-Tuning.

Let 𝒟 = { ( x , y ⋆ ) } \mathcal{D}=\{(x,y^{\star})\} denote a corpus of expert demonstrations, where y ⋆ y^{\star} is the complete reference response to the query x x . SFT minimizes the sentence-level cross-entropy: ℒ SFT  ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒟  [ − log ⁡ π θ  ( y ⋆ ∣ x ) ] . \mathcal{L}_{\mathrm{SFT}}(\theta)\;=\;\mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\bigl[-\log\pi_{\theta}\bigl(y^{\star}\mid x\bigr)\bigr]. (1) Its gradient is: ∇ θ ℒ SFT  ( θ ) = 𝔼 ( x , y ⋆ ) ∼ 𝒟  [ − ∇ θ  log  π θ  ( y ⋆ ∣ x ) ] . \nabla_{\theta}\mathcal{L}_{\mathrm{SFT}}(\theta)\;=\;\mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\bigl[-\nabla_{\theta}\log\pi_{\theta}\bigl(y^{\star}\mid x\bigr)\bigr]. (2)

#### Reinforcement Learning.

Let y y denote a response sampled from the policy π θ ( ⋅ ∣ x ) \pi_{\theta}(\cdot\mid x) for query x x . Given a reward function r ⁡ ( x , y ) ∈ ℝ r(x,y)\in\mathbb{R} , the policy objective is J ( θ ) = 𝔼 x ∼ 𝒟 x , y ∼ π θ ( ⋅ ∣ x ) [ r ( x , y ) ] . J(\theta)\;=\;\mathbb{E}_{x\sim\mathcal{D}_{x},\;y\sim\pi_{\theta}(\cdot\mid x)}\bigl[r(x,y)\bigr]. (3) Its policy gradient at the sentence level is ∇ θ J ( θ ) = 𝔼 x ∼ 𝒟 x , y ∼ π θ ( ⋅ ∣ x ) [ ∇ θ log π θ ( y ∣ x ) r ( x , y ) ] . \nabla_{\theta}J(\theta)\;=\;\mathbb{E}_{x\sim\mathcal{D}_{x},\;y\sim\pi_{\theta}(\cdot\mid x)}\bigl[\nabla_{\theta}\log\pi_{\theta}(y\mid x)\;r(x,y)\bigr]. (4)

### 3.2 Unify SFT and RL Gradient Expression

#### Rewriting SFT Gradient as Policy Gradient via Importance Sampling.

The SFT gradient in equation 2 is taken under the fixed demonstration distribution. We convert it to an on-policy expectation by inserting an importance weight that compares the expert (Dirac Delta) distribution with the model distribution. 𝔼 ( x , y ⋆ ) ∼ 𝒟  [ − ∇ θ  log  π θ  ( y ⋆ ∣ x ) ] = 𝔼 x ∼ 𝒟 x  𝔼 y ∼ π θ ( ⋅ ∣ x ) 𝟏 [ y = y ⋆ ] π θ  ( y ∣ x ) [ − ∇ θ log π θ ( y ∣ x ) ] ⏟ resample + reweight \mathbb{E}_{(x,y^{\star})\sim\mathcal{D}}\,\bigl[-\nabla_{\theta}\log\pi_{\theta}\bigl(y^{\star}\mid x\bigr)\bigr]=\mathbb{E}_{x\sim\mathcal{D}_{x}}\,\underbrace{\mathbb{E}_{y\sim\pi_{\theta}(\cdot\mid x)}\frac{\mathbf{1}[y=y^{\star}]}{\pi_{\theta}(y\mid x)}\,\bigl[-\nabla_{\theta}\log\pi_{\theta}\bigl(y\mid x\bigr)\bigr]}_{\text{resample + reweight}}\, (5)

Define the auxiliary variables (importance sampling weight) as w ( y ∣ x ) = 𝟏 π θ  ( y ∣ x ) , r ( x , y ) = 𝟏 [ y = y ⋆ ] . w(y\mid x)=\frac{\mathbf{1}}{\pi_{\theta}(y\mid x)},\quad r(x,y)=\mathbf{1}[y=y^{\star}]. Reorganizing equation 5 and rewriting it using the above auxiliary variables, we obtain the form ∇ θ ℒ SFT ( θ ) = − 𝔼 x ∼ 𝒟 x , y ∼ π θ ( ⋅ ∣ x ) [ w ( y ∣ x ) ∇ θ log π θ ( y ∣ x ) r ( x , y ) ] . \nabla_{\theta}\mathcal{L}_{\mathrm{SFT}}(\theta)=-\mathbb{E}_{x\sim\mathcal{D}_{x},\;y\sim\pi_{\theta}(\cdot\mid x)}\bigl[{\color[rgb]{0,0,1}w(y\mid x)}\,\nabla_{\theta}\log\pi_{\theta}(y\mid x)\,{\color[rgb]{0,0,1}r(x,y)}\bigr]. (6)

This form of the SFT gradient closely resembles the policy gradient in Equation equation 4 . Under this formulation, conventional SFT can be interpreted as an on-policy gradient method, where the reward is a sparse indicator function matching the expert trajectory, but biased by an importance weighting term 1 / π θ 1/\pi_{\theta} . We emphasize that this RL-style characterization serves solely as a theoretical lens: both the analysis and subsequent modifications are developed within the RL framework, while the final method remains fully implementable in standard SFT form for computational efficiency. Detailed derivations are provided in Appendix A.2 .