Finetuning with Sampling: SFT Learns Better Than You Think — part 1/3
Cleaned-text code points [0, 15641). Bibliography omitted here; source.md includes it.
# Finetuning with Sampling: SFT Learns Better Than You Think
###### Abstract
Introducing new capabilities to frontier models has long been the goal of posttraining, which predominantly employs supervised finetuning (SFT) and reinforcement learning (RL) to this end. Conventional wisdom dictates that RL enables strong generalization on new tasks without losing existing capabilities, while SFT is prone to weak generalization and catastrophic forgetting. At the same time, SFT can learn from off-policy expert data, whereas RL must rely on a model’s ability to find successful trajectories with repeated sampling. In our work, we seek to leverage the strength of on-policy learning while utilizing the privileged information contained in off-policy data. However, rather than modifying the learning objective to accommodate this data, we instead tailor the data distribution to better suit the learner. We introduce a Markov chain Monte Carlo (MCMC) sampling algorithm that progressively transforms off-policy traces to be more on-policy given a reference model for finetuning. Across tasks like scientific skill acquisition, mathematical reasoning, and open-ended expertise, our sampling algorithm enables SFT to rival prevailing posttraining techniques, often generalizing better and forgetting less than strong on-policy baselines. In addition, the resulting finetuned models exhibit strong distributional performance and are capable of learning beyond sharpening the base model distribution. At a higher level, our approach presents sampling as a model-native operator that shapes data for learnability , offering broader utility as a general-purpose primitive throughout the posttraining stack.
## 1 Introduction
Posttraining has brought about revolutionary advances in the capabilites of frontier large language models (LLMs) ( Guo et al., 2025 ; Hu et al., 2025 ) , giving sizeable performance gains across domains like math, science, and coding ( Hendrycks et al., 2021 ; Li et al., 2022 ; Rein et al., 2024 ) .
Still, a key open question remains: how can we introduce fundamentally new capabilities during posttraining without giving up existing ones ? A large body of recent literature has been dedicated to understanding the extent to which existing learning frameworks – i.e., supervised finetuning (SFT) and reinforcement learning (RL) – can achieve this end. Overwhelming evidence ( Chen et al., 2025 ; Chu et al., 2025 ; Shenfeld et al., 2025 ) points towards RL as the paradigm of choice, citing superior generalization on unseen domains that appears to leave prior abilities intact. Meanwhile, SFT suffers from weak generalization and catastrophic forgetting ( Kirkpatrick et al., 2017 ; Luo et al., 2023 ) : a phenomenon where finetuned models exhibit significant deterioration in existing capabilities.
Chen et al. (2025) ; Shenfeld et al. (2025) attribute this performance gap to the on-policy nature of RL: the model’s own samples dictate learning updates, constraining finetuning towards distributions that do not stray far from the original starting point. SFT on the other hand is off-policy , leaving finetuning more susceptible to drastic updates and shifts in behavior ( Xiong et al., 2025 ) .
At the same time, learning strictly on-policy is also a limitation, as it relies on the model’s own ability to find successful trajectories through repeated sampling. If the model cannot generate a correct rollout on a training sample, there is no collective variance in reward, resulting in no learning signal. Qu et al. (2026) observes this deficiency in practice, identifying a sizeable regime of hard training samples that are effectively discarded during RL as they are simply too difficult to yield any trainability.
This limitation is not ideal for the purposes of introducing new capabilities, where the model is unlikely to already be competent enough to produce a successful trajectory. Here SFT has an advantage, as it leverages trajectories constructed from off-policy privileged information that provide rich supervision signal without necessitating an explicit reward.
In this work, rather than modifying the learning algorithm to accommodate off-policy data, we ask if we can instead shape the data distribution to accommodate learning. More precisely, can we algorithmically sample on-policy trajectories that remain faithful to privileged off-policy information ? To this end, we propose a sampling algorithm that progressively transforms a dataset of off-policy expert traces to be distributionally closer to a given base model distribution.
Remarkably, this sampling step enables SFT to rival prevailing posttraining techniques, often generalizing better and forgetting less than strong on-policy learning algorithms .
Our contributions can be summarized as follows: • We introduce a framework that formalizes the task of transforming off-policy data into on-policy data for finetuning. Namely, given a base model and a dataset of expert trajectories, we specify a target data generation policy that preserves the information content of the original dataset while maintaining maximal proximity to the base model distribution.
• To generate boosted samples from this policy, we introduce projection sampling , an approximate sampling algorithm using Markov chain Monte Carlo (MCMC) techniques that iteratively refine expert trajectories according to their likelihoods with respect to the base model.
• We empirically demonstrate the effectiveness of our sampling procedure for SFT over a range of tasks, including scientific skill acquisition, mathematical reasoning, and open-ended expertise. Our results show that SFT with sampling can outperform on-policy counterparts like RL and self-distillation on new task generalization as well as prior capability retention.
Our results collectively illustrate that composing sampling with finetuning is a powerful paradigm that can enable a learning algorithm as standard as SFT to perform far beyond conventional expectations.
## 2 Related Works
SFT vs. RL. Many recent works have investigated the performance gap between SFT and RL in posttraining. Chu et al. (2025) was an early work that noticed the gap in generalization performance, asserting SFT is prone to memorization while RL can generalize beyond the training distribution. Xiong et al. (2025) offers a mathematical explanation for this phenomenon, suggesting that training off-policy can result in large and unstable gradient updates that encourage overfitting and harm generalization. Shenfeld et al. (2025) and Chen et al. (2025) explore the performance gap from the lens of prior capability retention, noting that SFT is highly susceptible to catastrophic forgetting ( Kirkpatrick et al., 2017 ; Luo et al., 2023 ) . Shenfeld et al. (2025) conducts extensive empirical analysis to attribute this to the off-policy nature of SFT and even shows that the level of forgetting correlates with the KL divergence of the finetuned model with the base model. Similarly, Chen et al. (2025) offers a mode-seeking rationale and concludes similarly that on-policy data mitigates forgetting while off-policy data facilitates it.
Several works also attempt to bridge the gap between SFT and RL by interpolating between off-policy and on-policy learning. Both Liu et al. (2025) and Qu et al. (2026) construct variants of GRPO ( Guo et al., 2025 ) that use expert data in-context to encourage generating correct rollouts for hard problems during RL, treating them as vanilla generations during training. On the SFT side, Wu et al. (2025) modulates the SFT objective with importance sampling weights to match the on-policy gradient, while Zhu et al. (2025) adds a KL divergence term to further encourage proximity to the base model. While these approaches alter the learning objective to better account for off-policy data, our approach leaves SFT intact, and instead focuses on modulating off-policy data to be more on-policy.
#### Self-Distillation.
An on-policy alternative to RL that has recently begun to gain recognition for incorporating off-policy data is self-distillation . Self-distillation typically constructs a teacher model from a base model by conditioning on privileged information ; e.g., expert data or environment feedback, which is then distilled via a KL divergence loss. Several works ( Shenfeld et al., 2026 ; Ye et al., 2026 ; Zhao et al., 2026 ) offer variations on this idea and demonstrate promising results. OPSD ( Shenfeld et al., 2026 ; Zhao et al., 2026 ) , which we refer to as a baseline in this paper, offers one promising avenue for continual learning: i.e., adapting to new tasks without forgetting.
#### MCMC Sampling with LLMs.
Prior research has also explored how to integrate MCMC methods into LLM sampling. Several works have explored using MCMC to tilt to an external reward function ( Zhao et al., 2024 ; Faria et al., 2024 ) or probabilistic program ( Lew et al., 2023 ) . Most related to our work are Karan and Du (2025) , Azizi et al. (2026) , Arzhantsev et al. (2026) , which use MCMC sampling to sample from a sharpened distribution for reasoning. Other works expand on this framework by targeting critical tokens ( Li et al., 2025 ) to make MCMC sampling more efficient ( Zhou et al., 2026 ; Ji et al., 2026 ) . In contrast, our approach uses sampling to transform off-policy data for subsequent finetuning.
## 3 Preliminaries
Let 𝒳 \mathcal{X} be a finite vocabulary of tokens, and let 𝒳 T \mathcal{X}^{T} denote the set of finite sequences of tokens x 0 : T = ( x 0 , x 1 , … , x T ) x_{0:T}=(x_{0},x_{1},\dots,x_{T}) , where x i ∈ 𝒳 x_{i}\in\mathcal{X} for all i i and T ∈ ℤ ≥ 0 T\in\mathbb{Z}_{\geq 0} is some nonnegative integer. For convenience, for a given t t , let x < t = ( x 0 , … , x t − 1 ) x_{<t}=(x_{0},\dots,x_{t-1}) and x > t = ( x t + 1 , … , x T ) x_{>t}=(x_{t+1},\dots,x_{T}) , with similar definitions for x ≤ t x_{\leq t} and x ≥ t x_{\geq t} . In general, 𝐱 \mathbf{x} refers to a token sequence x 0 : T x_{0:T} , where T T is implicitly given, and 𝒳 ∗ \mathcal{X}^{*} denotes the set of token sequences with finite but arbitrary length.
Then an LLM defines a distribution p p over token sequences 𝒳 ∗ \mathcal{X}^{*} by autoregressively learning the conditional token distributions p ( x t | x < t ) p(x_{t}|x_{<t}) for all t t , giving the joint distribution via the identity p ( x 0 : T ) = ∏ t = 0 T p ( x t | x < t ) . p(x_{0:T})=\prod_{t=0}^{T}p(x_{t}|x_{<t}). (1)
To sample a sequence from p p , we simply sample from the LLM token by token using the conditional distributions, which by ( 1 ) directly samples from the joint distribution.
#### Supervised Finetuning.
Let 𝒟 = { ( 𝐪 i , 𝐱 i ) } i = 1 N \mathcal{D}=\{(\mathbf{q}_{i},\mathbf{x}_{i})\}_{i=1}^{N} denote a dataset of expert trajectories 𝐱 i \mathbf{x}_{i} in response to queries 𝐪 i \mathbf{q}_{i} . Then SFT optimizes the cross-entropy loss objective ℒ SFT = 𝔼 ( 𝐪 , 𝐱 ) ∼ 𝒟 [ − log p ( 𝐱 ∣ 𝐪 ) ] , \mathcal{L}_{\text{\rm SFT}}=\mathbb{E}_{(\mathbf{q},\mathbf{x})\sim\mathcal{D}}\left[-\log p\left(\mathbf{x}\mid\mathbf{q}\right)\right], (2) via gradient updates on the LLM’s parametrization of p p .
## 4 Boosting Off-Policy Data with MCMC Sampling
In this section, we present our sampling algorithm that maps off-policy expert data closer to the base model distribution for finetuning. Our procedure formalizes interpolating off-policy information into on-policy learning as highlighted in Section 1 . To construct an informational-equivalent dataset that is easiest for the base model to internalize, we must provide a generation policy that maintains this information with minimal KL divergence. This specifies a target data generation distribution, and hence, all that remains is for us to explicitly provide an algorithm that samples from it.
The remainder of the section is organized as follows. Section 4.1 presents our formalization specifying this (unnormalized) boosted target data policy. Section 4.2 introduces a Markov chain Monte Carlo (MCMC) algorithm that approximately samples from this policy and ensures our sampling procedure produces generation policies that get progressively closer to the base model distribution. Finally, Section 4.3 outlines our implementation of this algorithm for LLMs.
### 4.1 Targeting the Information Projection
Suppose we are given a dataset of expert query-trajectory pairs 𝒟 = { ( 𝐪 i , 𝐱 i ) } i = 1 N \mathcal{D}=\{(\mathbf{q}_{i},\mathbf{x}_{i})\}_{i=1}^{N} . For notational convenience, we fix a query 𝐪 i \mathbf{q}_{i} for some i i for the remainder of the section and assume all distributions are implicitly conditioned on 𝐪 i \mathbf{q}_{i} . We can easily lift this assumption by stitching together the conditionals across 1 ≤ i ≤ N 1\leq i\leq N .
For a given equivalence relation on trajectories, let 𝒞 \mathcal{C} define the set of all trajectories 𝐱 ∈ 𝒳 ∗ \mathbf{x}\in\mathcal{X}^{*} that are equivalent to the associated expert trajectory 𝐱 i \mathbf{x}_{i} . For theoretical purposes, we assume the existence of such an equivalence relation a priori; however, in practice, 𝒞 \mathcal{C} refers to a task-dependent notion of ”correctness” or ”semantic equivalence”. For example, for math/science reasoning, a semantically equivalent trajectory 𝐱 \mathbf{x} refers to any trajectory leveraging information from expert trace 𝐱 i \mathbf{x}_{i} that results in a correct final answer. For fact learning, a semantic equivalent 𝐱 \mathbf{x} must contain the same factual content as 𝐱 i \mathbf{x}_{i} , as judged by an automated (LLM) grader.
###### Definition 1 .
Let 𝒫 𝒞 \mathcal{P}_{\mathcal{C}} denote the space of all distributions over token sequences 𝒳 ∗ \mathcal{X}^{*} that have conditional support on 𝒞 \mathcal{C} : that is, for a policy π ∈ 𝒫 𝒞 \pi\in\mathcal{P}_{\mathcal{C}} , we have π ( 𝐱 ) = 0 \pi(\mathbf{x})=0 if 𝐱 ∉ 𝒞 \mathbf{x}\notin\mathcal{C} . In other words, informally, 𝒫 𝒞 \mathcal{P}_{\mathcal{C}} is the set of all distributions that output only trajectories that are equivalent to the expert traces.
Our setup naturally admits a well-specified formalism for boosting off-policy data. The dataset of expert trajectories itself is drawn from some arbitrary policy q ∈ 𝒫 𝒞 q\in\mathcal{P}_{\mathcal{C}} , implying the SFT objective ( 2 ) minimizes the cross-entropy 𝔼 q [ − log p ] \mathbb{E}_{q}[-\log p] . However, for arbitary q q , this objective can be arbitarily off-policy. Given that we want to preserve the information signal of 𝒟 \mathcal{D} with trajectories that are as on-policy as possible, this is equivalent to finding the distribution in 𝒫 𝒞 \mathcal{P}_{\mathcal{C}} closest to the base model distribution p p .
###### Definition 2 .
The information projection ( Csiszár, 1975 ) of p p onto 𝒫 𝒞 \mathcal{P}_{\mathcal{C}} is defined as p 𝒞 := arg min π ∈ 𝒫 𝒞 KL ( π ∥ p ) . p_{\mathcal{C}}:=\arg\min_{\pi\in\mathcal{P}_{\mathcal{C}}}\mathrm{KL}\!\left(\pi\,\middle\|\,p\right). (3)
###### Proposition 1 .
The distribution that uniquely minimizes the objective ( 3 ) is the restriction of the base model onto equivalent trajectories: p 𝒞 ( 𝐱 ) = p ( 𝐱 ) ⋅ 𝟏 ( 𝐱 ∈ 𝒞 ) ∑ 𝐱 ′ ∈ 𝒳 ∗ p ( 𝐱 ′ ) ⋅ 𝟏 ( 𝐱 ′ ∈ 𝒞 ) , p_{\mathcal{C}}(\mathbf{x})=\frac{p(\mathbf{x})\cdot\mathbf{1}\bigl(\mathbf{x}\in\mathcal{C}\bigr)}{\sum_{\mathbf{x}^{\prime}\in\mathcal{X}^{*}}p(\mathbf{x}^{\prime})\cdot\mathbf{1}\bigl(\mathbf{x}^{\prime}\in\mathcal{C}\bigr)}, where 𝟏 ( 𝐱 ∈ 𝒞 ) \mathbf{1}\bigl(\mathbf{x}\in\mathcal{C}\bigr) takes the value 1 1 if 𝐱 ∈ 𝒞 \mathbf{x}\in\mathcal{C} and 0 0 otherwise.
###### Proof.