Finetuning with Sampling: SFT Learns Better Than You Think — part 2/3

Cleaned-text code points [15641, 31238). Bibliography omitted here; source.md includes it.

Note that for any π ∈ 𝒫 𝒞 \pi\in\mathcal{P}_{\mathcal{C}} we have that KL ⁡ ( π ∥ p ) \displaystyle\mathrm{KL}\!\left(\pi\,\middle\|\,p\right) = ∑ 𝐱 ∈ 𝒳 ∗ π ⁡ ( 𝐱 ) ​ log ​ π ⁡ ( 𝐱 ) p ⁡ ( 𝐱 ) = ∑ 𝐱 ∈ 𝒞 π ⁡ ( 𝐱 ) ​ log ​ π ⁡ ( 𝐱 ) p ⁡ ( 𝐱 ) , \displaystyle=\sum_{\mathbf{x}\in\mathcal{X}^{*}}\pi(\mathbf{x})\log\frac{\pi(\mathbf{x})}{p(\mathbf{x})}=\sum_{\mathbf{x}\in\mathcal{C}}\pi(\mathbf{x})\log\frac{\pi(\mathbf{x})}{p(\mathbf{x})}, since π ⁡ ( 𝐱 ) = 0 \pi(\mathbf{x})=0 if 𝐱 ∉ 𝒞 \mathbf{x}\notin\mathcal{C} . Now π \pi defines a distribution over 𝐱 ∈ 𝒞 \mathbf{x}\in\mathcal{C} , but p p is not normalized under the restriction to correct trajectories. However, we can rewrite p ⁡ ( 𝐱 ) = Z ⋅ p 𝒞 ​ ( 𝐱 ) p(\mathbf{x})=Z\cdot p_{\mathcal{C}}(\mathbf{x}) for 𝐱 ∈ 𝒞 \mathbf{x}\in\mathcal{C} , where Z = ∑ 𝐱 ′ ∈ 𝒞 p ⁡ ( 𝐱 ′ ) Z=\sum_{\mathbf{x}^{\prime}\in\mathcal{C}}p(\mathbf{x}^{\prime}) , so KL ⁡ ( π ∥ p ) \mathrm{KL}\!\left(\pi\,\middle\|\,p\right) simplifies to ∑ 𝐱 ∈ 𝒞 π ⁡ ( 𝐱 ) ​ log ​ π ⁡ ( 𝐱 ) Z ⋅ p 𝒞 ​ ( 𝐱 ) = − log ⁡ Z + ∑ 𝐱 ∈ 𝒞 π ⁡ ( 𝐱 ) ​ log ​ π ⁡ ( 𝐱 ) p 𝒞 ​ ( 𝐱 ) = KL ⁡ ( π ∥ p 𝒞 ) − log ⁡ Z , \sum_{\mathbf{x}\in\mathcal{C}}\pi(\mathbf{x})\log\frac{\pi(\mathbf{x})}{Z\cdot p_{\mathcal{C}}(\mathbf{x})}=-\log Z+\sum_{\mathbf{x}\in\mathcal{C}}\pi(\mathbf{x})\log\frac{\pi(\mathbf{x})}{p_{\mathcal{C}}(\mathbf{x})}=\mathrm{KL}\!\left(\pi\,\middle\|\,p_{\mathcal{C}}\right)-\log Z, (4) since ∑ 𝐱 ∈ 𝒞 π ⁡ ( 𝐱 ) = 1 \sum_{\mathbf{x}\in\mathcal{C}}\pi(\mathbf{x})=1 and both π \pi and p 𝒞 p_{\mathcal{C}} define a distribution over 𝒞 \mathcal{C} . Then since log ⁡ Z \log Z is a constant independent of π \pi , it follows that the expression is uniquely minimized at π = p 𝒞 \pi=p_{\mathcal{C}} , as desired. ∎

The unnormalized projection p 𝒞 ​ ( 𝐱 ) ∝ p ⁡ ( 𝐱 ) ⋅ 𝟏 ​ ( 𝐱 ∈ 𝒞 ) p_{\mathcal{C}}(\mathbf{x})\propto p(\mathbf{x})\cdot\mathbf{1}\bigl(\mathbf{x}\in\mathcal{C}\bigr) is thus our target distribution, as it is the closest distribution to the base model that also maintains the information content of the off-policy expert traces. Of course, if the base model p p can reliably generate expert trajectories itself, the easiest way to sample from p 𝒞 p_{\mathcal{C}} is to simply perform rejection sampling, which already lends itself to a rudimentary form of on-policy RL ( Xiong et al., 2025 ) .

However, we are interested in the regime where rejection sampling is infeasible, and the expert trajectories are difficult to attain via the base model alone. Then our task is now the following: given access to an arbitrary initial expert policy q q via dataset trajectories, can we algorithmically evolve samples from q q into informational-equivalent but more on-policy samples from p 𝒞 p_{\mathcal{C}} ?

### 4.2 The Metropolis-Hastings Algorithm

We indeed can, by appealing to Markov Chain Monte Carlo (MCMC) approximate sampling techniques. Specifically, we will use a Metropolis-Hastings (MH) sampler ( Metropolis et al., 1953 ; Hastings, 1970 ) , which constructs a Markov chain of sample sequences ( 𝐱 0 , 𝐱 1 , … , 𝐱 n ) (\mathbf{x}^{0},\mathbf{x}^{1},\dots,\mathbf{x}^{n}) using a proposal distribution κ ⁡ ( 𝐱 | 𝐱 i ) \kappa(\mathbf{x}|\mathbf{x}^{i}) to select the next candidate 𝐱 i + 1 \mathbf{x}^{i+1} . With probability A ⁡ ( 𝐱 , 𝐱 i ) = min ​ { 1 , p 𝒞 ​ ( 𝐱 ) ⋅ κ ⁡ ( 𝐱 i | 𝐱 ) p 𝒞 ​ ( 𝐱 i ) ⋅ κ ⁡ ( 𝐱 | 𝐱 i ) } , A(\mathbf{x},\mathbf{x}^{i})=\text{min}\left\{1,\frac{p_{\mathcal{C}}(\mathbf{x})\cdot\kappa(\mathbf{x}^{i}|\mathbf{x})}{p_{\mathcal{C}}(\mathbf{x}^{i})\cdot\kappa(\mathbf{x}|\mathbf{x}^{i})}\right\}, (5)

candidate 𝐱 \mathbf{x} is accepted as 𝐱 i + 1 \mathbf{x}^{i+1} ; otherwise, MH sets 𝐱 i + 1 = 𝐱 i \mathbf{x}^{i+1}=\mathbf{x}^{i} . It is a classic fact that as n → ∞ n\to\infty , this process converges to sampling from the target distribution p 𝒞 p_{\mathcal{C}} , provided that the Markov chain satisfies the following properties:

###### Definition 3 .

The proposal distribution κ \kappa is irreducible if under the induced Markov chain, for any states 𝐱 , 𝐱 ′ \mathbf{x},\mathbf{x}^{\prime} with nonzero mass under the target distribution p 𝒞 p_{\mathcal{C}} , the probability of transitioning to 𝐱 ′ \mathbf{x}^{\prime} starting from 𝐱 \mathbf{x} after some number of steps is nonzero. The proposal is aperiodic if the induced chain of samples does not return to the same sample after a fixed interval number of steps.

Since we are given a dataset of expert traces, we would like to initialize the Markov chain with such a trace and restrict our attention to proposal distributions κ 𝒞 \kappa_{\mathcal{C}} that ensure that 𝐱 i ∈ 𝒞 \mathbf{x}^{i}\in\mathcal{C} for all i i . We refer to such proposal distributions as information-preserving .

Although this might seem to violate the irreducibility criterion, notice we can rewrite ( 5 ) as the following: A ⁡ ( 𝐱 , 𝐱 i ) = min ​ { 1 , p ⁡ ( 𝐱 ) ⋅ κ ⁡ ( 𝐱 i | 𝐱 ) ⋅ 𝟏 ​ ( 𝐱 ∈ 𝒞 ) p ⁡ ( 𝐱 i ) ⋅ κ ⁡ ( 𝐱 | 𝐱 i ) ⋅ 𝟏 ​ ( 𝐱 i ∈ 𝒞 ) } = min ​ { 1 , p ⁡ ( 𝐱 ) ⋅ κ 𝒞 ​ ( 𝐱 i | 𝐱 ) p ⁡ ( 𝐱 i ) ⋅ κ 𝒞 ​ ( 𝐱 | 𝐱 i ) } . A(\mathbf{x},\mathbf{x}^{i})=\text{min}\left\{1,\frac{p(\mathbf{x})\cdot\kappa(\mathbf{x}^{i}|\mathbf{x})\cdot\mathbf{1}\bigl(\mathbf{x}\in\mathcal{C}\bigr)}{p(\mathbf{x}^{i})\cdot\kappa(\mathbf{x}|\mathbf{x}^{i})\cdot\mathbf{1}\bigl(\mathbf{x}^{i}\in\mathcal{C}\bigr)}\right\}=\text{min}\left\{1,\frac{p(\mathbf{x})\cdot\kappa_{\mathcal{C}}(\mathbf{x}^{i}|\mathbf{x})}{p(\mathbf{x}^{i})\cdot\kappa_{\mathcal{C}}(\mathbf{x}|\mathbf{x}^{i})}\right\}. (6) Assuming current candidate 𝐱 i ∈ 𝒞 \mathbf{x}^{i}\in\mathcal{C} , if the new candidate 𝐱 ∉ 𝒞 \mathbf{x}\notin\mathcal{C} , the acceptance ratio defaults to 0 0 . In particular:

###### Proposition 2 .

Running Metropolis-Hastings with irreducible proposals κ \kappa and target distribution p 𝒞 p_{\mathcal{C}} is equivalent to running MH with information-preserving proposals κ 𝒞 \kappa_{\mathcal{C}} and target distribution p p .

This gives us a convenient recipe for boosting an off-policy dataset: initialize at the expert trajectory, mutate the candidates with some information-preserving proposal distribution κ 𝒞 \kappa_{\mathcal{C}} , and use ( 6 ) to accept or reject candidates. In fact, not only does this process eventually converge to p 𝒞 p_{\mathcal{C}} , but it also guarantees a progressive decrease in the policy gap between the data generation policy and the base model.

###### Proposition 3 .

For all k ≥ 0 k\geq 0 , let π k \pi_{k} denote the distribution over candidates obtained after k k MCMC steps applied to initial expert policy q q . Then we have KL ⁡ ( π k + 1 ∥ p ) ≤ KL ⁡ ( π k ∥ p ) . \mathrm{KL}\!\left(\pi_{k+1}\,\middle\|\,p\right)\leq\mathrm{KL}\!\left(\pi_{k}\,\middle\|\,p\right). (7)

###### Proof.

Let Ψ \Psi denote the MCMC transition kernel; i.e., the transformation induced by applying one step of our Metropolis-Hastings procedure. Ψ \Psi has stationary distribution p 𝒞 p_{\mathcal{C}} , so Ψ ⁡ ( p 𝒞 ) = p 𝒞 \Psi(p_{\mathcal{C}})=p_{\mathcal{C}} . Since Ψ \Psi is Markovian, it follows from the data processing inequality that KL ⁡ ( π k + 1 ∥ p 𝒞 ) = KL ⁡ ( Ψ ⁡ ( π k ) ∥ Ψ ⁡ ( p 𝒞 ) ) ≤ KL ⁡ ( π k ∥ p 𝒞 ) . \mathrm{KL}\!\left(\pi_{k+1}\,\middle\|\,p_{\mathcal{C}}\right)=\mathrm{KL}\!\left(\Psi(\pi_{k})\,\middle\|\,\Psi(p_{\mathcal{C}})\right)\leq\mathrm{KL}\!\left(\pi_{k}\,\middle\|\,p_{\mathcal{C}}\right). (8) Now, for any π \pi , recall from ( 4 ) that KL ⁡ ( π ∥ p 𝒞 ) = KL ⁡ ( π ∥ p ) + log ⁡ Z \mathrm{KL}\!\left(\pi\,\middle\|\,p_{\mathcal{C}}\right)=\mathrm{KL}\!\left(\pi\,\middle\|\,p\right)+\log Z for some fixed constant Z Z , so it follows that KL ⁡ ( π k + 1 ∥ p ) ≤ KL ⁡ ( π k ∥ p ) \mathrm{KL}\!\left(\pi_{k+1}\,\middle\|\,p\right)\leq\mathrm{KL}\!\left(\pi_{k}\,\middle\|\,p\right) , as claimed. ∎

In other words, the Metropolis-Hastings procedure progressively shifts the original expert policy to informational-equivalent policies that get progressively closer to the base model distribution.

### 4.3 Projection Sampling for LLMs

As detailed in Karan and Du (2025) , a direct implementation of Metropolis-Hastings for LLMs is infeasible as it requires regenerating full-length token sequences with repeated LLM inference calls. This infeasibility is further compounded by slow mixing , where convergence can require exponentially many MCMC steps due to high dimensional sample spaces or poor choice of proposals and initializations.

To help avoid these issues, we adapt the implementation of Metropolis-Hastings for power sampling in Karan and Du (2025) , which we refer the reader to for further details. In short, we define restriction distributions of the target projection p 𝒞 p_{\mathcal{C}} over sequences of increasing length in units of block size B B . We use Metropolis-Hastings to sample within each distribution, providing a strong initialization to begin sampling from the next one. The full details are presented in Algorithm 1 .

It remains to define an appropriate proposal distribution, as this crucial object facilitates information-preservation and is the only component of Algorithm 1 that interacts with the expert trajectory.

To craft our choice of κ 𝒞 \kappa_{\mathcal{C}} , we use a trick also utilized in Qu et al. (2026) ; Shenfeld et al. (2026) which leverages the strong in-context instruction-following capabilities of pretrained LLMs. Suppose we are given candidate 𝐱 = ( x 0 , ⋯ x L ) \mathbf{x}=(x_{0},\cdots x_{L}) . Our proposal distribution uniformly at random selects an index t ∈ [ 1 , L ] t\in[1,L] and creates a partial trace by truncating the suffix after t t : 𝐱 ~ = ( x 0 , ⋯ , x t ) \tilde{\mathbf{x}}=(x_{0},\cdots,x_{t}) . Then, we pass this partial trace along with the expert trajectory τ \tau in-context to a simple prompt template similar to ones used in Qu et al. (2026) :

Then our proposal κ 𝒞 \kappa_{\mathcal{C}} is simply the distribution defined by conditioning our base model p p on this prompt. In practice, we find that this prompt can sufficiently generate correct trajectories from expert demonstrations, satisfying the constraint of information-preservation.

With this final piece specified, to construct our boosted dataset for finetuning, we simply apply Algorithm 1 to each query-trajectory pair in our expert dataset 𝒟 \mathcal{D} . For examples of boosted traces across different models and datasets, see Appendix A .

Since our Metropolis-Hastings procedure targets sampling from the information projection given information constraints 𝒞 \mathcal{C} , we refer to our sampling algorithm as projection sampling .

Unlike in Karan and Du (2025) , where the sampling algorithm incurs an inference-time cost, projection sampling only incurs a one-time cost , as it is conducted on the training dataset prior to finetuning. To quantify this cost, we can estimate the average number of tokens expended by Algorithm 1 . For each trajectory τ ∈ 𝒟 \tau\in\mathcal{D} , each MCMC step resamples an average of k ​ B 2 \frac{kB}{2} tokens, carried out N MCMC N_{\text{MCMC}} times. Summing, we have 𝔼 cost = | 𝒟 | ⋅ N MCMC ∑ k = 1 ⌈ T / B ⌉ k ​ B 2 ≈ | 𝒟 | ⋅ N MCMC ​ T 2 4 ​ B . \mathbb{E}_{\text{cost}}=|\mathcal{D}|\cdot N_{\text{MCMC}}\sum_{k=1}^{\lceil T/B\rceil}\frac{kB}{2}\approx|\mathcal{D}|\cdot\frac{N_{\text{MCMC}}T^{2}}{4B}. (9)

We can lower the cost by increasing the block size B B , but in general, since projection sampling is a fixed cost, we prefer smaller block sizes that provide higher resolution and less approximation error.

## 5 Experiments

We now empirically demonstrate that the trajectories generated by our sampling algorithm enable SFT to break its weak characterization, frequently outperforming existing on-policy learning algorithms for posttraining.

### 5.1 Experimental Setup

#### Tasks.

We select a set of learning tasks that are characteristic of the range of capabilities posttraining seeks to introduce, including novel skill acquisition, reasoning, and open-ended expertise. In the following discussion, we present three tasks. Each learning task is accompanied by expert trajectories which are either inherent to the dataset or generated by GPT-5.

• Chemistry: We examine whether a pretrained LLM can pick up a new set of skills that it is not explicitly finetuned on beforehand. Following Shenfeld et al. (2026) , we take scientific reasoning as the test domain. In this setting, we use Chemistry L-3 subset of SciKnowEval ( Feng et al., 2024 ) that includes skills like reaction prediction, molar weight calculation, and chemical equation balancing. Moreover, some questions follow a multiple choice format, while others (balancing) are freeform, representing a mixed dataset with a harder-to-specify reward for on-policy RL. There are 2400 problems total, with 1800 training problems and 600 test problems. Expert trajectories are generated by GPT-5.

• Math: We investigate whether a pretrained LLM with some general mathematical ability can improve performance on a set of hard math reasoning problems relative to the base model. In this setting, we use the MATH ( Hendrycks et al., 2021 ) dataset restricted to the hardest problem classes: Levels 3, 4, and 5. There are 9254 problems total, with 8230 training problems and 1024 test problems that are accompanied by existing expert solutions.

• Medical: We test whether our approach is effective in more open-ended domains that are not as immediately verifiable as the prior two. As in Shenfeld et al. (2026) , we look at medical question answering . We use the HuatuoGPT-o1 SFT dataset ( Chen et al., 2024 ) , which provides 19704 questions requiring clinical reasoning skills for diagnoses, treatments, and general medical knowledge, and each question has a provided expert response. For evaluation, we select 1000 questions uniformly at random from the HuatuoGPT-o1 RL dataset that are never seen during training.

#### Models.

For the given posttraining tasks, we select base models across varying model families and sizes that are not already proficient or finetuned on the task domains. For chemistry, we use Qwen2.5-7B-Instruct ( Yang et al., 2024 ) and Olmo-3-7B-Instruct ( Team Olmo et al., 2025 ) , which achieve test set accuracies 34.3 % 34.3\% and 32.8 % 32.8\% respectively. For math, we use Qwen2.5-3B, which achieves test accuracy 31.5 % 31.5\% . Finally, for medical, we use Qwen2.5-7B-Instruct, which achieves test accuracy 35.3 % 35.3\% .

#### Evaluation.

We include several benchmarks for each task that holistically measure both task-specific generalization and retention of prior capabilities . All benchmarks are scored based on single-shot accuracy.

• Chemistry: We measure generalization capabilities in domain with the test set for SciKnowEval. To measure prior capabilities, we evaluate on math benchmarks AMC, MATH500, and GSM8K ( Cobbe et al., 2021 ) alongside MMLU ( Hendrycks et al., 2020 ) and GPQA ( Rein et al., 2024 ) .

• Math: To measure generalization capabilities, we track performance on the test set MATH(3,4,5) as well as out-of-distribution generalization on AMC, MATH500, and GSM8K. To track prior capabilities, we evaluate on MMLU, Chemistry, and GPQA.

• Medical: Since medical reasoning is a more open-ended domain, we use an automated LLM grader (GPT-5-mini) to evaluate correctness on the test set (see Appendix C.3 for the exact grading prompt). For capability retention, we evaluate AMC, MATH500, GSM8K, MMLU, and GPQA.

Baselines. For each task, we benchmark against a wide variety of posttraining algorithms ranging from vanilla SFT to on-policy learning with privileged information via RL and self-distillation.