Finetuning with Sampling: SFT Learns Better Than You Think — part 3/3

Cleaned-text code points [31238, 44614). Bibliography omitted here; source.md includes it.

• Chemistry: We compare our method on all evaluations for this task to the base models, SFT with the expert dataset trajectories, and a variant of SFT using our sampling proposal prompt to rewrite each expert trajectory; i.e., “ 0 th 0^{\text{\rm th}} order MCMC”. In addition, we include on-policy self-distillation (OPSD) ( Shenfeld et al., 2026 ; Zhao et al., 2026 ) , which distills from the base model conditioned on privileged information.

• Math: We use the baselines used in chemistry alongside two RL baselines, since the rollouts are easily verifiable. We include GRPO ( Shao et al., 2024 ) as well as a stronger variant UFT ( Liu et al., 2025 ) , which uses the off-policy expert traces as privileged information during rollout generation.

• Medical: Same baselines as chemistry.

#### Sampling.

For all datasets, we use block number B = 32 B=32 with maximum sequence length T = 1856 T=1856 over N MCMC = 10 N_{\text{\rm MCMC}}=10 MCMC steps.

#### Training.

For all tasks, all SFT training runs are tuned with the following hyperparameters across the corresponding ranges: epoch number: { 1 , 2 } \{1,2\} , learning rate: { \{ 5e-5, 1e-5, 5e-6 } \} , and batch size: { 16 , 32 , 64 } \{16,32,64\} . For the medical task, we extend the epoch range to { 1 , 2 , 4 , 6 } \{1,2,4,6\} to accommodate a larger dataset. The optimizer is AdamW with a cosine scheduler and gradient clipping for norms greater than 1 1 . The on-policy self-distillation runs are tuned with hyperparameters from ( Shenfeld et al., 2026 ) while the RL baselines use the default hyperparameters from ( Liu et al., 2025 ) . All models are trained using H100s and H200s.

### 5.2 Main Results

We display our main results for the Qwen model family in Table 1 . For results on Olmo, see Section 5.3 and Appendix B .

Stronger generalization. Across different pretrained base models and different tasks, we see that projection sampling enables SFT to achieve an exceptionally strong performance when evaluated on new capabilities introduced in finetuning. In chemistry, sampling SFT is able to improve base model performance on chemistry by +31.7% , exceeding even the improvement yielded by the on-policy OPSD by +4.20% . Simple zero-shot rewrites of the off-policy traces do not perform as strongly in this setting, suggesting the importance of the full MCMC process in Algorithm 1 .

Math reasoning vividly illustrates that capability gains transfer to adjacent out-of-distribution test domains as well. Whereas vanilla SFT leads to a drop in performance across the board in math benchmarks, sampling SFT yields a +18.0% boost on the hardest problems in the MATH benchmark, even surpassing the reasoning boosts obtained by GRPO and UFT. This extends to a +14.4% boost on AMC and a +20.3% boost on GSM8K, which are on par with the boosts obtained by either RL algorithm. The performance gain on MATH500 is especially notable, providing a +33.7% boost to the base model and surpassing the next best performing (RL) baseline by +26.9% .

Not only does the sampling SFT model generalize well, but it also provides a strong initialization for RL that leads to even further gains. In the math section of Table 1 , we record GRPO initialized with the sampling SFT checkpoint (Sampling SFT-RL) and observe that it is the strongest performing model overall, achieving +40.7% performance in MATH500 and +25.1% performance in GSM8K: within reach of the 7B-Instruct model.

With just off-policy expert traces, SFT can fail to generalize well. But composed with our sampling algorithm, SFT can exhibit exceptionally strong generalization, even outperforming RL and OPSD baselines. The fact that this sampling procedure is custom to the base model distribution is crucial: Appendix B details results demonstrating that finetuning, e.g., Qwen on the same base task data but boosted for Olmo severely underperforms all baselines.

Retaining prior capabilities. This strong generalization does not come at the cost of catastrophic forgetting. For chemistry in Table 1 , vanilla SFT results in a considerable drop in capabilities. Meanwhile, sampling SFT does not lose any MMLU knowledge and reduces the average loss in prior task accuracy down to just -1.10% . In fact, our model forgets the least , even including OPSD. These results are mirrored in the medical domain, where sampling recovers a 16.3% loss in average prior capabilities with vanilla SFT, and again, forgets the least among all baselines.

### 5.3 Analysis

We now examine distributional properties of our sampled traces for finetuning as well as those of the resulting model capabilities.

Dataset likelihoods. We can directly observe the effect projection sampling has on expert traces.

Figure 3 plots histograms of the sequence log-likelihoods (averaged by length) under the base model of both the expert traces and the boosted traces for both science and math reasoning. The on-policy projection is apparent, yielding traces that are much higher-likelihood relative to the base model while preserving correctness.

Learning beyond sharpening. In Section 1 , we highlighted the overarching goal of posttraining as introducing fundamentally new capabilities that are not present in the base model. Given that projection sampling pushes off-policy data to be more in-distribution, can our finetuned models still acquire new behaviors beyond just sharpening existing ones?

We answer in the affirmative. In Figure 4 , we examine the pass@ k k accuracy of Olmo-3-7B-Instruct relative to finetuning with sampling (ours) as well as on-policy distillation with privileged information (OPSD) on the science task. If finetuning with our sampling algorithm was simply sharpening existing capabilities, we would expect our pass@ k k curve to eventually converge to the base model. However, we actually see large, consistent gaps in the pass@ k k rate up to very large k k , demonstrating that our finetuned model has fundamentally stronger capabilities than the base model. Moreover, for k > 2 k>2 , we see substantial gaps in our pass@ k k rate relative to OPSD, indicating that our method enables learning capabilities that OPSD does not learn .

We can also be more granular at the the level of individual evaluation problems. In other words, we can identify “hard problems” that the base model absolutely cannot solve (pass@ k k rate zero for highest k k ) but our finetuned model can start reliably solving: in fact, some evaluation tasks go from zero pass rate in the base model to pass rates that are up to 67.2% or 53.1% .

Scaling sampling compute. Proposition 3 observes that increasing the number of MCMC steps in projection sampling results in data generation policies π k \pi_{k} that are progressively closer to the base model in KL divergence. This frames sampling as a natural axis for scaling compute , expending more MCMC steps for more on-policy learning data.

We can directly observe this scaling empirically in Figure 5 . For base model Qwen2.5-7B-Instruct on the science task, we apply k k MCMC steps to the off-policy training data, for k ∈ [ 0 , 2 , 4 , 6 , 8 , 10 ] k\in[0,2,4,6,8,10] . Note that k = 0 k=0 corresponds to a data rewrite, i.e., asking the base model to simply rewrite off-policy data “in its own words”. On the left vertical axis, we plot KL ⁡ ( π k ∥ π base ) \mathrm{KL}\!\left(\pi_{k}\,\middle\|\,\pi_{\text{\rm base}}\right) , where π k \pi_{k} is the data generation policy induced by k k MCMC steps. We can directly estimate this KL divergence as trajectories are repeatedly generated by a proposal distribution ( κ 𝒞 ( ⋅ | τ ) \kappa_{\mathcal{C}}(\cdot|\tau) in Algorithm 1 ) parametrized by an LLM, from which we can extract next-token log probabilities. On the right vertical axis, we plot the accuracy of the base model finetuned on the boosted data for each k k . As we increase sampling compute, the boosted data distribution monotonically closes the KL gap with the base model, while the corresponding finetuned models trend upwards in accuracy. Hence, empirically, more MCMC steps lead to more on-policy data, which results in better generalization after finetuning.

## 6 Conclusion

In this work, we introduce a formalism that bridges an apparent disconnect between privileged off-policy information and on-policy training. By carefully boosting the likelihood of off - policy expert data towards the base model’s distribution, we obtain a more on - policy equivalent that is far more suitable for learning. Indeed, across science, math reasoning, and open-ended expertise, SFT on boosted data can generalize better and forget less than both vanilla SFT and strong on - policy baselines, including RL and self - distillation. The resulting models also exhibit strong distributional performance over multiple samples and are able to learn fundamentally new capabilities that are not present in the base model.

Beyond just SFT, sampling offers a powerful framework for the posttraining stack, acting as a model-native operator that shapes data for learnability. For example, for on-policy distillation, our MCMC process directly induces a teacher distribution from initial distillation traces that is much closer in KL divergence to the student, facilitating more stable updates. Likewise, conditioned on expert information, our method can simulate on-policy rollouts for RL on hard tasks where the base model is unable to generate signal. By expending sampling-time compute in exchange for stronger, more on-policy trace distributions constructed from privileged information, sampling provides a simple, robust, and scalable pathway for continually expanding LLM capabilities.

## 7 Acknowledgements

A.K. would like to thank the Paul and Daisy Soros Foundation, NDSEG Fellowship, and Kempner Institute for their support. S.C. was supported in part by NSF CAREER award CCF-2441635 and the Harvard Dean’s Competitive Fund for Promising Scholarship.

## Appendix A Examples of Boosted Off-Policy Data

## Appendix B Additional Results

We include results on a non-Qwen-family model to demonstrate that our method applies to a variety of pretrained model families. Namely, we provide results for Olmo-3-7B-Instruct on the chemistry task.

We also include two more comparative experiments for the Chemistry task for Qwen2.5-7B-Instruct to stress that the alignment between data distribution and learning distribution is crucial. For the first, we take the boosted dataset relative to Olmo-3-7B and finetune Qwen2.5-7B-Instruct on that. We find the accuracy on Chemistry drops to 57.33% , which is significantly worse than the other baselines ( 61.8% for vanilla SFT). For the second experiment, we train Qwen2.5-7B-Instruct on a 50/50 mix of boosted and original SFT data to see the effect of interpolation between off-and-on policy data. We find accuracy on Chemistry reaches 62.14% , in between standard SFT and SFT on the boosted dataset.

## Appendix C Further Experimental Details

### C.1 Information-Preservation

One condition on the proposal distribution in Algorithm 1 to verify is the preservation validity of our MCMC process on off-policy data. We measure this by reporting the accuracy of the boosted dataset for each task: for chemistry, our dataset retains 94.33% accuracy with Qwen and 93.94% with Olmo. For math, we retain 95.33% and for medical we retain 95.86% .

### C.2 Sampling

Note that a full implementation of Algorithm 1 can require two forward passes through an LLM per iteration: one for generating a candidate 𝐱 ′ \mathbf{x^{\prime}} via the proposal κ 𝒞 \kappa_{\mathcal{C}} and another for estimating the transition probability κ 𝒞 ​ ( 𝐱 | 𝐱 ′ ) \kappa_{\mathcal{C}}\left(\mathbf{x}|\mathbf{x^{\prime}}\right) . To reduce inference overhead, we approximate the swapping condition by swapping whenever we generate a higher likelihood candidate than the current. This has the effect of much more aggressively shifting to high-likelihood regions under the base model, but as Figure 5 demonstrates, these aggressive shifts still monotonically reduce the KL divergence between data generation policy and base distribution.

### C.3 Proposal and Grading Prompts

We provide a sample proposal kernel κ 𝒞 \kappa_{\mathcal{C}} for the chemistry task below (other tasks follow a similar prompt) as well as the semantic equivalence grading prompt for the medical task that is fed into GPT5-mini.

18 19 10

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .