Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved) — part 3/4
Cleaned-text code points [31563, 46967). Bibliography omitted here; source.md includes it.
Suppose we introduce importance weights (according to Equation ( 8 )) where q ( τ ) = π ( τ , θ ) q(\tau)=\pi(\tau;\theta) . Now over the course of training, we will adaptively put more weight on pull-right compared to pull-left actions until we recover a policy that will pull-right every time. This can also be seen in Figure 1 . Thus importance weighting can recover information from failures . Indeed, if we train with iw-SFT on this data, we recover the optimal policy of just choosing pull-right; answering Q2 : iw-SFT can “recover” information from discarded samples.
Finally regarding the failure modes ( Q3 ): while we recover the optimal policy with iw-SFT in this case, it does lead to collapse of the policy to a single action (which happens to be correct here). In general, the importance weights determine how far away from the reference policy we allow π θ \pi_{\theta} to deviate. If we place no bound on the importance weights one can easily see that in a high-dimensional case, they might grow out of bounds (the infinite variance problem [ Andradóttir et al., 1995 ] ) or lead to us discarding too much of the filtered data. As mentioned above, this can be mitigated by forcing q ( τ ) q(\tau) to stay close to π ref \pi_{\mathrm{ref}} –e.g. via clipping.
## 4 Experiments
We evaluate our proposed improvements to SFT, and compare to existing RL algorithms in two settings: 1) training LLMs for reasoning, 2) training control policies from offline data.
### 4.1 Training LLMs for reasoning via SFT on curated data
For our initial set of experiments we consider fine-tuning an open-source LLM (Qwen2.5-32B-Instruct [ Yang et al., 2024 ] ) to improve its reasoning capabilities for challenging maths problems. To finetune the model, we use the S1.1K dataset introduced in Muennighoff et al. [2025] . This is a heavily curated dataset consisting of 1,000 1,000 high quality reasoning traces. These examples were created by first collecting a set of 59K candidate question/answer pairs from existing datasets [ LI et al., 2024 , of America, 2024 , Gao et al., 2024 , Zhong et al., 2023 ] and generating reasoning traces and responses for the respective questions using Gemini Flash [ Google, 2024 ] . The dataset is then decontaminated against the test datasets and filtered down to 1k examples by quality and diversity as described in Muennighoff et al. [2025] . This is thus a great example of a high-quality, curated dataset.
The hyperparameters of our model are set the same as in [ Muennighoff et al., 2025 ] (we use cosine annealing for the learning rate and AdamW [ Loshchilov et al., 2017 ] ) details are given in the appendix. We also use g ( x ) = 0.1 × clip ( x , 0.2 , 1.8 ) g(x)=0.1\times\mathrm{clip}(x,0.2,1.8) to smooth/clip the importance weights. Given that we do not have access to the model weights for Gemini Flash (which was used to generate the reasoning traces), or its original training data, we also have to rely on an approximation for the reference distribution π ref \pi_{\mathrm{ref}} in the importance weights. Ideally, we would pick a model that is “close” to Gemini-Flash as an approximation. We tested multiple models at the 32B parameter level, comparing their summed log-probabilities over examples from the dataset and find that they assign similar likelihood (albeit giving different top-k next token predictions) and settled on simply using the model we start from as the reference model such that q , π ref q,\pi_{\mathrm{ref}} and π θ \pi_{\theta} are all close at the beginning of training.
All results and comparisons to models from the literature are shown in Table 1 . In the table we also include an ablation of our method which uses importance weighting but on the token level rather than sequence level; this result is shown as iw-SFT(per-step iw). As can be seen from the table, our method outperforms the original s1 results (which also performs SFT, starting from the same checkpoint) on AIME 2024 as well as GPQA Diamond while matching it on MATH 500. Interestingly, we find that iw-SFT does not require budget forcing (a method introduced in Muennighoff et al. [2025] to force the model to keep thinking by injecting “Wait” tokens) to reach its peak performance. This suggests that the importance weighted objective itself can help extract information from the training data, foregoing the need for extra test time optimization. For ablations please see Appendix G . The model that only uses per-step importance weights also reaches high-performance across benchmarks, albeit losing 2 percent compared to full sequence importance weighting on GPQA Diamond.
Overall, training with iw-SFT on the curated dataset S1.1K introduced by Muennighoff et al. [2025] surpasses training with SFT only (s1 / s1.1 in the table) by up to 7% depending on the AIME benchmark. This sets the state-of-the art for open models trained on open data; although better closed models exist as shown in the comparison table.
### 4.2 Offline RL for Continuous Control
To probe for the general applicability of our approach, we turn to learning policies for continuous control using the offline RL benchmark datasets from D4RL [ Fu et al., 2020 ] . We consider the standard locomotion tasks from the benchmark (halfcheetah, hopper, walker2d) both in the setting where data comes from expert policies (Expert (v2) below) and in the setting where data was extracted from a standard RL training run (Med. Replay (v2)) and is thus more diverse.
Since all trajectories in the datasets come with reward information (which we can interpret as unambiguous quality scores), we consider the quality sampling setting. We first filter the data to only contain the top 10 % 10\% of trajectories and then turn the cumulative episode rewards into ordinal quality annotations by binning them into 3 overlapping bins, see Appendix E for details. Our model is a simple 3 layer MLP policy as listed in Appendix F and instead of updating the importance weighting distribution q q at discrete time-points we use exponential weight averaging θ q = α θ q + ( 1 − α ) θ \theta_{q}=\alpha\theta_{q}+(1-\alpha)\theta during each step of the optimization (this is similar to how target networks in continuous control RL algorithms are often updated [ Lillicrap et al., 2015 , Fujimoto and Gu, 2021 ] ). We smooth the importance weights with g ( x ) = k x g(x)=kx . We obtain a reference distribution π ref \pi_{\text{ref}} by first performing simple behavior cloning on all data (e.g. optimizing Equation ( 5 ) on all data instead of filtered data).
Results are given in Table 2 . We evaluated SFT, SFT(Q) and iw-SFT(Q) and list results alongside baselines from Kostrikov et al. [2021] consisting of behavior cloning (BC), which is equivalent to SFT but with parameters initialized randomly. BC on the top-10% of data (BC 10%) as well as four state-of-the-art offline RL algorithms (AWAC [ Nair et al., 2020b ] , TD3+BC [ Fujimoto and Gu, 2021 ] , CQL [ Kumar et al., 2020 ] , IQL [ Kostrikov et al., 2021 ] ). First, we can observe that SFT performs similar to BC (10%), suggesting that initializing to a policy trained on all data does not lead to meaningful generalization benefits in these tasks. Second, SFT(Q) clearly outperforms BC, BC(10%) and SFT in all tasks. This shows that training with quality scores, where available, is indeed helpful. Furthermore, while there are algorithms achieving better results in some domains, SFT(Q) is competitive with strong RL baselines in all settings; confirming that the bound optimized by SFT(Q) can meaningfully optimize for the RL objective. Finally iw-SFT(Q) further leads to a slight improvement over SFT(Q), albeit not as large of an improvement as in the LLM case; suggesting that perhaps this data (which stems from trained policies) is too prototypical, making it hard to outperform quality filtering. Overall, we show that our improved SFT variants can be competitive to offline RL for continuous control.
### 4.3 Fine-tuning control policies with little data and noisy quality scores
In a final experiment we test whether fine-tuning can be helpful for control tasks if less prototypical data is available. For this purpose we consider the the Franka kitchen task [ Fu et al., 2020 ] , in which a robot has to interact with a number of items in a simulated kitchen in a specific order. We first pre-train a BC policy on the “partial” part of the dataset (containing only some successful trajectories) and then fine-tune on the top 5 % 5\% trajectories from the “complete” dataset of trajectories (which are all successful but are not enough trajectories to learn the task from) and top 5 % 5\% of data from the “mixed” part of the dataset (which solve parts of the task but are not fully successful). Results are depicted in Table 3 . As can be seen, BC on the complete data is reasonably successful while training only on the 5 % 5\% filtered data fails. Fine-tuning on this data maintains performance, and closes the gap to BC on complete data significantly. Weighting by quality (SFT(Q)) closes the gap further and iw-SFT(Q) finally is able to recover performance.
## 5 Conclusion and Discussion
In this work, we draw on a connection between reinforcement learning and supervised fine tuning on filtered datasets. From this perspective, we show that SFT is optimizing a lower bound of the RL objective and propose simple modifications that lead to improved variants. First, we confirm that SFT with data sampling proportional to quality scores can form a tighter bound on the RL objective for optimizing human preferences. Second, using importance weighting, iw-SFT, can further tighten the bound with respect to the RL objective leading to strong results for training LLMS to reason. We also demonstrate the versatility of the approach by applying our method for control tasks.
There are several limitations of our wok: while we demonstrate improvements over SFT we do so only for fine-tuning in narrow domains, partly restricted by prohibitively large compute costs for large LLM experiments. More extensive evaluations considering a variety of domains for which reasoning is known to improve LLMs could be considered in the future. Furthermore both for the language modeling and control domains our results do not bridge the gap to the best existing models in all cases. It remains to be seen whether the approach presented here could reach the same performance given the ’right’ data. Finally, we want to highlight that, as with any method that is used to align ML models to human preferences, we hope our work will help produce more human aligned models, but acknowledge that it could also be used to produce more harmful models (by combining it with adversarial data curation).
## Appendix A Derivation for the iw-SFT objective
For trajectories τ = ( s 0 , a 0 , ⋯ , s T ) ∈ ℬ \tau=(s_{0},a_{0},\cdots,s_{T})\in\mathcal{B} we aim to maximize the expected cumulative reward or the return, R ( τ ) = ∑ t = 0 T r ( t ) R(\tau)=\sum_{t=0}^{T}r(t) , of a given trajectory by the following J ( θ ) \displaystyle J(\theta) = ∫ ℬ p ( τ , θ ) R ( τ ) 𝑑 τ = 𝔼 p ( τ , θ ) [ R ( τ ) ] \displaystyle=\int_{\mathcal{B}}p(\tau;\theta)R(\tau)\mathrm{d}\tau=\mathbb{E}_{p(\tau;\theta)}[R(\tau)] (14) p ( τ , θ ) \displaystyle p(\tau;\theta) = p ( s 0 ) ∏ t = 0 T − 1 p ( s t + 1 | s t , a t ) π ( a t | s t ; θ ) . \displaystyle=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi(a_{t}|s_{t};\theta). (15)
Let’s assume that we have a fixed dataset, { τ 1 , ⋯ τ n } = 𝒟 \{\tau_{1},\cdots\tau_{n}\}=\mathcal{D} where τ ∈ π ref ( τ ) \tau\in\pi_{\mathrm{ref}}(\tau) where π ref ( τ ) > 0 ∀ p ( τ ) > 0 \pi_{\mathrm{ref}}(\tau)>0\quad\forall\quad p(\tau)>0 (for example we can assume π ref \pi_{\mathrm{ref}} as Gaussian), thus we can rewrite J ( θ ) J(\theta) as J ( θ ) = ∫ ℬ π ref ( τ ) p ( τ , θ ) π ref ( τ ) R ( τ ) 𝑑 τ = 𝔼 π ref ( τ ) [ p ( τ , θ ) π ref ( τ ) R ( τ ) ] \displaystyle J(\theta)=\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\mathrm{d}\tau=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\right] (16)
Using the following inequality: x ≥ 1 + log ( x ) , x\geq 1+\log(x), we can make the observation that: J ( θ ) ≥ 𝔼 π ref ( τ ) [ ( 1 + log p ( τ , θ ) π ref ( τ ) ) R ( τ ) ] . \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\left(1+\log\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}\right)R(\tau)\right]. (17)
Note that as π ref \pi_{\mathrm{ref}} is fixed; this bound cannot be tightened. Here we can introduce a distribution q ( τ ) q(\tau) : J ( θ ) = 𝔼 π ref ( τ ) [ q ( τ ) π ref ( τ ) p ( τ , θ ) q ( τ ) R ( τ ) ] . \displaystyle J(\theta)=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\frac{p(\tau;\theta)}{q(\tau)}R(\tau)\right]. (18) Which leads to the relation: J ( θ ) ≥ 𝔼 π ref ( τ ) [ q ( τ ) π ref ( τ ) ( 1 + log p ( τ , θ ) q ( τ ) ) R ( τ ) ] . \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\left(1+\log\frac{p(\tau;\theta)}{q(\tau)}\right)R(\tau)\right]. (19) Note that in this case if q ( τ ) → p ( τ , θ ) q(\tau)\rightarrow p(\tau;\theta) , the bound becomes equality. We note that we can thus optimize the following surrogate objective: J ~ ( θ ) \displaystyle\tilde{J}(\theta) = 𝔼 π ref ( τ ) [ q ( τ ) π ref ( τ ) ( 1 + log p ( τ , θ ) q ( τ ) ) R ( τ ) ] \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\left(1+\log\frac{p(\tau;\theta)}{q(\tau)}\right)R(\tau)\right] (20) = 𝔼 π ref ( τ ) [ q ( τ ) π ref ( τ ) log p ( τ , θ ) R ( τ ) ] + cst . \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)R(\tau)\right]+\mathrm{cst}. (21) We can drop the constant terms from this objective without loss of generality. From this we can make the following observation where we change our cumulative reward into a filter function; the objective becomes importance weighted maximum likelihood optimization J ~ ( θ ) \displaystyle\tilde{J}(\theta) = 𝔼 π ref ( τ ) [ q ( τ ) π ref ( τ ) log p ( τ , θ ) 𝕀 ( R ( τ ) ) ] \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)\mathbb{I}(R(\tau))\right] (22) ≈ ∑ τ ∈ 𝒟 + q ( τ ) π ref ( τ ) log p ( τ , θ ) \displaystyle\approx\sum_{\tau\in\mathcal{D}^{+}}\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta) (23) ≈ ∑ τ ∈ 𝒟 + 1 n ∑ i n q i ( τ ) π ref ( τ ) log p ( τ , θ ) , \displaystyle\approx\sum_{\tau\in\mathcal{D}^{+}}\frac{1}{n}\sum_{i}^{n}\frac{q_{i}(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta), (24) note that, here 𝒟 + \mathcal{D}^{+} is the set of data points with positive advantage in our fixed dataset 𝒟 \mathcal{D} . When q = π ref q=\pi_{\mathrm{ref}} , this collapses to maximum likelihood over the positive examples.
## Appendix B Detailed algorithm
A detailed algorithm listing of our method is given in Algorithm 1 .
## Appendix C Hyperparameters for LLM experiments