# Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

###### Abstract

Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.

## 1 Introduction

Post-training of large language models (LLMs) is an increasingly important component in modern large model training pipelines since it can help bring out latent capabilities dormant in pre-trained LLMs. Whether post-training should be performed via reinforcement learning (RL) or instead use simple supervised finetuning (SFT), and how these two steps can be combined, is a hot topic in the community. Generally, Reinforcement learning (RL) from human [ Ouyang et al., 2022 , Ziegler et al., 2019 ] or AI feedback [ Lee et al., 2024 ] has become a crucial step in aligning large language models (LLMs) to better follow human intent, such as instructions, or to steer models to generate outputs that are more helpful and factual [ Bai et al., 2022 ] . Recently, it has been established that RL is an essential part for eliciting reasoning capabilities in LLMs as exeplified by the recent OpenAI o1 and DeepSeek-R1 [ Guo et al., 2025 ] models.

Unfortunately, RL is difficult to interpret and tune. As a result there is a large body of research aiming to further stabilize or simplify existing RL algorithms (see e.g. Shao et al. [2024] , Ahmadian et al. [2024] , Li et al. [2023] ). Many recent post-training approaches even attempt to side-step RL training entirely – e.g. by directly training LLMs to maximize preferences such as in DPO Rafailov et al. [2024] or other RL-free approaches [ Abdolmaleki et al., 2024 ] . Intriguingly, some recent work [ Muennighoff et al., 2025 , Wang et al., 2025 ] goes as far as showing that one can improve reasoning in LLMs via SFT on a carefully filtered/curated dataset and then scaling inference compute.

In this paper, we draw a on a known connection between RL as inference [ Peters and Schaal, 2007 , Kober and Peters, 2008 , Levine, 2018 ] and (iterative) maximum likelihood supervised learning that we believe is underexplored. As we show, this perspective suggests that when we perform SFT on filtered data we are in fact optimizing for a loose lower bound on the RL objective in a sparse reward setting (induced by the filtering of the data). By taking a fresh look at SFT from this perspective, we show that this bound can be tightened significantly in two ways: by sampling data proportional to quality scores where they are available, and via importance re-weighting of the maximum likelihood SFT objective. Intuitively, this re-weighting can also be seen as a way of performing ‘adaptive-filtering’ since the reweighted data points should place increasingly heavier weights on ’preferred’ data points. The resulting small modifications to SFT – which we refer to as SFT from quality sampled data (SFT(Q)) and importance weighted SFT (iw-SFT) respectively – are easy to implement and can match or surpass SFT in many settings. Our experiments show that when optimizing with iw-SFT on curated data, reasoning naturally emerges in LLMs with no need to further scale inference compute (e.g. via some form of budget forcing [ Muennighoff et al., 2025 ] ); leading to high performance on benchmarks such as AIME (66.7%) or GPQA (64.1%) outperforming standard SFT on the same data. We further show the generality by adapting our method to train continuous control policies in an offline RL setting. Our contributions are as follows: • We demonstrate that supervised fine tuning is optimizing a lower bound on the RL objective in the sparse reward setting and introduce an improved, weighted, variant of SFT, iw-SFT.

• We perform extensive experiments, including training LLMs for reasoning (evaluating on AIME 2024 and MATH500 [ Hendrycks et al., 2021 ] ), outperforming the state-of-the-art for training open-models on open data Muennighoff et al. [2025] . We also show how iw-SFT can improve upon BC in offline RL for control on the D4RL benchmark [ Fu et al., 2020 ] .

## 2 Related Work

Our work connects several different fields of study which are all closely related. Several works have previously considered connections between supervised fine-tuning and RL. For example, methods from the literature on reward weighted regression [ Dayan and Hinton, 1997 , Peters and Schaal, 2007 ] (RWR) for RL have been adopted to improve supervised learning in cases where reward functions are available; e.g. in the literature on iterative maximum likelihood [ Ni et al., 2023 , Agarwal et al., 2019 , Norouzi et al., 2016 ] . Most related to our approach, recent work has considered reward weighted iterative filtering (or rejection sampling) for supervised fine-tuning [ Yuan et al., 2024 , Du et al., 2025 , Gulcehre et al., 2023 ] , showing success in fine-tuning LLMs. Among them the ReST family of methods [ Gulcehre et al., 2023 , Singh et al., 2024 ] can be understood as iteratively optimizing the SFT bound we discuss below (though without importance weighting). In a related direction, recently, DeepSeekMath [ Shao et al., 2024 ] also drew on the connection between expectation maximization, RWR and supervised learning, treating methods such SFT, RFT, DPO, PPO, GRPO as maximizing the obective: 𝔼 ( q , o ) ∼ 𝒟 ​ ( 1 | o | ​ ∑ | o | G ⁡ ( q , o , r ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t | q , o < t ) ) , \mathbb{E}_{(q,o)\sim\mathcal{D}}\left(\frac{1}{|o|}\sum^{|o|}G(q,o,r)\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right), where G G denotes generalized returns.

In this work, we build on a similar connection between SFT and RL. In particular, we realize that by following prior work on RL as inference (see e.g. Dayan and Hinton [1997] , Peters and Schaal [2007] , Peters et al. [2010] , Levine [2018] ) we can interpret SFT as optimizing a lower bound to the RL objective. In particular we build on a bound that is a special case of a general form developed in Roux [2016a] . In this work, the authors show that the well known PoWER [ Kober and Peters, 2008 ] and reward weighted regression (RWR) [ Dayan and Hinton, 1997 , Peters and Schaal, 2007 ] algorithms can be derived by finding a lower bound to the RL objective, and can be generalized to an iterative scheme. Follow-up work [ Roux, 2016b ] also considered the applicability of this loss to supervised classification, albeit in a different way than what we consider here. The authors consider introducing an iterated supervised learning scheme which leads to optimizing a tighter bound to the classification accuracy than the commonly used negative log likelihood loss. In contrast, we here are concerned with fine-tuning models on curated (or filtered) datasets that were originally drawn from some reference distribution.

In the literature one can also find general discussions on training generative models on self-generated data after human curation e.g. in Solaiman and Dennison [2021] , Ferbach et al. [2024] . Most relevant to the ideas presented here is recent work on interpreting generative models that consume their own, human curated, data as optimizing human preferences via RLHF [ Ferbach et al., 2024 ] . This work again invokes the well known connections between SFT and EM based policy search.

Our connection between RL and SFT is focused on the sparse reward regime. As a result, it is closely related to other works that aim to improve training on preference data. This includes different RLHF methods such as Ouyang et al. [2022] , Ziegler et al. [2019] . As well as methods which bypass explicit RL such as DPO [ Rafailov et al., 2024 ] or expectation maximization [ Abdolmaleki et al., 2024 ] .

In the literature on RL for control, many recent works have explored the connection between expectation maximization and KL regularized policy optimization [ Abdolmaleki et al., 2018 , Abdolmaleki et al., 2021 , Peng et al., 2019 , Nair et al., 2020a , Schulman et al., 2017 ] . Similar to these, our methodology can be used both in the off-policy or online setting. In this work, we mostly focus on the offline setting. For our experimentation on MuJoCo Locomotion tasks, we compare with highly performant offline RL algorithms such as AWAC [ Nair et al., 2020a ] and IQL [ Kostrikov et al., 2021 ] as well as top-K percentile based behavior cloning (BC) [ Brandfonbrener et al., 2022 , Chen et al., 2021 ] .

## 3 SFT on Curated Data as Reinforcement Learning

Reinforcement Learning (RL) is the predominant approach for improving alignment and reasoning in LLMs; as well as for training agents in (continuous) control settings. RL, however, is often compute and data intensive and hard to tune. In contrast, SFT on curated (or filtered) datasets is stable, requires no change in the loss-function between pre-training and fine-tuning and can be surprisingly effective.

Here, we make a connection between these two algorithms: SFT and RL. Specifically, we show that SFT on filtered data optimizes a lower bound on the RL objective for a sparse reward task induced by the curation/filtering process. From this perspective we can observe that this lower bound becomes looser as the model moves further away from the expert that generated the data (the reference policy). Interestingly, following existing literature on bounding the RL objective [ Roux, 2016a ] we can show that one can introduce an adaptive (importance) reweighting scheme (iw-SFT) that asymptotically maximizes the RL objective as the bound tightens over the course of training. As we will see this simple change from SFT to iw-SFT not only theoretically narrows the gap between SFT and RL but also practically can achieve better results in the same settings that SFT excels in.

### 3.1 Notation and Background

To connect RL and SFT we formalize the general RL problem and then show how SFT can be seen as maximizing a lower bound on this objective; specializing a well known bound from the RL literature [ Peters and Schaal, 2007 , Peters et al., 2010 , Roux, 2016b ] to the sparse / binary reward setting.

#### Notation

In the following, s ∈ 𝒮 s\in\mathcal{S} and a ∈ 𝒜 a\in\mathcal{A} denote states and actions in a Markov Decision Process (MDP) respectively. We define a trajectory as a sequence of states and actions: τ = ( s 0 , a 0 , ⋯ , s T ) ∈ 𝒮 × 𝒜 ⋯ 𝒮 = ℬ \tau=(s_{0},a_{0},\cdots,s_{T})\in\mathcal{S}\times\mathcal{A}\cdots\mathcal{S}=\mathcal{B} . The actions are drawn from a parametric policy π ⁡ ( a t | s t , θ ) \pi(a_{t}|s_{t},\theta) (a probability distribution over actions) and the dynamics of the trajectory through time depend on the transition probabilities p ⁡ ( s t + 1 | s t , a t ) p(s_{t+1}|s_{t},a_{t}) . The reward function is dependent on the state and action r ⁡ ( s t , a t ) r(s_{t},a_{t}) , we simplify this to r ⁡ ( t ) r(t) ; where r ⁡ ( T ) r(T) then corresponds to the terminal reward. We aim to maximize the discounted cumulative reward or the return, R ⁡ ( τ ) = ∑ t = 0 T γ ⁡ ( t ) ​ r ​ ( t ) R(\tau)=\sum_{t=0}^{T}\gamma(t)r(t) , with discount factor γ \gamma . Note that in a sparse reward setting where we only observe a binary terminal reward r ⁡ ( T ) r(T) and assume ∀ t ≠ T : r ⁡ ( t ) = 0 \forall t\neq T:r(t)=0 and use γ = 1 \gamma=1 the return simplifies to 𝕀 ⁡ ( R ⁡ ( τ ) ) = ∑ t = 0 T r ⁡ ( t ) = r ⁡ ( T ) \mathbb{I}(R(\tau))=\sum_{t=0}^{T}r(t)=r(T) (where 𝕀 \mathbb{I} denotes an indicator function). Often such a sparse reward setting is induced by checking if a continuous function exceeds a threshold at the last timestep.

#### Reinforcement learning

Using the above, we can define the probability of a given trajectory as: p π ​ ( τ , θ ) \displaystyle p_{\pi}(\tau;\theta) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ​ ( a t | s t ; θ ) . \displaystyle=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi(a_{t}|s_{t};\theta). (1) We further note that, in the language modeling setting the states s s and actions a a are both given by the tokens emitted by the model – i.e. 𝒮 = 𝒜 \mathcal{S}=\mathcal{A} – and the transition dynamics are deterministic. Thus, the policy directly determines the next state and we can equivalently write p ⁡ ( τ , θ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 π ⁡ ( s t + 1 | s 0 , ⋯ , s t ; θ ) p(\tau;\theta)=p(s_{0})\prod_{t=0}^{T-1}\pi(s_{t+1}|s_{0},\cdots,s_{t};\theta) , which is the autoregressive model probability of the text tokens. Note that in this case the transitions are not Markovian and we thus condition on all previous states but all derivations below apply without loss of generality in both cases.

The objective of training a policy π \pi via reinforcement learning (RL) is defined as maximizing the expected cumulative return, where the expectation is over observed trajectories: max θ ​ J ​ ( θ ) = ∫ ℬ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 p ⁡ ( τ , θ ) ​ [ R ⁡ ( τ ) ] . \displaystyle\mathrm{max}_{\theta}~J(\theta)=\int_{\mathcal{B}}p(\tau;\theta)R(\tau)\mathrm{d}\tau=\mathbb{E}_{p(\tau;\theta)}\big[R(\tau)\big]. (2)

To see the connection between RL and SFT let us first assume that we only have access to trajectories τ ∈ π ref \tau\in\pi_{\mathrm{ref}} generated by some reference policy. This could be a human demonstrating desired trajectories/outputs/behaviour or simply a pre-trained model that we query for outputs. To use these trajectories for optimizing the RL objective we have to re-express it in terms of these samples. Here we need to make the assumption that the support of p π p_{\pi} is a subset of π ref \pi_{\mathrm{ref}} . Using importance sampling (see [ Kahn and Marshall, 1953 , Rubinstein and Kroese, 2016 , Ripley, 2009 ] for a review), we can then express our RL objective from Eq. 2 as: J ⁡ ( θ ) \displaystyle J(\theta) = ∫ ℬ π ref ​ ( τ ) ​ p π ​ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 π ref ​ ( τ ) ​ [ p π ​ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ] , \displaystyle=\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{p_{\pi}(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\mathrm{d}\tau=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\frac{p_{\pi}(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\Big], (3) we overload the notation of π ref \pi_{\text{ref}} for simplicity, where the expectation is over the reference π ref ​ ( τ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ref ​ ( a t | s t ) \pi_{\mathrm{ref}}(\tau)=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi_{\mathrm{ref}}(a_{t}|s_{t}) and weight samples via the importance weight p π ​ ( τ , θ ) / π ref ​ ( τ ) \nicefrac{{p_{\pi}(\tau;\theta)}}{{\pi_{\mathrm{ref}}(\tau)}} .

### 3.2 SFT lower bounds RL

We can derive a lower bound on this RL objective via a well known inequality [ Kober and Peters, 2008 , Roux, 2016a ] . Assuming strictly positive rewards, we can utilize the inequality x ≥ 1 + log ⁡ ( x ) x\geq 1+\log(x) applied to the importance weights, to show that (for more details see Appendix A ): J ⁡ ( θ ) ≥ ∫ ℬ π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ​ 𝑑 τ + cst = 𝔼 π ref ​ ( τ ) ​ [ R ⁡ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ] + cst , \displaystyle J(\theta)\geq\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\log p(\tau;\theta)R(\tau)\mathrm{d}\tau+\mathrm{cst}=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[R(\tau)\log p(\tau;\theta)\Big]+\mathrm{cst}, (4) where cst \mathrm{cst} denotes constant terms independent of θ \theta . This is a well known objective in the RL literature [ Dayan and Hinton, 1997 , Peters and Schaal, 2007 , Kober and Peters, 2008 , Roux, 2016a ] often referred to as reward weighted regression. We can relate this objective to supervised learning in two cases (i) we assume access to a dataset of filtered “good data” (e.g. a natural setting in cases where we have success indicators assigned by humans); (ii) we additionally assume access to quality scores for all “good data” (i.e. we assume each τ \tau can be given a quality score on an ordinal scale). For case (i) we can find a relation by assuming binary, sparse, rewards in the RL objective. Then the return simplifies to an indicator function, R ⁡ ( τ ) = 𝕀 ⁡ ( S ⁡ ( τ ) > 0 ) R(\tau)=\mathbb{I}(S(\tau)>0) (where S S is some score function indicating success), collapsing Equation 4 to the maximum likelihood supervised fine tuning objective: J ⁡ ( θ ) ≥ 𝔼 π ref ​ ( τ ) ​ [ 𝕀 ⁡ ( S ⁡ ( τ ) > 0 ) ​ log ⁡ p ⁡ ( τ , θ ) ] = c ref ​ 𝔼 τ ∈ 𝒟 + ​ [ log ⁡ p ⁡ ( τ , θ ) ] ⏟ 𝒥 SFT ​ ( θ ) , \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\mathbb{I}(S(\tau)>0)\log p(\tau;\theta)\Big]=c_{\mathrm{ref}}\underbrace{\mathbb{E}_{\tau\in\mathcal{D}^{+}}\Big[\log p(\tau;\theta)\Big]}_{\mathcal{J}_{\text{SFT}}(\theta)}, (5) where 𝒟 + = { τ i | τ i ∼ π ref ( τ ) , 𝕀 ( S ( τ i ) > 0 ) } i = 1 N \mathcal{D}^{+}=\{\tau_{i}|\tau_{i}\sim\pi_{\mathrm{ref}}(\tau),\mathbb{I}(S(\tau_{i})>0)\}_{i=1}^{N} is a dataset of trajectories sampled from π ref \pi_{\textrm{ref}} and filtered by the binary reward indicator and c ref = 𝔼 π ref ​ ( τ ) ​ [ 𝕀 ⁡ ( S ⁡ ( τ ) > 0 ) ] c_{\mathrm{ref}}=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}[\mathbb{I}(S(\tau)>0)] is constant independent of θ \theta . And we start optimization of J ⁡ ( θ ) J(\theta) by initializing to the reference distribution, setting π θ 0 = π ref ​ ( τ ) \pi_{\theta_{0}}=\pi_{\mathrm{ref}}(\tau) . If π ref \pi_{\mathrm{ref}} is not available in closed form we approximate it by first learning it via standard behavior cloning on the full dataset 𝒟 \mathcal{D} . Thus we can interpret SFT on filtered data as maximizing a lower bound on the RL objective .

Analogously, for case (ii) we can assume rewards R ⁡ ( τ ) = ∑ i = 0 C 𝕀 ⁡ ( S ⁡ ( τ ) > c i ) R(\tau)=\sum_{i=0}^{C}\mathbb{I}(S(\tau)>c_{i}) , with C C (ordinal) quality classes. Then RL simplifies to maximum likelihood of trajectories proportional to quality: J ⁡ ( θ ) ≥ 𝔼 π ref ​ ( τ ) ​ [ ∑ i = 0 C 𝕀 ⁡ ( S ⁡ ( τ ) > c i ) ​ log ⁡ p ⁡ ( τ , θ ) ] = c ref Q ​ 𝔼 τ ∈ 𝒟 Q + ​ [ log ⁡ p ⁡ ( τ , θ ) ] ⏟ 𝒥 SFT(Q) ​ ( θ ) , \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\sum_{i=0}^{C}\mathbb{I}(S(\tau)>c_{i})\log p(\tau;\theta)\Big]=c_{\mathrm{ref}}^{Q}\underbrace{\mathbb{E}_{\tau\in\mathcal{D}_{Q}^{+}}\Big[\log p(\tau;\theta)\Big]}_{\mathcal{J}_{\text{SFT(Q)}}(\theta)}, (6) where 𝒟 Q + = 𝒟 c 1 + ∪ ⋯ ∪ 𝒟 c C + \mathcal{D}_{Q}^{+}=\mathcal{D}_{c_{1}}^{+}\cup\dots\cup\mathcal{D}_{c_{C}}^{+} with 𝒟 c + = { τ i | τ i ∼ π ref ( τ ) , 𝕀 ( S ( τ i ) > c i ) } i = 1 N \mathcal{D}_{c}^{+}=\{\tau_{i}|\tau_{i}\sim\pi_{\mathrm{ref}}(\tau),\mathbb{I}(S(\tau_{i})>c_{i})\}_{i=1}^{N} . Note c ref Q = 𝔼 π ref ​ ( τ ) ​ [ ∑ i = 0 C 𝕀 ⁡ ( S ⁡ ( τ ) > c i ) ] c_{\mathrm{ref}}^{Q}=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}[\sum_{i=0}^{C}\mathbb{I}(S(\tau)>c_{i})] . And we have the relation ∀ i : c i > c i − 1 \forall i:c_{i}>c_{i-1} . Hence, we can interpret SFT over quality curated data as maximizing a lower bound on the RL objective.

It is important to realize, however, that both lower bounds, become looser throughout training, as the distance between p ⁡ ( τ , θ ) p(\tau;\theta) and π ref ​ ( τ ) \pi_{\mathrm{ref}}(\tau) increases. This connection between RL via reward weighted regression and maximum likelihood is not new [ Roux, 2016a , Roux, 2016b , Ferbach et al., 2024 ] but we believe it is underexplored in the context of SFT and RL for large models.

### 3.3 Improving SFT from the perspective of RL

The bound SFT optimizes becomes looser as we train, thus weakening the connection to RL. We show that by re-framing the RL problem we can introduce a tighter bound that only requires a minimal change to the maximum likelihood objective used in SFT and can achieve better results.

To derive this variant we first introduce an auxiliary distribution q ⁡ ( τ ) q(\tau) that we are free to choose. And we will choose it later to trade off the tension between staying close to the reference policy for stable optimization and tightening the bound. Using this auxiliary we can re-write the RL objective as: J ⁡ ( θ ) = ∫ ℬ π ref ​ ( τ ) ​ q ⁡ ( τ ) π ref ​ ( τ ) ​ p ⁡ ( τ , θ ) q ⁡ ( τ ) ​ R ​ ( τ ) ​ 𝑑 τ . \displaystyle J(\theta)=\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\frac{p(\tau;\theta)}{q(\tau)}R(\tau)\mathrm{d}\tau. (7) By applying the inequality x ≥ 1 + log ⁡ ( x ) x\geq 1+\log(x) to x = p ⁡ ( τ , θ ) / q ⁡ ( τ ) x=\nicefrac{{p(\tau;\theta)}}{{q(\tau)}} , as previously considered in [ Roux, 2016b , Kober and Peters, 2008 ] we can obtain an analogous bound to ( 4 ): J ⁡ ( θ ) \displaystyle J(\theta) ≥ ∫ ℬ π ref ​ ( τ ) ​ q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ​ 𝑑 τ + cst = 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ R ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ] + cst . \displaystyle\geq\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)R(\tau)\mathrm{d}\tau+\mathrm{cst}=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\log p(\tau;\theta)\Big]+\mathrm{cst}. For simplicity we drop the constant (wlog). If we assume sparse rewards as above we can further turn this into a training objective over filtered data that results in an importance weighted version of SFT: J ⁡ ( θ ) \displaystyle J(\theta) ≥ 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ 𝕀 ​ ( S ⁡ ( τ ) ) ​ log ⁡ p ⁡ ( τ , θ ) ] = c ref ​ 𝔼 τ ∈ 𝒟 + ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ] ⏟ 𝒥 iw-SFT , \displaystyle\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\mathbb{I}(S(\tau))\log p(\tau;\theta)\Big]=c_{\mathrm{ref}}\underbrace{\mathbb{E}_{\tau\in\mathcal{D}^{+}}\Big[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)\Big]}_{\mathcal{J}_{\text{iw-SFT}}}, (8) where 𝒟 + = { τ i | τ i ∼ π ref ( τ ) , 𝕀 ( S ( τ i ) ) > 0 } i = 1 N \mathcal{D}^{+}=\{\tau_{i}|\tau_{i}\sim\pi_{\mathrm{ref}}(\tau),\mathbb{I}(S(\tau_{i}))>0\}_{i=1}^{N} and c ref = 𝔼 π ref ​ ( τ ) ​ [ 𝕀 ⁡ ( S ⁡ ( τ ) > 0 ) ] c_{\mathrm{ref}}=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}[\mathbb{I}(S(\tau)>0)] . Here, the auxiliary distribution q ⁡ ( τ ) q(\tau) is now used to importance weight the examples in the dataset. And an analogous bound for quality scored data 𝒟 Q + = 𝒟 1 + ∪ ⋯ ∪ 𝒟 C + \mathcal{D}_{Q}^{+}=\mathcal{D}_{1}^{+}\cup\dots\cup\mathcal{D}_{C}^{+} as in Eq. ( 6 ) can be found: J ⁡ ( θ ) \displaystyle J(\theta) ≥ c ref Q ​ 𝒥 iw-SFT(Q) = 𝔼 τ ∈ 𝒟 Q + ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ] . \displaystyle\geq c_{\mathrm{ref}}^{Q}\mathcal{J}_{\text{iw-SFT(Q)}}=\mathbb{E}_{\tau\in\mathcal{D}_{Q}^{+}}\Big[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)\Big]. (9) Critically, both objectives have the property that as q ⁡ ( τ ) → p ⁡ ( τ , θ ) q(\tau)\rightarrow p(\tau;\theta) we have 𝒥 iw-SFT ​ ( θ ) → J ⁡ ( θ ) \mathcal{J}_{\text{iw-SFT}}(\theta)\rightarrow J(\theta) . Namely, as q q gets closer to our policy distribution, the bound tightens. However, at the same time, tightening the bound will increase variance in the importance weights and it is thus not generally safe to just set q ⁡ ( τ ) = p ⁡ ( τ , θ ) q(\tau)=p(\tau,\theta) . That being said, and in contrast to standard importance weighting for policy gradients [ Jiang and Li, 2016 , Metelli et al., 2018 ] , we are free to choose q q to smooth out the bound. We can thus side-step common problems with large variance in the importance weights/blow-up by enforcing importance weights to not be too extreme. Optimizing 𝒥 iwSFT \mathcal{J}_{\text{iwSFT}} or 𝒥 iwSFT(Q) \mathcal{J}_{\text{iwSFT(Q)}} then can be achieved by sampling from the filtered data distribution and computing importance weights on the fly (requiring one extra reference model to be kept in memory) as shown in Algorithm 1 in the appendix.

### 3.4 The importance of choosing q ⁡ ( τ ) q(\tau)

As mentioned, we are free to choose q ⁡ ( τ ) q(\tau) in our method. We would generally like to achieve q ⁡ ( τ ) → p ⁡ ( τ , θ ) q(\tau)\rightarrow p(\tau;\theta) over the course of training so that the bound becomes tight. However, as q ⁡ ( τ ) q(\tau) moves away from π ref \pi_{\textrm{ref}} the variance of the bound (and thus the variance of its gradient) increases. As a result we want to bound the KL divergence K L ( q ( τ ) ∥ π ref ( τ ) ) < β KL(q(\tau)\|\pi_{\textrm{ref}}(\tau))<\beta but need to trade this off with tightening the bound; a well known problem when dealing with importance sampling based objectives.

In contrast to standard off-policy policy gradient algorithms, choosing q ⁡ ( τ ) q(\tau) such that it strikes this trade-off (e.g. via clipping or other means as discussed below) will not bias our algorithm–but may nonetheless lead to different optima and thus better (or worse) policies. In practice, we choose q ⁡ ( τ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ​ ( a t | s t ; θ q ) \displaystyle q(\tau)=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi(a_{t}|s_{t};\theta_{q}) (10) where θ q → θ \theta_{q}\rightarrow\theta towards the end of training such that q ⁡ ( τ ) → p π ​ ( τ , θ ) q(\tau)\rightarrow p_{\pi}(\tau;\theta) .

For controlling the variance, we consider two different variants of bounding the difference between q ⁡ ( τ ) q(\tau) and π ref \pi_{\textrm{ref}} by controlling the importance weights. In either variant we set θ q \theta_{q} to be a time-lagged version of θ \theta , the parameters we are optimizing (see experiments section for details) and then constrain the importance weights to control the evolution of p ⁡ ( τ , θ ) p(\tau,\theta) during training (and hence indirectly q ⁡ ( θ ) q(\theta) ). We note that for each variant we always consider importance weights over full trajectories .

#### Clipping per step importance weights

A simple way to control the importance weights is to simply bound the per step importance weights to be within a certain range. That is we choose: q ⁡ ( τ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 ( p ⁡ ( s t + 1 | s t , a t ) ​ clip ​ ( π ⁡ ( a t | s t ; θ ) , α min , α max ) CLOSE q(\tau)=p(s_{0})\prod_{t=0}^{T-1}(p(s_{t+1}|s_{t},a_{t})\text{clip}(\pi(a_{t}|s_{t};\theta),\alpha_{\text{min}},\alpha_{\text{max}}) , where we typically set α min \alpha_{\text{min}} and α max \alpha_{\text{max}} fixed, i.e. in the language model experiments we will choose ( α min , α max ) = ( 1 − 0.8 , 1 + 0.8 ) (\alpha_{\text{min}},\alpha_{\max})=(1-0.8,1+0.8) . We will then do an overall clipping of the importance weight: clip ​ ( q ⁡ ( τ ) / π ref ​ ( τ ) , β min , β max ) \text{clip}(q(\tau)/\pi_{\text{ref}}(\tau),\beta_{\min},\beta_{\max}) .

#### Smoothing importance weights

As a second, more general, option we consider ensuring low variance by smoothing the importance weights on the trajectory level. In particular we consider transforming the weights as: q ⁡ ( τ ) π ref ​ ( τ ) = exp ⁡ ( ∑ i g ⁡ ( ρ i ) ) , ρ i = log ⁡ ( π q ​ ( a i | s i ) ) − log ⁡ ( π ref ​ ( a i | s i ) ) \displaystyle\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}=\exp\left(\sum_{i}g(\rho_{i})\right),\quad\rho_{i}=\log(\pi_{q}(a_{i}|s_{i}))-\log(\pi_{\mathrm{ref}}(a_{i}|s_{i})) (11) Here, i i denotes the i i -th token in the sequence τ \tau , and g ⁡ ( x ) g(x) is a monotonically increasing function with respect to x x . We use a simple temperature function for continuous control experiments 1 1 1 But the above clipping can also be expressed using this formulation as g ⁡ ( x ) = clip ⁡ ( x , min , max ) g(x)=\mathrm{clip}(x,\min,\max) . : g ⁡ ( x ) = k ​ x g(x)=kx , where k k is a hyperparameter, for all our experiments in continuous control domains. By setting k = 1 / / ​ ∑ i M i k=\nicefrac{{1}}{{/}}{\sum_{i}M_{i}} (where M M is the token mask for a transformer LLM) we can implement using the averaged log importance weight. We instead use k = α ​ ∑ i M i k=\alpha\sum_{i}M_{i} which gives us an extra degree of freedom to smooth out the importance weights. This way we can smoothly move from no importance weighting ( α → 0 \alpha\rightarrow 0 ) to more and more “peaked” weights. We ablate the choice of α \alpha in the supplementary material.

### 3.5 A Toy Example to build Intuition for SFT on curated/filtered data

We consider a toy example to get an intuition for when the bound optimized via SFT is problematic–and thus when the importance weighted version can be beneficial. We construct this toy example to answer the following questions: Q1: Does the loose lower bound lead to suboptimal policies? Q2: What is the effect of the importance weighting? Can it help learn better policies? Q3: What are the failure modes of the importance weighted variant, and how can they be avoided?

Consider a simple bandit optimization problem in which an agent has a choice of two actions 𝒜 = { pull-left , pull-right } \mathcal{A}=\{\text{pull-left},\text{pull-right}\} . The episode terminates after picking a single action. R ⁡ ( τ ) R(\tau) is either 0 or 1; the probability distribution of R ⁡ ( τ ) R(\tau) is defined by: ℙ ⁡ ( R ⁡ ( τ ) = 1 ) = { 0.5 if ​ τ = pull-left 1 if ​ τ = pull-right \displaystyle\mathbb{P}(R(\tau)=1)=\left\{\begin{array}[]{l l}0.5&\text{if }\tau=\text{pull-left}\\ 1&\text{if }\tau=\text{pull-right}\end{array}\right.

Thus the optimal strategy is to always pull the right arm. We assume that we can observe the behavior of the system only by watching (collecting data from) a reference policy π ref ​ ( a ) = 𝒰 ⁡ { pull-left , pull-right } \pi_{\mathrm{ref}}(a)=\mathcal{U}\{\text{pull-left},\text{pull-right}\} that picks between the two arms uniformly at random.

We collect a dataset 𝒟 \mathcal{D} using this policy which will contain 50 % 50\% pull-left and 50 % 50\% pull-right actions. Filtering this dataset by the binary success reward to produce 𝒟 + = { τ i | τ i ∼ π ref ( τ ) , R ( τ i ) = 1 } i = 1 N \mathcal{D}^{+}=\{\tau_{i}|\tau_{i}\sim\pi_{\mathrm{ref}}(\tau),R(\tau_{i})=1\}_{i=1}^{N} then results in a dataset with 2 times more pull-right than pull-left actions. Subsequently training via SFT on this data results in a policy π SFT \pi_{\text{SFT}} that chooses pull-left one third of the time and pull-right two thirds of the time. The resulting average reward is 5 / 6 5/6 . See Figure 1 for a numerical experiment. This policy improves on the reference policy, but is also not optimal. The key problem here is that the SFT procedure cannot incorporate knowledge from failures. For Q1 : while SFT on curated data can be seen as optimizing a lower bound on the RL objective and will improve on the reference policy it can result in sub-optimal behavior.

Suppose we introduce importance weights (according to Equation ( 8 )) where q ⁡ ( τ ) = π ⁡ ( τ , θ ) q(\tau)=\pi(\tau;\theta) . Now over the course of training, we will adaptively put more weight on pull-right compared to pull-left actions until we recover a policy that will pull-right every time. This can also be seen in Figure 1 . Thus importance weighting can recover information from failures . Indeed, if we train with iw-SFT on this data, we recover the optimal policy of just choosing pull-right; answering Q2 : iw-SFT can “recover” information from discarded samples.

Finally regarding the failure modes ( Q3 ): while we recover the optimal policy with iw-SFT in this case, it does lead to collapse of the policy to a single action (which happens to be correct here). In general, the importance weights determine how far away from the reference policy we allow π θ \pi_{\theta} to deviate. If we place no bound on the importance weights one can easily see that in a high-dimensional case, they might grow out of bounds (the infinite variance problem [ Andradóttir et al., 1995 ] ) or lead to us discarding too much of the filtered data. As mentioned above, this can be mitigated by forcing q ⁡ ( τ ) q(\tau) to stay close to π ref \pi_{\mathrm{ref}} –e.g. via clipping.

## 4 Experiments

We evaluate our proposed improvements to SFT, and compare to existing RL algorithms in two settings: 1) training LLMs for reasoning, 2) training control policies from offline data.

### 4.1 Training LLMs for reasoning via SFT on curated data

For our initial set of experiments we consider fine-tuning an open-source LLM (Qwen2.5-32B-Instruct [ Yang et al., 2024 ] ) to improve its reasoning capabilities for challenging maths problems. To finetune the model, we use the S1.1K dataset introduced in Muennighoff et al. [2025] . This is a heavily curated dataset consisting of 1,000 1,000 high quality reasoning traces. These examples were created by first collecting a set of 59K candidate question/answer pairs from existing datasets [ LI et al., 2024 , of America, 2024 , Gao et al., 2024 , Zhong et al., 2023 ] and generating reasoning traces and responses for the respective questions using Gemini Flash [ Google, 2024 ] . The dataset is then decontaminated against the test datasets and filtered down to 1k examples by quality and diversity as described in Muennighoff et al. [2025] . This is thus a great example of a high-quality, curated dataset.

The hyperparameters of our model are set the same as in [ Muennighoff et al., 2025 ] (we use cosine annealing for the learning rate and AdamW [ Loshchilov et al., 2017 ] ) details are given in the appendix. We also use g ⁡ ( x ) = 0.1 × clip ⁡ ( x , 0.2 , 1.8 ) g(x)=0.1\times\mathrm{clip}(x,0.2,1.8) to smooth/clip the importance weights. Given that we do not have access to the model weights for Gemini Flash (which was used to generate the reasoning traces), or its original training data, we also have to rely on an approximation for the reference distribution π ref \pi_{\mathrm{ref}} in the importance weights. Ideally, we would pick a model that is “close” to Gemini-Flash as an approximation. We tested multiple models at the 32B parameter level, comparing their summed log-probabilities over examples from the dataset and find that they assign similar likelihood (albeit giving different top-k next token predictions) and settled on simply using the model we start from as the reference model such that q , π ref q,\pi_{\mathrm{ref}} and π θ \pi_{\theta} are all close at the beginning of training.

All results and comparisons to models from the literature are shown in Table 1 . In the table we also include an ablation of our method which uses importance weighting but on the token level rather than sequence level; this result is shown as iw-SFT(per-step iw). As can be seen from the table, our method outperforms the original s1 results (which also performs SFT, starting from the same checkpoint) on AIME 2024 as well as GPQA Diamond while matching it on MATH 500. Interestingly, we find that iw-SFT does not require budget forcing (a method introduced in Muennighoff et al. [2025] to force the model to keep thinking by injecting “Wait” tokens) to reach its peak performance. This suggests that the importance weighted objective itself can help extract information from the training data, foregoing the need for extra test time optimization. For ablations please see Appendix G . The model that only uses per-step importance weights also reaches high-performance across benchmarks, albeit losing 2 percent compared to full sequence importance weighting on GPQA Diamond.

Overall, training with iw-SFT on the curated dataset S1.1K introduced by Muennighoff et al. [2025] surpasses training with SFT only (s1 / s1.1 in the table) by up to 7% depending on the AIME benchmark. This sets the state-of-the art for open models trained on open data; although better closed models exist as shown in the comparison table.

### 4.2 Offline RL for Continuous Control

To probe for the general applicability of our approach, we turn to learning policies for continuous control using the offline RL benchmark datasets from D4RL [ Fu et al., 2020 ] . We consider the standard locomotion tasks from the benchmark (halfcheetah, hopper, walker2d) both in the setting where data comes from expert policies (Expert (v2) below) and in the setting where data was extracted from a standard RL training run (Med. Replay (v2)) and is thus more diverse.

Since all trajectories in the datasets come with reward information (which we can interpret as unambiguous quality scores), we consider the quality sampling setting. We first filter the data to only contain the top 10 % 10\% of trajectories and then turn the cumulative episode rewards into ordinal quality annotations by binning them into 3 overlapping bins, see Appendix E for details. Our model is a simple 3 layer MLP policy as listed in Appendix F and instead of updating the importance weighting distribution q q at discrete time-points we use exponential weight averaging θ q = α ​ θ q + ( 1 − α ) ​ θ \theta_{q}=\alpha\theta_{q}+(1-\alpha)\theta during each step of the optimization (this is similar to how target networks in continuous control RL algorithms are often updated [ Lillicrap et al., 2015 , Fujimoto and Gu, 2021 ] ). We smooth the importance weights with g ⁡ ( x ) = k ​ x g(x)=kx . We obtain a reference distribution π ref \pi_{\text{ref}} by first performing simple behavior cloning on all data (e.g. optimizing Equation ( 5 ) on all data instead of filtered data).

Results are given in Table 2 . We evaluated SFT, SFT(Q) and iw-SFT(Q) and list results alongside baselines from Kostrikov et al. [2021] consisting of behavior cloning (BC), which is equivalent to SFT but with parameters initialized randomly. BC on the top-10% of data (BC 10%) as well as four state-of-the-art offline RL algorithms (AWAC [ Nair et al., 2020b ] , TD3+BC [ Fujimoto and Gu, 2021 ] , CQL [ Kumar et al., 2020 ] , IQL [ Kostrikov et al., 2021 ] ). First, we can observe that SFT performs similar to BC (10%), suggesting that initializing to a policy trained on all data does not lead to meaningful generalization benefits in these tasks. Second, SFT(Q) clearly outperforms BC, BC(10%) and SFT in all tasks. This shows that training with quality scores, where available, is indeed helpful. Furthermore, while there are algorithms achieving better results in some domains, SFT(Q) is competitive with strong RL baselines in all settings; confirming that the bound optimized by SFT(Q) can meaningfully optimize for the RL objective. Finally iw-SFT(Q) further leads to a slight improvement over SFT(Q), albeit not as large of an improvement as in the LLM case; suggesting that perhaps this data (which stems from trained policies) is too prototypical, making it hard to outperform quality filtering. Overall, we show that our improved SFT variants can be competitive to offline RL for continuous control.

### 4.3 Fine-tuning control policies with little data and noisy quality scores

In a final experiment we test whether fine-tuning can be helpful for control tasks if less prototypical data is available. For this purpose we consider the the Franka kitchen task [ Fu et al., 2020 ] , in which a robot has to interact with a number of items in a simulated kitchen in a specific order. We first pre-train a BC policy on the “partial” part of the dataset (containing only some successful trajectories) and then fine-tune on the top 5 % 5\% trajectories from the “complete” dataset of trajectories (which are all successful but are not enough trajectories to learn the task from) and top 5 % 5\% of data from the “mixed” part of the dataset (which solve parts of the task but are not fully successful). Results are depicted in Table 3 . As can be seen, BC on the complete data is reasonably successful while training only on the 5 % 5\% filtered data fails. Fine-tuning on this data maintains performance, and closes the gap to BC on complete data significantly. Weighting by quality (SFT(Q)) closes the gap further and iw-SFT(Q) finally is able to recover performance.

## 5 Conclusion and Discussion

In this work, we draw on a connection between reinforcement learning and supervised fine tuning on filtered datasets. From this perspective, we show that SFT is optimizing a lower bound of the RL objective and propose simple modifications that lead to improved variants. First, we confirm that SFT with data sampling proportional to quality scores can form a tighter bound on the RL objective for optimizing human preferences. Second, using importance weighting, iw-SFT, can further tighten the bound with respect to the RL objective leading to strong results for training LLMS to reason. We also demonstrate the versatility of the approach by applying our method for control tasks.

There are several limitations of our wok: while we demonstrate improvements over SFT we do so only for fine-tuning in narrow domains, partly restricted by prohibitively large compute costs for large LLM experiments. More extensive evaluations considering a variety of domains for which reasoning is known to improve LLMs could be considered in the future. Furthermore both for the language modeling and control domains our results do not bridge the gap to the best existing models in all cases. It remains to be seen whether the approach presented here could reach the same performance given the ’right’ data. Finally, we want to highlight that, as with any method that is used to align ML models to human preferences, we hope our work will help produce more human aligned models, but acknowledge that it could also be used to produce more harmful models (by combining it with adversarial data curation).

## Appendix A Derivation for the iw-SFT objective

For trajectories τ = ( s 0 , a 0 , ⋯ , s T ) ∈ ℬ \tau=(s_{0},a_{0},\cdots,s_{T})\in\mathcal{B} we aim to maximize the expected cumulative reward or the return, R ⁡ ( τ ) = ∑ t = 0 T r ⁡ ( t ) R(\tau)=\sum_{t=0}^{T}r(t) , of a given trajectory by the following J ⁡ ( θ ) \displaystyle J(\theta) = ∫ ℬ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 p ⁡ ( τ , θ ) ​ [ R ⁡ ( τ ) ] \displaystyle=\int_{\mathcal{B}}p(\tau;\theta)R(\tau)\mathrm{d}\tau=\mathbb{E}_{p(\tau;\theta)}[R(\tau)] (14) p ⁡ ( τ , θ ) \displaystyle p(\tau;\theta) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ​ ( a t | s t ; θ ) . \displaystyle=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi(a_{t}|s_{t};\theta). (15)

Let’s assume that we have a fixed dataset, { τ 1 , ⋯ τ n } = 𝒟 \{\tau_{1},\cdots\tau_{n}\}=\mathcal{D} where τ ∈ π ref ​ ( τ ) \tau\in\pi_{\mathrm{ref}}(\tau) where π ref ​ ( τ ) > 0 ∀ p ⁡ ( τ ) > 0 \pi_{\mathrm{ref}}(\tau)>0\quad\forall\quad p(\tau)>0 (for example we can assume π ref \pi_{\mathrm{ref}} as Gaussian), thus we can rewrite J ⁡ ( θ ) J(\theta) as J ⁡ ( θ ) = ∫ ℬ π ref ​ ( τ ) ​ p ⁡ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 π ref ​ ( τ ) ​ [ p ⁡ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ] \displaystyle J(\theta)=\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\mathrm{d}\tau=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\right] (16)

Using the following inequality: x ≥ 1 + log ⁡ ( x ) , x\geq 1+\log(x), we can make the observation that: J ⁡ ( θ ) ≥ 𝔼 π ref ​ ( τ ) ​ [ ( 1 + log ⁡ p ⁡ ( τ , θ ) π ref ​ ( τ ) ) ​ R ​ ( τ ) ] . \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\left(1+\log\frac{p(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}\right)R(\tau)\right]. (17)

Note that as π ref \pi_{\mathrm{ref}} is fixed; this bound cannot be tightened. Here we can introduce a distribution q ⁡ ( τ ) q(\tau) : J ⁡ ( θ ) = 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ p ⁡ ( τ , θ ) q ⁡ ( τ ) ​ R ​ ( τ ) ] . \displaystyle J(\theta)=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\frac{p(\tau;\theta)}{q(\tau)}R(\tau)\right]. (18) Which leads to the relation: J ⁡ ( θ ) ≥ 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ ( 1 + log ⁡ p ⁡ ( τ , θ ) q ⁡ ( τ ) ) ​ R ​ ( τ ) ] . \displaystyle J(\theta)\geq\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\left(1+\log\frac{p(\tau;\theta)}{q(\tau)}\right)R(\tau)\right]. (19) Note that in this case if q ⁡ ( τ ) → p ⁡ ( τ , θ ) q(\tau)\rightarrow p(\tau;\theta) , the bound becomes equality. We note that we can thus optimize the following surrogate objective: J ~ ​ ( θ ) \displaystyle\tilde{J}(\theta) = 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ ( 1 + log ⁡ p ⁡ ( τ , θ ) q ⁡ ( τ ) ) ​ R ​ ( τ ) ] \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\left(1+\log\frac{p(\tau;\theta)}{q(\tau)}\right)R(\tau)\right] (20) = 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ] + cst . \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)R(\tau)\right]+\mathrm{cst}. (21) We can drop the constant terms from this objective without loss of generality. From this we can make the following observation where we change our cumulative reward into a filter function; the objective becomes importance weighted maximum likelihood optimization J ~ ​ ( θ ) \displaystyle\tilde{J}(\theta) = 𝔼 π ref ​ ( τ ) ​ [ q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) ​ 𝕀 ​ ( R ⁡ ( τ ) ) ] \displaystyle=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\left[\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta)\mathbb{I}(R(\tau))\right] (22) ≈ ∑ τ ∈ 𝒟 + q ⁡ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) \displaystyle\approx\sum_{\tau\in\mathcal{D}^{+}}\frac{q(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta) (23) ≈ ∑ τ ∈ 𝒟 + 1 n ​ ∑ i n q i ​ ( τ ) π ref ​ ( τ ) ​ log ⁡ p ⁡ ( τ , θ ) , \displaystyle\approx\sum_{\tau\in\mathcal{D}^{+}}\frac{1}{n}\sum_{i}^{n}\frac{q_{i}(\tau)}{\pi_{\mathrm{ref}}(\tau)}\log p(\tau;\theta), (24) note that, here 𝒟 + \mathcal{D}^{+} is the set of data points with positive advantage in our fixed dataset 𝒟 \mathcal{D} . When q = π ref q=\pi_{\mathrm{ref}} , this collapses to maximum likelihood over the positive examples.

## Appendix B Detailed algorithm

A detailed algorithm listing of our method is given in Algorithm 1 .

## Appendix C Hyperparameters for LLM experiments

The hyperparameters of our model are set the same as in [ Muennighoff et al., 2025 ] : we use cosine annealing for the learning rate, decaying to zero over a maximum of 5 epochs and using a warm-up ratio of 0.05. The target learning rate for AdamW [ Loshchilov et al., 2017 ] is 1 ​ e − 5 1e-5 , β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 and the batch size is 8 8 . Weight decay is set to 1 ​ e − 4 1e-4 . We also use g ⁡ ( x ) = 0.1 × clip ⁡ ( x , 0.2 , 1.8 ) g(x)=0.1\times\mathrm{clip}(x,0.2,1.8) to smooth/clip the importance weights. Given that we do not have access to the model weights for Gemini Flash (which was used to generate the reasoning traces), or its original training data, we also have to rely on an approximation for the reference distribution π ref \pi_{\mathrm{ref}} in the importance weights. Ideally, we would pick a model that is “close” to Gemini-Flash as an approximation. We tested multiple models at the 32B parameter level, comparing their summed log-probabilities over examples from the dataset and find that they assign similar likelihood (albeit giving different top-k next token predictions) and settled simply using the model we start from as the reference model such that q , π ref q,\pi_{\mathrm{ref}} and π θ \pi_{\theta} are all close at the beginning of training.

## Appendix D Compute resources and infrastructure

#### LLM training

All LLM experiments are performed using a single machine with 8 H100 NVIDIA GPUs and 192GB of host ram. Training is performed using pytorch [ Paszke et al., 2019 ] and the HuggingFace accelerate library [ Gugger et al., 2022 ] . Training completes within 18 hours.

#### Continuous control experiments

For training models on D4RL [ Fu et al., 2020 ] we use a single machine with an L40 NVIDIA GPU. Though the models here are small and could be trained on any consumer GPU. We use a batch size of 256 where each element consists of a full trajectory (thus we are able to compute normalized importance weights on the fly). Training time depends on the task and accelerator used (we estimate 30-60 minutes per task).

## Appendix E Dataset curation for control tasks

The creation of curated datasets is described in Algorithm 2 .

## Appendix F Additional details for training on D4RL control tasks

For all training runs we use a linear warm-up for the learning rate followed by cosine annealing [ Loshchilov and Hutter, 2017 ] we train with Adam [ Kingma and Ba, 2015 ] . We also use a three layer MLP with 256 as hidden dimension. We first train a BC policy for 10000 steps with learning rate 1 ​ e − 3 1e-3 and batch size 32. This policy is used as initialization for π θ \pi_{\theta} . For subsequent fine tuning, we use batch size 256 for all tasks. For SFT(Q), we use learning rate 4e-5, warmup steps 300, total steps 4500, percentile cutoffs [90,95, 98], apart from the hopper-medium-expert-v2 where we use learning rate 4e-4. Hyperparameters for iw-SFT(Q) can be found in 4 , we note that we first divide

## Appendix G Ablations with Budget Forcing

We perform an additoinal experiment to evaluate whether budget forcing as in Muennighoff et al. [2025] further improves iw-SFTṪhe results are shown in Figure 2 . As can be seen, when we control the exact amount of thinking tokens and slowly increase it (forcing longer thinking times by injecting "Wait" as proposed in Muennighoff et al. [2025] ) we do not observe an improvement upon the result with no forcing (see Table 1 , not included in figure). We note that the graph here was created using an earlier training run with slightly different settings (e.g. for the importance weight truncation) we did not re-run with all token budgets for the latest model due to the additional costs incurred.

## Appendix H Code

The code to reproduce our results can be found on github. For LLM training the code is available under https://github.com/emmyqin/iw_sft . The checkpoint is avaliable at huggingface under https://huggingface.co/ChongliQin/iw-SFT-32B .

The code for control experiments can also be found on github under https://github.com/emmyqin/iw_sft_control .

### H.1 Expanded D4RL results

Expanded Offline RL results for control suite environments from the D4RL suite of tasks using our method are given in Table 5 .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .