Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved) — part 1/4

Cleaned-text code points [0, 15691). Bibliography omitted here; source.md includes it.

# Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

###### Abstract

Behavior Cloning (BC) on curated (or filtered) data is the predominant paradigm for supervised fine-tuning (SFT) of large language models; as well as for imitation learning of control policies. Here, we draw on a connection between this successful strategy and the theory and practice of finding optimal policies via Reinforcement Learning (RL). Building on existing literature, we clarify that SFT can be understood as maximizing a lower bound on the RL objective in a sparse reward setting. Giving support to its often observed good performance. From this viewpoint, we realize that a small modification to SFT leads to an importance weighted variant that behaves closer to training with RL as it: i) optimizes a tighter bound to the RL objective and, ii) can improve performance compared to SFT on curated data. We refer to this variant as importance weighted supervised fine-tuning (iw-SFT). We show that it is easy to implement and can be further generalized to training with quality scored data. The resulting SFT variants are competitive with more advanced RL algorithms for large language models and for training policies in continuous control tasks. For example achieving 66.7% on the AIME 2024 dataset.

## 1 Introduction

Post-training of large language models (LLMs) is an increasingly important component in modern large model training pipelines since it can help bring out latent capabilities dormant in pre-trained LLMs. Whether post-training should be performed via reinforcement learning (RL) or instead use simple supervised finetuning (SFT), and how these two steps can be combined, is a hot topic in the community. Generally, Reinforcement learning (RL) from human [ Ouyang et al., 2022 , Ziegler et al., 2019 ] or AI feedback [ Lee et al., 2024 ] has become a crucial step in aligning large language models (LLMs) to better follow human intent, such as instructions, or to steer models to generate outputs that are more helpful and factual [ Bai et al., 2022 ] . Recently, it has been established that RL is an essential part for eliciting reasoning capabilities in LLMs as exeplified by the recent OpenAI o1 and DeepSeek-R1 [ Guo et al., 2025 ] models.

Unfortunately, RL is difficult to interpret and tune. As a result there is a large body of research aiming to further stabilize or simplify existing RL algorithms (see e.g. Shao et al. [2024] , Ahmadian et al. [2024] , Li et al. [2023] ). Many recent post-training approaches even attempt to side-step RL training entirely – e.g. by directly training LLMs to maximize preferences such as in DPO Rafailov et al. [2024] or other RL-free approaches [ Abdolmaleki et al., 2024 ] . Intriguingly, some recent work [ Muennighoff et al., 2025 , Wang et al., 2025 ] goes as far as showing that one can improve reasoning in LLMs via SFT on a carefully filtered/curated dataset and then scaling inference compute.

In this paper, we draw a on a known connection between RL as inference [ Peters and Schaal, 2007 , Kober and Peters, 2008 , Levine, 2018 ] and (iterative) maximum likelihood supervised learning that we believe is underexplored. As we show, this perspective suggests that when we perform SFT on filtered data we are in fact optimizing for a loose lower bound on the RL objective in a sparse reward setting (induced by the filtering of the data). By taking a fresh look at SFT from this perspective, we show that this bound can be tightened significantly in two ways: by sampling data proportional to quality scores where they are available, and via importance re-weighting of the maximum likelihood SFT objective. Intuitively, this re-weighting can also be seen as a way of performing ‘adaptive-filtering’ since the reweighted data points should place increasingly heavier weights on ’preferred’ data points. The resulting small modifications to SFT – which we refer to as SFT from quality sampled data (SFT(Q)) and importance weighted SFT (iw-SFT) respectively – are easy to implement and can match or surpass SFT in many settings. Our experiments show that when optimizing with iw-SFT on curated data, reasoning naturally emerges in LLMs with no need to further scale inference compute (e.g. via some form of budget forcing [ Muennighoff et al., 2025 ] ); leading to high performance on benchmarks such as AIME (66.7%) or GPQA (64.1%) outperforming standard SFT on the same data. We further show the generality by adapting our method to train continuous control policies in an offline RL setting. Our contributions are as follows: • We demonstrate that supervised fine tuning is optimizing a lower bound on the RL objective in the sparse reward setting and introduce an improved, weighted, variant of SFT, iw-SFT.

• We perform extensive experiments, including training LLMs for reasoning (evaluating on AIME 2024 and MATH500 [ Hendrycks et al., 2021 ] ), outperforming the state-of-the-art for training open-models on open data Muennighoff et al. [2025] . We also show how iw-SFT can improve upon BC in offline RL for control on the D4RL benchmark [ Fu et al., 2020 ] .

## 2 Related Work

Our work connects several different fields of study which are all closely related. Several works have previously considered connections between supervised fine-tuning and RL. For example, methods from the literature on reward weighted regression [ Dayan and Hinton, 1997 , Peters and Schaal, 2007 ] (RWR) for RL have been adopted to improve supervised learning in cases where reward functions are available; e.g. in the literature on iterative maximum likelihood [ Ni et al., 2023 , Agarwal et al., 2019 , Norouzi et al., 2016 ] . Most related to our approach, recent work has considered reward weighted iterative filtering (or rejection sampling) for supervised fine-tuning [ Yuan et al., 2024 , Du et al., 2025 , Gulcehre et al., 2023 ] , showing success in fine-tuning LLMs. Among them the ReST family of methods [ Gulcehre et al., 2023 , Singh et al., 2024 ] can be understood as iteratively optimizing the SFT bound we discuss below (though without importance weighting). In a related direction, recently, DeepSeekMath [ Shao et al., 2024 ] also drew on the connection between expectation maximization, RWR and supervised learning, treating methods such SFT, RFT, DPO, PPO, GRPO as maximizing the obective: 𝔼 ( q , o ) ∼ 𝒟 ​ ( 1 | o | ​ ∑ | o | G ⁡ ( q , o , r ) ​ ∇ θ ​ log ⁡ π θ ​ ( o t | q , o < t ) ) , \mathbb{E}_{(q,o)\sim\mathcal{D}}\left(\frac{1}{|o|}\sum^{|o|}G(q,o,r)\nabla_{\theta}\log\pi_{\theta}(o_{t}|q,o_{<t})\right), where G G denotes generalized returns.

In this work, we build on a similar connection between SFT and RL. In particular, we realize that by following prior work on RL as inference (see e.g. Dayan and Hinton [1997] , Peters and Schaal [2007] , Peters et al. [2010] , Levine [2018] ) we can interpret SFT as optimizing a lower bound to the RL objective. In particular we build on a bound that is a special case of a general form developed in Roux [2016a] . In this work, the authors show that the well known PoWER [ Kober and Peters, 2008 ] and reward weighted regression (RWR) [ Dayan and Hinton, 1997 , Peters and Schaal, 2007 ] algorithms can be derived by finding a lower bound to the RL objective, and can be generalized to an iterative scheme. Follow-up work [ Roux, 2016b ] also considered the applicability of this loss to supervised classification, albeit in a different way than what we consider here. The authors consider introducing an iterated supervised learning scheme which leads to optimizing a tighter bound to the classification accuracy than the commonly used negative log likelihood loss. In contrast, we here are concerned with fine-tuning models on curated (or filtered) datasets that were originally drawn from some reference distribution.

In the literature one can also find general discussions on training generative models on self-generated data after human curation e.g. in Solaiman and Dennison [2021] , Ferbach et al. [2024] . Most relevant to the ideas presented here is recent work on interpreting generative models that consume their own, human curated, data as optimizing human preferences via RLHF [ Ferbach et al., 2024 ] . This work again invokes the well known connections between SFT and EM based policy search.

Our connection between RL and SFT is focused on the sparse reward regime. As a result, it is closely related to other works that aim to improve training on preference data. This includes different RLHF methods such as Ouyang et al. [2022] , Ziegler et al. [2019] . As well as methods which bypass explicit RL such as DPO [ Rafailov et al., 2024 ] or expectation maximization [ Abdolmaleki et al., 2024 ] .

In the literature on RL for control, many recent works have explored the connection between expectation maximization and KL regularized policy optimization [ Abdolmaleki et al., 2018 , Abdolmaleki et al., 2021 , Peng et al., 2019 , Nair et al., 2020a , Schulman et al., 2017 ] . Similar to these, our methodology can be used both in the off-policy or online setting. In this work, we mostly focus on the offline setting. For our experimentation on MuJoCo Locomotion tasks, we compare with highly performant offline RL algorithms such as AWAC [ Nair et al., 2020a ] and IQL [ Kostrikov et al., 2021 ] as well as top-K percentile based behavior cloning (BC) [ Brandfonbrener et al., 2022 , Chen et al., 2021 ] .

## 3 SFT on Curated Data as Reinforcement Learning

Reinforcement Learning (RL) is the predominant approach for improving alignment and reasoning in LLMs; as well as for training agents in (continuous) control settings. RL, however, is often compute and data intensive and hard to tune. In contrast, SFT on curated (or filtered) datasets is stable, requires no change in the loss-function between pre-training and fine-tuning and can be surprisingly effective.

Here, we make a connection between these two algorithms: SFT and RL. Specifically, we show that SFT on filtered data optimizes a lower bound on the RL objective for a sparse reward task induced by the curation/filtering process. From this perspective we can observe that this lower bound becomes looser as the model moves further away from the expert that generated the data (the reference policy). Interestingly, following existing literature on bounding the RL objective [ Roux, 2016a ] we can show that one can introduce an adaptive (importance) reweighting scheme (iw-SFT) that asymptotically maximizes the RL objective as the bound tightens over the course of training. As we will see this simple change from SFT to iw-SFT not only theoretically narrows the gap between SFT and RL but also practically can achieve better results in the same settings that SFT excels in.

### 3.1 Notation and Background

To connect RL and SFT we formalize the general RL problem and then show how SFT can be seen as maximizing a lower bound on this objective; specializing a well known bound from the RL literature [ Peters and Schaal, 2007 , Peters et al., 2010 , Roux, 2016b ] to the sparse / binary reward setting.

#### Notation

In the following, s ∈ 𝒮 s\in\mathcal{S} and a ∈ 𝒜 a\in\mathcal{A} denote states and actions in a Markov Decision Process (MDP) respectively. We define a trajectory as a sequence of states and actions: τ = ( s 0 , a 0 , ⋯ , s T ) ∈ 𝒮 × 𝒜 ⋯ 𝒮 = ℬ \tau=(s_{0},a_{0},\cdots,s_{T})\in\mathcal{S}\times\mathcal{A}\cdots\mathcal{S}=\mathcal{B} . The actions are drawn from a parametric policy π ⁡ ( a t | s t , θ ) \pi(a_{t}|s_{t},\theta) (a probability distribution over actions) and the dynamics of the trajectory through time depend on the transition probabilities p ⁡ ( s t + 1 | s t , a t ) p(s_{t+1}|s_{t},a_{t}) . The reward function is dependent on the state and action r ⁡ ( s t , a t ) r(s_{t},a_{t}) , we simplify this to r ⁡ ( t ) r(t) ; where r ⁡ ( T ) r(T) then corresponds to the terminal reward. We aim to maximize the discounted cumulative reward or the return, R ⁡ ( τ ) = ∑ t = 0 T γ ⁡ ( t ) ​ r ​ ( t ) R(\tau)=\sum_{t=0}^{T}\gamma(t)r(t) , with discount factor γ \gamma . Note that in a sparse reward setting where we only observe a binary terminal reward r ⁡ ( T ) r(T) and assume ∀ t ≠ T : r ⁡ ( t ) = 0 \forall t\neq T:r(t)=0 and use γ = 1 \gamma=1 the return simplifies to 𝕀 ⁡ ( R ⁡ ( τ ) ) = ∑ t = 0 T r ⁡ ( t ) = r ⁡ ( T ) \mathbb{I}(R(\tau))=\sum_{t=0}^{T}r(t)=r(T) (where 𝕀 \mathbb{I} denotes an indicator function). Often such a sparse reward setting is induced by checking if a continuous function exceeds a threshold at the last timestep.

#### Reinforcement learning

Using the above, we can define the probability of a given trajectory as: p π ​ ( τ , θ ) \displaystyle p_{\pi}(\tau;\theta) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ​ ( a t | s t ; θ ) . \displaystyle=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi(a_{t}|s_{t};\theta). (1) We further note that, in the language modeling setting the states s s and actions a a are both given by the tokens emitted by the model – i.e. 𝒮 = 𝒜 \mathcal{S}=\mathcal{A} – and the transition dynamics are deterministic. Thus, the policy directly determines the next state and we can equivalently write p ⁡ ( τ , θ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 π ⁡ ( s t + 1 | s 0 , ⋯ , s t ; θ ) p(\tau;\theta)=p(s_{0})\prod_{t=0}^{T-1}\pi(s_{t+1}|s_{0},\cdots,s_{t};\theta) , which is the autoregressive model probability of the text tokens. Note that in this case the transitions are not Markovian and we thus condition on all previous states but all derivations below apply without loss of generality in both cases.

The objective of training a policy π \pi via reinforcement learning (RL) is defined as maximizing the expected cumulative return, where the expectation is over observed trajectories: max θ ​ J ​ ( θ ) = ∫ ℬ p ⁡ ( τ , θ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 p ⁡ ( τ , θ ) ​ [ R ⁡ ( τ ) ] . \displaystyle\mathrm{max}_{\theta}~J(\theta)=\int_{\mathcal{B}}p(\tau;\theta)R(\tau)\mathrm{d}\tau=\mathbb{E}_{p(\tau;\theta)}\big[R(\tau)\big]. (2)

To see the connection between RL and SFT let us first assume that we only have access to trajectories τ ∈ π ref \tau\in\pi_{\mathrm{ref}} generated by some reference policy. This could be a human demonstrating desired trajectories/outputs/behaviour or simply a pre-trained model that we query for outputs. To use these trajectories for optimizing the RL objective we have to re-express it in terms of these samples. Here we need to make the assumption that the support of p π p_{\pi} is a subset of π ref \pi_{\mathrm{ref}} . Using importance sampling (see [ Kahn and Marshall, 1953 , Rubinstein and Kroese, 2016 , Ripley, 2009 ] for a review), we can then express our RL objective from Eq. 2 as: J ⁡ ( θ ) \displaystyle J(\theta) = ∫ ℬ π ref ​ ( τ ) ​ p π ​ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ​ 𝑑 τ = 𝔼 π ref ​ ( τ ) ​ [ p π ​ ( τ , θ ) π ref ​ ( τ ) ​ R ​ ( τ ) ] , \displaystyle=\int_{\mathcal{B}}\pi_{\mathrm{ref}}(\tau)\frac{p_{\pi}(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\mathrm{d}\tau=\mathbb{E}_{\pi_{\mathrm{ref}}(\tau)}\Big[\frac{p_{\pi}(\tau;\theta)}{\pi_{\mathrm{ref}}(\tau)}R(\tau)\Big], (3) we overload the notation of π ref \pi_{\text{ref}} for simplicity, where the expectation is over the reference π ref ​ ( τ ) = p ⁡ ( s 0 ) ​ ∏ t = 0 T − 1 p ⁡ ( s t + 1 | s t , a t ) ​ π ref ​ ( a t | s t ) \pi_{\mathrm{ref}}(\tau)=p(s_{0})\prod_{t=0}^{T-1}p(s_{t+1}|s_{t},a_{t})\pi_{\mathrm{ref}}(a_{t}|s_{t}) and weight samples via the importance weight p π ​ ( τ , θ ) / π ref ​ ( τ ) \nicefrac{{p_{\pi}(\tau;\theta)}}{{\pi_{\mathrm{ref}}(\tau)}} .

### 3.2 SFT lower bounds RL