Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved) — part 4/4

Cleaned-text code points [46967, 52204). Bibliography omitted here; source.md includes it.

The hyperparameters of our model are set the same as in [ Muennighoff et al., 2025 ] : we use cosine annealing for the learning rate, decaying to zero over a maximum of 5 epochs and using a warm-up ratio of 0.05. The target learning rate for AdamW [ Loshchilov et al., 2017 ] is 1 ​ e − 5 1e-5 , β 1 = 0.9 , β 2 = 0.95 \beta_{1}=0.9,\beta_{2}=0.95 and the batch size is 8 8 . Weight decay is set to 1 ​ e − 4 1e-4 . We also use g ⁡ ( x ) = 0.1 × clip ⁡ ( x , 0.2 , 1.8 ) g(x)=0.1\times\mathrm{clip}(x,0.2,1.8) to smooth/clip the importance weights. Given that we do not have access to the model weights for Gemini Flash (which was used to generate the reasoning traces), or its original training data, we also have to rely on an approximation for the reference distribution π ref \pi_{\mathrm{ref}} in the importance weights. Ideally, we would pick a model that is “close” to Gemini-Flash as an approximation. We tested multiple models at the 32B parameter level, comparing their summed log-probabilities over examples from the dataset and find that they assign similar likelihood (albeit giving different top-k next token predictions) and settled simply using the model we start from as the reference model such that q , π ref q,\pi_{\mathrm{ref}} and π θ \pi_{\theta} are all close at the beginning of training.

## Appendix D Compute resources and infrastructure

#### LLM training

All LLM experiments are performed using a single machine with 8 H100 NVIDIA GPUs and 192GB of host ram. Training is performed using pytorch [ Paszke et al., 2019 ] and the HuggingFace accelerate library [ Gugger et al., 2022 ] . Training completes within 18 hours.

#### Continuous control experiments

For training models on D4RL [ Fu et al., 2020 ] we use a single machine with an L40 NVIDIA GPU. Though the models here are small and could be trained on any consumer GPU. We use a batch size of 256 where each element consists of a full trajectory (thus we are able to compute normalized importance weights on the fly). Training time depends on the task and accelerator used (we estimate 30-60 minutes per task).

## Appendix E Dataset curation for control tasks

The creation of curated datasets is described in Algorithm 2 .

## Appendix F Additional details for training on D4RL control tasks

For all training runs we use a linear warm-up for the learning rate followed by cosine annealing [ Loshchilov and Hutter, 2017 ] we train with Adam [ Kingma and Ba, 2015 ] . We also use a three layer MLP with 256 as hidden dimension. We first train a BC policy for 10000 steps with learning rate 1 ​ e − 3 1e-3 and batch size 32. This policy is used as initialization for π θ \pi_{\theta} . For subsequent fine tuning, we use batch size 256 for all tasks. For SFT(Q), we use learning rate 4e-5, warmup steps 300, total steps 4500, percentile cutoffs [90,95, 98], apart from the hopper-medium-expert-v2 where we use learning rate 4e-4. Hyperparameters for iw-SFT(Q) can be found in 4 , we note that we first divide

## Appendix G Ablations with Budget Forcing

We perform an additoinal experiment to evaluate whether budget forcing as in Muennighoff et al. [2025] further improves iw-SFTṪhe results are shown in Figure 2 . As can be seen, when we control the exact amount of thinking tokens and slowly increase it (forcing longer thinking times by injecting "Wait" as proposed in Muennighoff et al. [2025] ) we do not observe an improvement upon the result with no forcing (see Table 1 , not included in figure). We note that the graph here was created using an earlier training run with slightly different settings (e.g. for the importance weight truncation) we did not re-run with all token budgets for the latest model due to the additional costs incurred.

## Appendix H Code

The code to reproduce our results can be found on github. For LLM training the code is available under https://github.com/emmyqin/iw_sft . The checkpoint is avaliable at huggingface under https://huggingface.co/ChongliQin/iw-SFT-32B .

The code for control experiments can also be found on github under https://github.com/emmyqin/iw_sft_control .

### H.1 Expanded D4RL results

Expanded Offline RL results for control suite environments from the D4RL suite of tasks using our method are given in Table 5 .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .