Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)

arxiv:2507.12856

Original source · Paper JSON

Data, training, optimization · 2025-07-17

Cites (5) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
• Abstract
• 1 Introduction
• 2 Related Work
• 3 SFT on Curated Data as Reinforcement Learning
• 3.1 Notation and Background
• Notation
• Reinforcement learning
• 3.2 SFT lower bounds RL
• 3.3 Improving SFT from the perspective of RL
• 3.4 The importance of choosing q ⁡ ( τ ) q(\tau)
• Clipping per step importance weights
• Smoothing importance weights
• 3.5 A Toy Example to build Intuition for SFT on curated/filtered data
• 4 Experiments
• 4.1 Training LLMs for reasoning via SFT on curated data
• 4.2 Offline RL for Continuous Control
• 4.3 Fine-tuning control policies with little data and noisy quality scores
• 5 Conclusion and Discussion
• Appendix A Derivation for the iw-SFT objective
• Appendix B Detailed algorithm
• Appendix C Hyperparameters for LLM experiments
• Appendix D Compute resources and infrastructure
• LLM training
• Continuous control experiments
• Appendix E Dataset curation for control tasks
• Appendix F Additional details for training on D4RL control tasks
• Appendix G Ablations with Budget Forcing
• Appendix H Code
• H.1 Expanded D4RL results
• Instructions for reporting errors