On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification

arxiv:2508.05629

Original source · Paper JSON

Post-training · 2025-08-07

Cites (6) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Method
• 3.1 Preliminaries
• Supervised Fine-Tuning.
• Reinforcement Learning.
• 3.2 Unify SFT and RL Gradient Expression
• Rewriting SFT Gradient as Policy Gradient via Importance Sampling.
• 3.3 Proposed Method
• Reward Rectification via Dynamic Reweighting.
• 4 Experiments
• 4.1 Main Experiment - Mathematical Reasoning Task
• Implementation details.
• 4.2 Exploratory Experiment - Offline RL Setting
• Implementation details.
• 4.3 Exploratory Experiment - Code Generation Task
• Implementation details.
• 4.4 Exploratory Experiment - Multi-Modal Reasoning
• Implementation details.
• 4.5 Limitations of DFT: A Case Study on Factual Knowledge
• 4.6 An Empirical Comparison with Sentence-Level Weighting
• 4.7 Analysis of Probabilities
• 5 Conclusion
• Limitations.
• Acknowledgements
• Ethics Statement
• Reproducibility Statement
• Appendix A Appendix
• A.1 Usage of LLM
• A.2 Detailed Derivation of Equation (5)
• A.3 Discussions and Insights
• Gradient Analysis of DFT.
• Learning from Noisy Data.
• A.4 Comparision with Concurrent work iw-SFT
• A.5 Exploratory Experiment - OpenR1-Math Training Dataset
• A.6 Exploratory Experiment - PEFT Training Setting
• A.7 Training Hyper-Parameters Ablation
• Instructions for reporting errors