On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting

arxiv:2508.11408

Original source · Paper JSON

Data, training, optimization · 2025-08-15

Cites (10) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting Thanks: Corresponding author. Email to {zwh434786, yaliang.li}@alibaba-inc.com
• Abstract
• 1 Introduction
• 2 Preliminaries
• 3 Chord : Harmonizing Off-Policy and On-Policy Learning
• 3.1 The Shift-Readapt-Overfit Progression When Utilizing Off-Policy Data
• 3.2 Controlling the Influence of Off-Policy Expert Data via μ \mu
• 3.3 Enhancing the Stability of Off-policy Learning via ϕ ⁡ ( ⋅ ) \phi(\cdot)
• 4 Experiments
• 4.1 Setup
• 4.2 Comparisons
• 4.3 Analysis on the Effects of μ \mu and ϕ ⁡ ( ⋅ ) \phi(\cdot)
• 4.4 Further Analysis
• 5 Related Works
• 6 Conclusions and future directions
• Acknowledgements
• Reproducibility Statement
• Appendix A Experimental Setups
• A.1 Hyperparameters
• A.2 Implementation Details
• A.3 Prompts
• Prompt for Math Problems
• Prompt for the MMLU-Pro Dataset
• Prompt for the Tool-use Tasks
• Appendix B Experimental Results and Analysis
• B.1 Adaptive Tuning μ \mu
• B.2 Varying the ϕ \phi Function
• Evaluated ϕ \phi Variants.
• Experimental Setup.
• Results and Analysis.
• B.3 Varying Expert Data Source
• Pattern Examples
• B.4 Non-Verifiable Tasks
• B.5 SFT/RL synergy for Weaker Policy Models
• B.6 Diverse Model Architectures
• B.7 Tuning μ \mu in Conjunction with ϕ \phi
• B.8 Experimental Results on Tool-use Training
• B.9 Experimental Results on the MMLU-pro dataset
• Appendix C Detailed Discussions of Related Works
• C.1 Finetuning for LLMs
• SFT for LLMs.
• RL for LLMs.
• C.2 On- and Off-policy Reinforcement Learning
• Combining On-policy and Off-policy Data in Traditional RL
• Combining On-policy and Off-policy Data in RL for LLM
• Appendix D Further Discussions
• D.1 The Influence of Off-Policy Data on Base vs. Instruction Models
• D.2 On Different Task-Related Performance
• D.3 Scaling SFT Is Not Enough: The Necessity of On-Policy Learning
• Appendix E Case Studies
• Use of Large Language Models
• Instructions for reporting errors