{"id":"arxiv:2508.11408","title":"On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting","date":"2025-08-15","section":"Data, training, optimization","quality":0.16439,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/","source_url":"https://arxiv.org/abs/2508.11408","text_available":true,"authors":["Wenhao Zhang","Yuexiang Xie","Yuchang Sun"],"outgoing":[{"source":"arxiv:2508.11408","target":"arxiv:2402.03300","evidence":"id","id":"arxiv:2402.03300","title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/636fb871904bc06f742cfdea069796f9e9665f95bd58dc63260eaff5154ef985/"},{"source":"arxiv:2508.11408","target":"arxiv:2407.21783","evidence":"id","id":"arxiv:2407.21783","title":"The Llama 3 Herd of Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de3cebb4c28bc5d41a843279f219dca14476284991ea5c7c0aeab7068a40e306/"},{"source":"arxiv:2508.11408","target":"arxiv:2502.03387","evidence":"id","id":"arxiv:2502.03387","title":"LIMO: Less is More for Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/714392528bb842fb3a61cee5367942f697536f0131b0fcac85c6eb8d2212125c/"},{"source":"arxiv:2508.11408","target":"arxiv:2503.14476","evidence":"id","id":"arxiv:2503.14476","title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/548e934b912ef84691ace63bf1cfd670e7b0257d9d6986dde926f569e7b2f8b7/"},{"source":"arxiv:2508.11408","target":"arxiv:2504.13837","evidence":"id","id":"arxiv:2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b90f70269aeb02311d74ffaadf47e139683424a47118dc18619000ba72174d17/"},{"source":"arxiv:2508.11408","target":"arxiv:2506.01939","evidence":"id","id":"arxiv:2506.01939","title":"Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/e0f4a55bca612bce5895b6b98f89b0efc14cbd8cf45946a7743347cf6daee1a1/"},{"source":"arxiv:2508.11408","target":"arxiv:2506.13585","evidence":"id","id":"arxiv:2506.13585","title":"MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5beb2ec6c7e6362f9d957219d7a205dd1adcc53f739acc0c7fd5f1f2bded4d83/"},{"source":"arxiv:2508.11408","target":"arxiv:2507.12856","evidence":"id","id":"arxiv:2507.12856","title":"Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5f6c1906eaa67369d9ff15b974756b89141d1b4cf9d7c0f6554e0d56f26e79be/"},{"source":"arxiv:2508.11408","target":"arxiv:2508.05629","evidence":"id","id":"arxiv:2508.05629","title":"On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/0a17fccfeffa8a04070bb16903a7217d8adee804cf42e0cfc8dd128d3390474f/"},{"source":"arxiv:2508.11408","target":"url:rlhfbook.com","evidence":"title","id":"url:rlhfbook.com","title":"Reinforcement Learning from Human Feedback","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/20e354559fad09ad4d6338d104c9c3f822a48e7fbf86cd8afa1ddf951b223658/"}],"incoming":[],"text_parts":[{"start":0,"end":15869,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html"},{"start":15869,"end":31659,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html"},{"start":31659,"end":47651,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html"},{"start":47651,"end":63296,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html"},{"start":63296,"end":77430,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html"}],"contents":[{"title":"On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting Thanks: Corresponding author. Email to {zwh434786, yaliang.li}@alibaba-inc.com","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"Abstract","offset":205,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"1 Introduction","offset":1791,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"2 Preliminaries","offset":6515,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"3 Chord : Harmonizing Off-Policy and On-Policy Learning","offset":10317,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"3.1 The Shift-Readapt-Overfit Progression When Utilizing Off-Policy Data","offset":10377,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"3.2 Controlling the Influence of Off-Policy Expert Data via μ \\mu","offset":13442,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-1.html#text"},{"title":"3.3 Enhancing the Stability of Off-policy Learning via ϕ ⁡ ( ⋅ ) \\phi(\\cdot)","offset":17528,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html#text"},{"title":"4 Experiments","offset":22225,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html#text"},{"title":"4.1 Setup","offset":22243,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html#text"},{"title":"4.2 Comparisons","offset":24663,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html#text"},{"title":"4.3 Analysis on the Effects of μ \\mu and ϕ ⁡ ( ⋅ ) \\phi(\\cdot)","offset":29346,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-2.html#text"},{"title":"4.4 Further Analysis","offset":34973,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"5 Related Works","offset":37611,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"6 Conclusions and future directions","offset":39077,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Acknowledgements","offset":41092,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Reproducibility Statement","offset":41300,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Appendix A Experimental Setups","offset":41390,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"A.1 Hyperparameters","offset":41425,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"A.2 Implementation Details","offset":42315,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"A.3 Prompts","offset":43860,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Prompt for Math Problems","offset":43877,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Prompt for the MMLU-Pro Dataset","offset":44225,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Prompt for the Tool-use Tasks","offset":44505,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Appendix B Experimental Results and Analysis","offset":44824,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"B.1 Adaptive Tuning μ \\mu","offset":44873,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"B.2 Varying the ϕ \\phi Function","offset":46927,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Evaluated ϕ \\phi Variants.","offset":47281,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-3.html#text"},{"title":"Experimental Setup.","offset":48059,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"Results and Analysis.","offset":48437,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.3 Varying Expert Data Source","offset":50664,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"Pattern Examples","offset":53039,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.4 Non-Verifiable Tasks","offset":53720,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.5 SFT/RL synergy for Weaker Policy Models","offset":55067,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.6 Diverse Model Architectures","offset":56930,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.7 Tuning μ \\mu in Conjunction with ϕ \\phi","offset":57762,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.8 Experimental Results on Tool-use Training","offset":59821,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"B.9 Experimental Results on the MMLU-pro dataset","offset":60254,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"Appendix C Detailed Discussions of Related Works","offset":60474,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"C.1 Finetuning for LLMs","offset":60527,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"SFT for LLMs.","offset":60556,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"RL for LLMs.","offset":61792,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"C.2 On- and Off-policy Reinforcement Learning","offset":63181,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"Combining On-policy and Off-policy Data in Traditional RL","offset":63232,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-4.html#text"},{"title":"Combining On-policy and Off-policy Data in RL for LLM","offset":64183,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"Appendix D Further Discussions","offset":67216,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"D.1 The Influence of Off-Policy Data on Base vs. Instruction Models","offset":67251,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"D.2 On Different Task-Related Performance","offset":71049,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"D.3 Scaling SFT Is Not Enough: The Necessity of On-Policy Learning","offset":73639,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"Appendix E Case Studies","offset":74849,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"Use of Large Language Models","offset":76294,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"},{"title":"Instructions for reporting errors","offset":76466,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/part-5.html#text"}],"text_code_points":77430,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48594da2e3f2cdfeb17195dd033d047235fc4b4ffb65030800f64c6faedc4fbe/source.md"}