{"id":"arxiv:2509.25123","title":"From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones","date":"2025-09-29","section":"Post-training","quality":-0.057993,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/","source_url":"https://arxiv.org/abs/2509.25123","text_available":true,"authors":["Lifan Yuan","Weize Chen","Yuchen Zhang","Ganqu Cui"],"outgoing":[{"source":"arxiv:2509.25123","target":"arxiv:2402.03300","evidence":"title","id":"arxiv:2402.03300","title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/636fb871904bc06f742cfdea069796f9e9665f95bd58dc63260eaff5154ef985/"},{"source":"arxiv:2509.25123","target":"arxiv:2407.21783","evidence":"title","id":"arxiv:2407.21783","title":"The Llama 3 Herd of Models","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de3cebb4c28bc5d41a843279f219dca14476284991ea5c7c0aeab7068a40e306/"},{"source":"arxiv:2509.25123","target":"arxiv:2503.14476","evidence":"title","id":"arxiv:2503.14476","title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/548e934b912ef84691ace63bf1cfd670e7b0257d9d6986dde926f569e7b2f8b7/"},{"source":"arxiv:2509.25123","target":"arxiv:2503.20783","evidence":"title","id":"arxiv:2503.20783","title":"Understanding R1-Zero-Like Training: A Critical Perspective","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b49a056f3ce8e269dd65e3b11605927f3c4d0ef074bc68e0b1d7f84e6c9c6c6a/"},{"source":"arxiv:2509.25123","target":"arxiv:2504.13837","evidence":"title","id":"arxiv:2504.13837","title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b90f70269aeb02311d74ffaadf47e139683424a47118dc18619000ba72174d17/"},{"source":"arxiv:2509.25123","target":"arxiv:2504.20571","evidence":"id","id":"arxiv:2504.20571","title":"Reinforcement Learning for Reasoning in Large Language Models with One Training Example","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b2b21c3df49619588c371411dba9467a41f81b3fd4da5441898b5145bb0de120/"},{"source":"arxiv:2509.25123","target":"arxiv:2505.15134","evidence":"title","id":"arxiv:2505.15134","title":"The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/12b01b403e280a32d41fd991a27fa30f33315e09f0f42a4f43e9bece5ad9c53d/"},{"source":"arxiv:2509.25123","target":"arxiv:2506.10947","evidence":"title","id":"arxiv:2506.10947","title":"Spurious Rewards: Rethinking Training Signals in RLVR","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f48e13a9656d8eff1882eb80988e07a820c4799401298947deac735c004e8592/"}],"incoming":[],"text_parts":[{"start":0,"end":15561,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html"},{"start":15561,"end":31441,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html"},{"start":31441,"end":37430,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html"}],"contents":[{"title":"From f ⁡ ( x ) f(x) and g ⁡ ( x ) g(x) to f ⁡ ( g ⁡ ( x ) ) f(g(x)) : LLMs Learn New Skills in RL by Composing Old Ones","offset":56,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"Abstract","offset":179,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"1 Introduction","offset":2271,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"2 Background","offset":9290,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"3 Research Framework","offset":13330,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"Hypothesis 1 (The RL Compositionality Hypothesis) .","offset":13699,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"3.1 Task Design: Deductive Reasoning on String Transformation Prediction","offset":13995,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-1.html#text"},{"title":"3.2 Training and Evaluation Protocol","offset":16109,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4 RL as a Pathway to Generalizable Skill Acquisition","offset":18658,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4.1 LLMs Acquire New Compositional Skills during RL","offset":18715,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4.2 RL is the Key Ingredient to the New Compositional Skills","offset":20553,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4.3 Compositional Skills Learned in RL are Transferable, but Atomic Skills are Prerequisites","offset":21778,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4.4 RL Expanding Performance Limits is Not a False Promise","offset":25363,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"4.5 Behavioral Analysis: RL Transforms Failure Modes","offset":27984,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"5 Conclusion","offset":29996,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"Limitation","offset":31426,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-2.html#text"},{"title":"Appendix A Training Details","offset":32201,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"A.1 Stage 1","offset":32233,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Data Generation.","offset":32423,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Rejection Fine-Tuning.","offset":33044,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"A.2 Stage 2","offset":33317,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Data Generation.","offset":33334,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Reinforcement Learning.","offset":33852,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Rejection Fine-Tuning.","offset":34412,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Appendix B Evaluation Details","offset":34877,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"String Transformation Prediction Task.","offset":34911,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Countdown Task.","offset":35262,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Appendix C Example Prompts for Stage 1 and Stage 2","offset":35461,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Appendix D A Complete List of String Transformation Functions","offset":35960,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Appendix E Example for Countdown Task","offset":36188,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Appendix F Model Response Examples","offset":36230,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"F.1 Full Responses to the Level 2 Problem","offset":36372,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"F.2 Full Responses to the Level 3 Problem","offset":36419,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"},{"title":"Instructions for reporting errors","offset":36466,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/part-3.html#text"}],"text_code_points":37430,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7fdfe7abe26ee324fe7abad3ce802148163129a7c157d77bdec859c990868e03/source.md"}