From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones

arxiv:2509.25123

Original source · Paper JSON

Post-training · 2025-09-29

Cites (8) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• From f ⁡ ( x ) f(x) and g ⁡ ( x ) g(x) to f ⁡ ( g ⁡ ( x ) ) f(g(x)) : LLMs Learn New Skills in RL by Composing Old Ones
• Abstract
• 1 Introduction
• 2 Background
• 3 Research Framework
• Hypothesis 1 (The RL Compositionality Hypothesis) .
• 3.1 Task Design: Deductive Reasoning on String Transformation Prediction
• 3.2 Training and Evaluation Protocol
• 4 RL as a Pathway to Generalizable Skill Acquisition
• 4.1 LLMs Acquire New Compositional Skills during RL
• 4.2 RL is the Key Ingredient to the New Compositional Skills
• 4.3 Compositional Skills Learned in RL are Transferable, but Atomic Skills are Prerequisites
• 4.4 RL Expanding Performance Limits is Not a False Promise
• 4.5 Behavioral Analysis: RL Transforms Failure Modes
• 5 Conclusion
• Limitation
• Appendix A Training Details
• A.1 Stage 1
• Data Generation.
• Rejection Fine-Tuning.
• A.2 Stage 2
• Data Generation.
• Reinforcement Learning.
• Rejection Fine-Tuning.
• Appendix B Evaluation Details
• String Transformation Prediction Task.
• Countdown Task.
• Appendix C Example Prompts for Stage 1 and Stage 2
• Appendix D A Complete List of String Transformation Functions
• Appendix E Example for Countdown Task
• Appendix F Model Response Examples
• F.1 Full Responses to the Level 2 Problem
• F.2 Full Responses to the Level 3 Problem
• Instructions for reporting errors