From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones — part 2/3

Cleaned-text code points [15561, 31441). Bibliography omitted here; source.md includes it.

Difficulty Level . We control compositional complexity through Difficulty Levels corresponding to nesting depth, with Level n n involving n n -function composition. For instance, Level 1 involves single function application (e.g., func_16(x) as shown in Fig. 1 ), while Level 2 involves two-function composition (e.g., func_16(func_15(x)) ). The controlled difficulty provides a fine-grained inspection of model performance, rather than a vague overall number as adopted in prior work ( Yue et al., 2025 ; Liu et al., 2025a ; Wu et al., 2025a ) .

### 3.2 Training and Evaluation Protocol

Training consists of two stages to separate atomic skill acquisition from compositional skill learning, simulating realistic post-training pipelines.

Stage 1 Training: Atomic Skills Acquisition via RFT . Models learn “atomic skills” in this stage via rejection fine-tuning Dong et al. (2023) . Specifically, we collect training data by prompting the model with explicit function definitions to generate correct reasoning trajectories. However, we remove these definitions from the prompts during the fine-tuning process. This compels the model to predict the output based solely on the function identifier, ensuring it internalizes the function’s behavior, defined here as the atomic skill, rather than relying on in-context instructions. Crucially, the data collection phase of this stage is the only time models are exposed to the function implementations. An example can be found in Fig. 7 .

Stage 2 Training: Compositional Skill Training via Either RFT or RL . In this stage, models see only function names and compositions, such as func_2(func_16(x)) , with function definitions hidden. See Fig. 8 for examples. This forces reliance on internalized atomic knowledge while learning systematic composition. We compare two approaches: (1) Composition via online RL provides models with binary rewards based on output correctness and updates through Group Relative Preference Optimization (GRPO) Shao et al. (2024) , testing whether RL is necessary for the acquisition of compositional skills. (2) Composition via offline RFT trains models with NTP on correct reasoning trajectories for compositional problems, serving as a baseline to examine whether exposure to compositional examples alone enables composition.

We use Llama-3.1-8B-Instruct, which is identified as a cleaner testbed for RL by recent work ( Shao et al., 2025 ; Agarwal et al., 2025 ; Wu et al., 2025b ) , to further minimize the effect of data contamination besides our string tasks. For more details, please refer to Appendix A .

Held-out, Easy-to-Hard, and Cross-Task Evaluation . We assess generalization using rigorous held-out evaluation. In Stage 1, models are trained on all 25 atomic functions (Appendix D ). In Stage 2, the functions are partitioned into two disjoint sets: the model trains only on compositions from one set, while the other is held out for evaluation. We test model generalization across various difficulty levels, using Countdown ( Gandhi et al., 2024 ; Pan et al., 2025 ) as a testbed for task transfer.

## 4 RL as a Pathway to Generalizable Skill Acquisition

### 4.1 LLMs Acquire New Compositional Skills during RL

Our first experiment directly test our RL Compositionality Hypothesis (Hypothesis 1 ). To do so, we start from an identical Stage 1 base model and apply three different Stage 2 training configurations, allowing us to isolate the impact of incentivizing composition during RL: (1) RL Level 1 , trained only on atomic tasks; (2) RL Level 2 , trained only on two-level compositions; and (3) RL Level 1+2 , trained on a uniform mix. We then evaluate their ability to generalize to held-out tasks from Level 1 up to Level 6, testing whether they can solve problems with unseen function compositions and higher nesting levels than seen in RL training.

As shown in blue curves in Fig. 2 , training on Level 1 alone leads to high accuracy on Level 1, peaking at around 90%, but fails to generalize. Its accuracy on Level 2 task remains below 25%, and on Level 3 through 6, it is consistently near zero . This demonstrates that learning only the atomic skills through RL is insufficient for learning effective composition.

In contrast, incorporating compositional tasks into RL training yields transformative results. Both the RL Level 2 and RL Level 1+2 models demonstrate strong performance to generalize to problems with nesting depths exceeding their training data. On Level 3, their accuracy improves from 5% to around 30%, and from 1% to 15% on Level 4, which are all significant improvements over the RL Level 1 model. And this trend continues on even Level 5, indicating both models learn a generalizable principle of compositional reasoning rather than merely memorizing solutions. This validates our hypothesis that RL can teach genuinely new skills, but only when the training objective explicitly incentivizes their use. These results provide us with evidence to answer RQ1:

### 4.2 RL is the Key Ingredient to the New Compositional Skills

Our previous experiment shows that compositional data is necessary for RL to teach new compositional skills, but can a supervised method, such as RFT, achieve the same results as RL when given the exact same compositional (Level 2) data? To address this question, we train a model with iterative RFT on the same Level 2 problems and conduct a head-to-head comparison against the RL Level 2 model from § 4.1 , with both having started from the identical Stage 1 base model.

The results in Fig. 3 show a significant difference in performance from Fig. 2 . The RFT model’s accuracy is significantly worse than RL across all compositional levels and has only marginal improvement over the first iteration. For example, on Level 3 it never surpasses 2.6%. In contrast, the RL Level 2 model achieves 64% on Level 2 and 27% on Level 3, significantly outperforming the RFT model. Surprisingly, the RFT model attains only 15% accuracy on Level-2 problems. This indicates that RFT fails to generalize even to held-out compositional problems of the same difficulty as its training data, let alone higher difficulties. These results provide the evidence to answer RQ2:

### 4.3 Compositional Skills Learned in RL are Transferable, but Atomic Skills are Prerequisites

While our experiments demonstrate that RL can teach generalizable compositional skills within a task, collecting compositional RL data for every new domain is impractical. We therefore test the transferability of the learned compositional skill. Specifically, we conjecture that RL enables models to compose atomic skills on Task B after learning composition on Task A, if the model has already acquired the necessary atomic skills for Task B.

Experimental Setup. We test this conjecture on the Countdown task, where a model must construct a mathematical expression from a given set of integers to reach a target number (see § E for examples). In Countdown, a Level ℓ \ell task requires the model to construct a mathematical expression using ℓ \ell given integers to reach a target number. The minimum level for Countdown is Level 2. We compare four models to test our hypothesis, as detailed in Tab. 1 . These configurations allow us to compare a “atomic-skill-only” baseline (Multi-Base) against models with either transferred atomic RL (Multi-Base + RL L1) or transferred compositional RL (Multi-Base + RL L1+2), as well as a control model from § 4.1 that has the compositional skill but lacks the necessary atomic knowledge of Countdown (String-Base + RL L1+2). Note that none of the models are trained on Countdown with RL in Stage 2, and are only trained on our string task.

We evaluate these models on unseen, more challenging Countdown problems (Levels 3-5). We report the Avg@32, the average accuracy across 32 responses sampled at temperature 1.0.

Results. The results in Fig. 4 provide clear evidence supporting our hypothesis. The String-Base + RL L1+2 model fails completely. The Multi-Base model achieves reasonable accuracy of approximately 17% at Level 3 but still struggles at higher levels. Multi-Base + RL L1 shows marginal improvement over Multi-Base, increasing accuracy to around 20% at Level 3, with the advantage diminishing on more complex problems. The Multi-Base + RL L1+2 model achieves surprisingly strong performance. It achieves a 35% accuracy at Level 3, outperforming the Multi-Base baseline by more than 18% accuracy. This advantage persists at higher complexities, reaching approximately 6% at Level 4, where other models largely fail and achieve near-zero accuracy. The results show that the compositional skill learned from string transformation transfers to countdown, acting as a meta-skill that enhances the use of the target task’s atomic knowledge. Finally, the comparison between Multi-base + RL and String-Base + RL L1+2 confirms our fundamental assumption that task-specific atomic skills are prerequisites for compositional skills to be effective.

These results may explain recent findings on generalizable RL improvements. For example, Logic-RL ( Xie et al., 2025 ) reports performance gains on mathematical problems after training on logic puzzles, and Guru ( Cheng et al., 2025 ) shows that domains with greater pre-training exposure benefit more from cross-task generalization. We suggest that LLMs have already acquired essential atomic skills through large-scale pre-training, particularly in mathematics and coding. Thus, incentivizing compositional skills through RL in one task helps combine task-specific skills more effectively across domains. In contrast, domains with less pre-training exposure may lack sufficient atomic skills, limiting compositional skill transfer to downstream tasks. With this finding, we answer RQ3:

### 4.4 RL Expanding Performance Limits is Not a False Promise

Our findings strongly suggest that RL can teach compositional skills that are novel to the base model. This directly challenges recent arguments that RL merely “reranks” model responses, distilling pass@ k k performance of the base model into pass@ 1 1 ( Yue et al., 2025 ; Wu et al., 2025a ) . This conclusion is drawn based on a shrinking pass@ k k performance gap between base and RL-tuned models as k k increases. However, we argue this conclusion may stem from two issues: (1) evaluating on mixed-skill benchmarks, therefore an improvement in a specific skill, like composition, can be masked in pass@ k k if other required skills remain a bottleneck, and (2) using RL training that does not properly incentivize the new skill in the first place.

Our controlled framework allows us to dissect both issues. By isolating the compositional skill at varying difficulty levels, we can reliably assess skill acquisition (addressing issue 1), and by comparing different RL training setups (§ 4.1 ), we can test the effect of proper incentivization (addressing issue 2). We compare pass@ 1000 1000 performance at each difficulty level of our test set, selecting k = 1000 k=1000 as a sufficiently large and practically meaningful budget. Larger budgets would become impractical, as any reasonable model could theoretically achieve pass@ ∞ = 1 \infty=1 .

The results are presented in Fig. 5 . Both RL Level 1 and RL Level 1+2 models are trained from RFT base model using RL in Stage 2. The RL Level 1 model, which is not incentivized properly to learn composition, exhibits a similar trend to the RFT base across almost all levels. On easier problems (Levels 1 and 2) where the RFT base model already shows solving potential evidenced by high pass@k, the performance gaps between RL Level 1+2 model and the RFT model shrink as k k increases, aligning with the trends observed in Yue et al. (2025) ; Wu et al. (2025a) . However, a completely different trend is observed on more challenging compositional problems (Levels 3-6). The RL Level 1+2 model’s performance substantially outperforms the RFT base with an increasing gap as k k grows. For example, at Level 5, the performance gap over the RFT base grows from 4% at pass@1 to approximately 25% at pass@1024. This divergence is clear evidence of new skill acquisition. The results suggest that the pessimistic observation of “RL does not push performance limits” in prior work may be explained by the lack of incentive for RL to learn new skills, as the base model already achieves high pass@k performance.

### 4.5 Behavioral Analysis: RL Transforms Failure Modes

While our results show that training with compositional data unlocks promising generalization, a fundamental question remains: do models trained under different setups exhibit different behaviors, or do they simply differ in capability while showing similar failure modes? To investigate this, we analyze the failure modes of different models on Level 3 problems of our string task.

We use Gemini-2.5-Pro to classify responses into five categories: (1) Correct , (2) Ignores Composition (e.g., analyzing only a single function), (3) Incomplete Trace (recognizes composition but terminates early), (4) Incorrect Composition (e.g., misinterprets nesting), and (5) Atomic Error (errors in atomic functions prediction without the above). Categories 2-4 indicate difficulties with handling compositional problems. And while still incorrect, category 5 represents appropriate compositional behavior, as the error is not due to a lack of compositional skill.

We compare four models: RFT Base (after Stage 1 training), RFT Level 2 (after Stage 2 training on Level 2 problems with RFT), RL Level 1 , and RL Level 2 , all from previous sections. The latter three models are all trained from the RFT Base.

Fig. 6 reveals substantial similarities in the failure patterns of RFT Base, RFT Level 2, and RL Level 1 models. Their failures are dominated by ignoring the composition entirely (all > > 50%) and misunderstanding the compositional structure (all > > 35%).

In contrast, the RL Level 2 model demonstrates fundamentally different behaviors. It completely eliminates ”Ignores Composition” errors and correctly solves 28.1% of the problems. Crucially, its primary failure mode becomes “Atomic Error.” This shows that compositional RL not only improves accuracy but teaches models to parse and execute compositional plans, shifting failures from high-level misunderstandings to lower-level execution errors. See § F for examples of different model responses.

## 5 Conclusion

The debate over whether RL can teach LLMs new skills has been clouded by experiments on benchmarks where LLMs already perform well, using coarse-grained metrics that obscure the learning of new capabilities. By stepping back to a cleaner, more controllable experimental environment, our findings provide a clear and optimistic answer: RL can teach genuinely new and powerful skills when the training task properly incentivize composition.

Our work focuses on the paradigm where RL acts upon pre-existing atomic competencies, mirroring the standard practice of carrying out RL based on pre-trained models. While our results show that possessing atomic skills is a sufficient condition for RL to unlock compositional capabilities, we do not claim it is strictly necessary. However, given the inefficiency of random exploration in discovering complex atomic behaviors from scratch, we argue that the synergy between supervised learning for atomic skill acquisition and RL for compositional generalization represents the most commonly adopted and promising path. Our findings suggest that the pessimistic conclusion that RL does not learn new skills may stem from inappropriate evaluation setups rather than fundamental constraints of RL itself. Future work may investigate the open question of whether RL can be scaled to acquire both atomic and compositional skills simultaneously without supervised scaffolding.

## Limitation