From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones — part 1/3

Cleaned-text code points [0, 15561). Bibliography omitted here; source.md includes it.

\DeclareCaptionType listing[Listing][List of Listings]

# From f ⁡ ( x ) f(x) and g ⁡ ( x ) g(x) to f ⁡ ( g ⁡ ( x ) ) f(g(x)) : LLMs Learn New Skills in RL by Composing Old Ones

###### Abstract

Does reinforcement learning (RL) teach large language models (LLMs) genuinely new skills, or does it merely activate existing ones? This question lies at the core of ongoing debates about the role of RL in LLM post-training. On one side, strong empirical results can be achieved with RL even without preceding supervised finetuning; on the other, critics argue that RL contributes little beyond reweighting existing reasoning strategies. This work provides concrete evidence that LLMs can acquire genuinely new skills during RL by composing existing ones, mirroring one of the central mechanisms by which humans acquire new cognitive skills ( Anderson, 1982 ) . To mitigate data contamination and other confounding factors, and to allow precise control over task complexity, we develop a synthetic framework for our investigation. Specifically, we define a skill as the ability to infer the output of a string transformation function f ⁡ ( x ) f(x) given x x . When an LLM has already learned f f and g g prior to RL, our experiments reveal that RL enables it to learn unseen compositions of them h ⁡ ( x ) = g ⁡ ( f ⁡ ( x ) ) h(x)=g(f(x)) . Further, this compositional ability generalizes to more difficult problems such as compositions of > 2 >2 functions unseen during RL training. Our experiments provide surprising evidence that this compositional ability, acquired on the source task, transfers to a different target task. This transfer occurs even though the model has never trained on any compositional problems in the target task, and the only requirement is that the model has acquired the target task’s atomic skills before its RL training on the source. Our qualitative analysis shows that RL fundamentally changes the reasoning behaviors of the models. In contrast, none of the findings is observed in next-token prediction training with the same data. Our systematic experiments provide fresh insights into the learning behaviors of widely-used post-training approaches for LLMs. They suggest the value of building base models with the necessary basic skills.

## 1 Introduction

Reinforcement learning (RL) has achieved broad success in improving large language models (LLMs) on a variety of tasks especially reasoning ( OpenAI, 2024 ; DeepMind, 2025 ) , even directly building upon the base model without any preceding supervised fine-tuning ( DeepSeek-AI et al., 2025 ) . Despite the profound success, recent work finds the exploration of RL is impeded by the entropy collapse phenomenon ( Cui et al., 2025b ; Liu et al., 2025a ; Yu et al., 2025 ) , and the performance gaps between base and RL-trained models diminish as the number of samples ( k k ) increases in pass@ k k evaluations ( Yue et al., 2025 ) . In addition, some argue that the “aha moments” in RL training ( OpenAI, 2024 ; DeepSeek-AI et al., 2025 ) are not emergent but merely the result of amplifying existing cognitive behaviors present in base models ( Gandhi et al., 2025 ; Liu et al., 2025b ; Zhao et al., 2025 ) , which casts shadow on whether LLMs learn new skills during RL training ( Wu et al., 2025a ) . Such observations diverge from established RL findings that predate LLMs, where models were trained from scratch and learned new skills ( Silver et al., 2016 ; Silver et al., 2017 ; OpenAI et al., 2019 ) . The fact that LLMs are pretrained on vast data prior to RL may contribute to these divergences and call for further investigation into the following important research questions: (1) Does RL teach new skills to LLMs? (2) If so, how to incentivize it? (3) Are the skills generalizable? Answering these questions will advance our understanding of LLM learning behaviors and inform the high-stakes trade-off in resource allocation between pretraining and post-training.

We provide concrete evidence that LLMs indeed learn new skills in RL, by composing and generalizing existing skills to solve more complex problems; For such learning to happen, there should be proper incentivization in RL. Our investigation is grounded in the cognitive skill acquisition process by humans, inspired by Anderson (1982) , which argues that humans learn new skills by composing and then internalizing existing ones. Unlike prior works ( Gandhi et al., 2025 ; Yue et al., 2025 ) , we choose to construct a controlled synthetic framework that facilitates: • Decontaminated evaluation : We design a string transformation prediction task with unique functions assigned meaningless identifiers (e.g., func_16 ) to prevent inference from function names. During RL, function definitions are hidden. The tasks will then be unsolvable without going through our atomic skills acquisition training. This setup enables us to investigate the RQs controlling for confounders.

• Well-defined atomic and compositional skills : We define atomic skills as single, non-decomposable transformations, and compositional skills as their nested combinations. For example, given input string x x , func_16( x x ) represents an atomic skill, while func_15(func_16( x x )) requires compositional reasoning.

• Controllable difficulty : As each skill is instantiated as a Python function, we control difficulty of composition through the depth of nesting. As shown in Fig. 1 , the model must perform deductive reasoning to give the output string after a given transformation, e.g., a Level-1 difficulty problem func_16( x x ) and a Level-2 one func_16(func_15( x x )) . Here the difficulty level is determined by the number of atomic functions composed.

With our framework and a two-stage training protocol that separates atomic from compositional skill acquisition, we conduct experiments with Llama-3.1-8B-Instruct ( Dubey et al., 2024 ) and answer the RQs as follows:

• RL teaches new compositional skills. RL on Level-2 problems, receiving only correctness-based outcome rewards without reasoning demonstrations, substantially improves generalization on more difficult problems: performance on unseen Level-3 tasks improves from near-zero to 30%, and Level-4 to 15%. This generalization does not occur in a baseline trained with rejection fine-tuning (RFT) on the same Level-2 problems. This shows that RL enables the acquisition of compositional skills.

• Both RL and compositional incentives are essential for skill acquisition. In contrast to the substantial accuracy improvements from RL on Level-2 compositional problems, RFT on the same data and RL on Level-1 atomic problems both yield little improvements on problems higher than Level-2 (e.g., less than 1% improvement at Level-3). This may explain why Sun et al. (2025) conclude that RL does not promote compositional generalization, as their training includes no explicit incentive for composition.

• The learning achieved by RL generalizes to held-out evaluation, more difficult problems, and even a different task. All findings above are based on held-out evaluation of compositional problems consisting of atomic skills (functions) unseen in RL training. And as aforementioned, models RL-trained on Level 2 problems show non-trivial gains on problems up to Level 4. For cross-task transfer, compositional RL on the string task boosts accuracy on the unseen Level-3 Countdown problems to 35% for a model with the prerequisite Countdown atomic skills.

Our findings challenge the recent view that current RL with verifiable rewards (RLVR) ( Lambert et al., 2025 ) merely utilizes reasoning patterns in base models rather than learning new reasoning abilities ( Yue et al., 2025 ; Wu et al., 2025a ) . This view is based on the observation that the pass@ k k performance gap between RL-trained and base models narrows as k k increases ( Yue et al., 2025 ) . We conjecture that this observation arises, at least in part, from evaluating and RL training on tasks where base models already achieve high pass@ k k , possibly due to pretraining on similar tasks that is beyond the control of most academic researchers; thus RL has little incentive to learn a skill that the base model already has. To confirm this conjecture, our experiments show that RL substantially improves pass@ k k on challenging compositional problems where base model’s pass@ k k is near zero (See Fig. 5 ). This reveals what we term the “reranking illusion,” namely aggregate metrics on mixed-difficulty benchmarks can mask genuine skill acquisition by conflating capabilities of different types. Our qualitative analysis confirms that RL fundamentally changes reasoning behavior. As shown in Fig. 6 , compositional errors, i.e., ignoring composition and misunderstanding function relationships, drop substantially, while failures shift primarily to atomic prediction errors (55%). This behavioral transformation indicates genuine acquisition of compositional skills.

Our findings have important implications for LLM development and highlight RL’s critical role in post-training, particularly its potential for easy-to-hard generalization and cross-task transfer. They call for closer coordination between base model development and post-training strategy from a skill acquisition perspective.

## 2 Background

The Recent Pessimistic View on Whether RL Teaches New Skills to LLMs . RL in LLMs builds on a model pretrained on vast data. While supervised warm-starts are a common technique in traditional RL ( Silver et al., 2016 ; Vinyals et al., 2019 ; De La Cruz Jr et al., 2019 ; Silva and Gombolay, 2021 ) , the large-scale and general-purpose nature of LLM creates a different scenario. On one hand, this strong prior enables base LLM to sample reasonable rollouts and thus perform RL directly without any preceding supervised fine-tuning ( DeepSeek-AI et al., 2025 ; Pan et al., 2025 ; Zeng et al., 2025 ) ; on the other hand, it becomes difficult to distinguish genuine skill acquisition from activation of existing capabilities during RL training.

Recent work tries to investigate this but uses loose definitions of “skill”, often relying on proxies such as the continually increasing frequency of certain reasoning patterns ( Gandhi et al., 2025 ; Zhao et al., 2025 ; Liu et al., 2025b ) or the diminishing gaps between the pass@ k k accuracy of models before and after RL, as shown in the bottom right chart in Fig. 1 ( Yue et al., 2025 ; Liu et al., 2025a ; Wu et al., 2025a ; He et al., 2025 ; Wen et al., 2025 ; Zhu et al., 2025 ) . Although these studies show that RL activates behaviors already present in the base model, they did not directly prove that no new skill is learned during the process. Moreover, the pass@ k k results can be misinterpreted for many reasons: (1) The causal relation between performance and each skill remains unclear, thus it is not guaranteed that everything learned can be translated into improvements in pass@ k k accuracy on downstream tasks. (2) The evaluation tasks only provide an obscure overall view, lacking fine-grained analysis on problems of different difficulty levels or domains. (3) The result is confounded by the possibility that the model has limited room or incentive to learn new skills if it already performs well prior to RL, especially when RL is conducted on data that overlaps with or closely resembles the data used during next-token prediction (NTP) training. ( Wu et al., 2025c ; Shao et al., 2025 ; Wang et al., 2025 ; Cui et al., 2025a ; Yu et al., 2025 ; Liu et al., 2025a ; Wu et al., 2025b ) . Together, these highlight the urgent need for a deeper analysis of tasks through a clean framework, in which the skills are clearly defined and contribute to the performance causally, and evaluated in a finer granularity.

Compositional Learning as a Testbed Grounded in Cognitive Skill Acquisition in Humans . Although the pessimistic conclusions about RL in LLMs from prior works are debatable, they at least indicate that the success of RL depends on strong base models. This motivates our study of skill composition, where RL learns new abilities by leveraging those already acquired by the base model. Compositional reasoning provides an ideal framework for investigating skill acquisition because it naturally separates atomic knowledge, which mirrors how humans learn cognitive skills ( Anderson, 1982 ) . Notably, it is established in cognitive science that both composed skills and the meta-ability to learn composition are non-trivial new skills ( Anderson, 1982 ; Lake et al., 2016 ) . For clarity, we refer to learning new skills as the former throughout this paper. Learning compositional skills helps the model to generalize to more challenging problems and new domains beyond training data, which we will show later. In the field of AI, compositional reasoning has been widely studied before LLMs and has been considered a necessary property of generalization. ( Fodor and Pylyshyn, 1988 ; Lake et al., 2016 ; Andreas et al., 2015 ) . More recently, Yin et al. (2025) achieved compositional improvements through in-context learning rather than RL, while Sun et al. (2025) found that directly RL in atomic skills fails in compositional generalization. Comparing the two works, we conjecture that an explicit incentive to composition is necessary.

## 3 Research Framework

In this work, we define “new skills” as novel reasoning strategies that enable models to solve previously unsolvable problems through systematic combination of existing capabilities. We address three critical research questions: (1) Does RL teach new skills to LLMs? (2) If so, how to incentivize it? (3) Are the learned skills generalizable?

###### Hypothesis 1 (The RL Compositionality Hypothesis) .

Once a model has acquired the necessary atomic, non-decomposable skills for a task through NTP training, RL with proper incentivization can teach the model to learn new skills by composing atomic skills into more complex capabilities.

### 3.1 Task Design: Deductive Reasoning on String Transformation Prediction

To test our hypothesis while avoiding confounders from data contamination and unclear skill boundaries, we design a controlled synthetic task with the following properties: (1) Atomic skills are well defined so that models can learn the fundamental skills separately before RL. Each string transformation function has clear, deterministic behaviors that can be learned independently. (2) Task difficulty can be controlled by adjusting the compositional complexity of the atomic skills, allowing us to test generalization across complexity levels. (3) RL and evaluation tasks do not appear in the LLM pretraining corpus, ensuring that improvements stem from learning rather than memorization.

Task Definition . Specifically, our task involves deductive reasoning on string transformations. Given an input string x x and a composition of deterministic transformation functions such as f ⁡ ( ⋅ ) f(\cdot) and g ⁡ ( ⋅ ) g(\cdot) , models must predict the output string after applying the specified transformation (e.g., y = f ⁡ ( g ⁡ ( x ) ) y=f(g(x)) ). We construct 25 unique string transformation functions as atomic skill spanning various computational patterns including character manipulation, reordering, filtering, and structural modifications (see Appendix § D for complete specifications). To mitigate potential contamination, we assign meaningless identifiers to string functions as shown in Fig. 1 , so that it is impossible to infer the functionality with function names only.