From $f(x)$ and $g(x)$ to $f(g(x))$: LLMs Learn New Skills in RL by Composing Old Ones — part 3/3
Cleaned-text code points [31441, 37430). Bibliography omitted here; source.md includes it.
One limitation of this work is the reliance on synthetic tasks for evaluation. We deliberately designed our string transformation framework to enable rigorous experiments with well-defined atomic and compositional skills, thus isolating causal learning mechanisms. However, our synthetic tasks may not fully capture the complexity and nuance of real-world reasoning scenarios where skills are less clearly delineated and compositional structures are more varied. We acknowledge that demonstrating these findings in natural reasoning domains, such as mathematical problem-solving, code generation, or scientific reasoning, represents an important direction for future work. Extending these findings to realistic applications remains a valuable open challenge.
## Appendix A Training Details
### A.1 Stage 1
All models in our experiments, except for the Multi-Base variants in § 4.3 , start from the same Stage 1 base model. The process for creating this model is detailed below.
#### Data Generation.
We first generate 50k Level-1 problems by randomly sampling a function and a string input (length 3 to 10). For each problem, we collect 10 responses from the Llama-3.1-8B-Instruct model (temperature 1.0, max length 8192). To focus training on problems the model finds non-trivial, we discard any problems where the base model achieves 100% accuracy. We then collect all remaining correct responses to form the SFT dataset. Before fine-tuning, we remove the function definitions from the prompts, keeping only the function identifier and the input. This results in around 116k training instances.
#### Rejection Fine-Tuning.
We fine-tune the model for 2 epochs with a learning rate of 2 × 10 − 5 2\times 10^{-5} and a global batch size of 128. Other hyperparameters follow the default settings provided by the veRL framework 1 1 1 https://github.com/volcengine/verl .
### A.2 Stage 2
#### Data Generation.
The problems used in Stage 2 are created using a similar process as in Stage 1, with the critical differences being that function definitions are hidden and problems can involve composition. For Level-2 problems, we randomly select and compose two functions. Example prompts for Stage 1 vs. Stage 2 can be found in Appendix C . For the Level 1 only and Level 2 only dataset, we create 50k problems. For the Level 1+2 mixed configuration, we create 25k problems for each level and combine them.
#### Reinforcement Learning.
For RL experiments, we use DAPO ( Yu et al., 2025 ) as the optimization algorithm. We enforce a strictly on-policy setup by setting both the training batch size and mini-batch size to 16. For each prompt, we generate 16 rollouts using a sampling temperature of 1.0 and a maximum response length of 8192. During training, we filter out any problems for which all rollouts are correct or all are incorrect. We use a learning rate of 1 × 10 − 6 1\times 10^{-6} and set the coefficients for both KL divergence and entropy loss to 0.
#### Rejection Fine-Tuning.
For the iterative RFT baseline, we use a learning rate of 2 × 10 − 5 2\times 10^{-5} and a batch size of 128. The process is iterative: the model from the previous iteration is used to generate a new set of rollouts. These rollouts are then filtered for correctness to construct a new SFT dataset, following the same procedure used in Stage 1. The model is then trained on this new dataset to produce the model for the next iteration.
## Appendix B Evaluation Details
#### String Transformation Prediction Task.
For the string transformation prediction task, the evaluation set consists of 256 randomly generated problems for each level from 1 to 8, resulting in a total of 2048 test problems. During evaluation, we generate responses using a sampling temperature of 1.0 and a maximum response length of 8192 tokens.
#### Countdown Task.
The evaluation set for the Countdown task consists of 128 problems for each level. The evaluation setup is the same as evaluating on the string transformation prediction task.
## Appendix C Example Prompts for Stage 1 and Stage 2
Here we present the example prompts used in our experiments. Fig. 7 shows the prompt used for Stage 1 data generation, where the function definition is provided to guide the model. However, during the actual Stage 1 training (and throughout Stage 2), this definition is removed, forcing the model to rely on the internalized atomic skills associated with the function identifier. Fig. 8 shows the prompt used for Stage 1 and Stage 2 training.
## Appendix D A Complete List of String Transformation Functions
Here we provide the full list of string functions we use. Note that we replace the function names with meaningless identifiers such as func_1 in our experiment.
## Appendix E Example for Countdown Task
## Appendix F Model Response Examples
The following sections contain the complete, unfolded responses from each model for the case studies.
### F.1 Full Responses to the Level 2 Problem
### F.2 Full Responses to the Level 3 Problem
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .