Recursive Language Models — part 2/5

Cleaned-text code points [15811, 31526). Bibliography omitted here; source.md includes it.

Compaction agent. Following Sun et al. (2025) ; Wu et al. (2025) ; Yu et al. (2025) , we consider an iterative agent that compacts the context as it is filled. For example, given a corpus of documents, it will iteratively accumulate the documents and summarize when full. In cases where a single document exceeds the model window, the agent will chunk the document and iteratively compact it. For the GPT-5 experiments, due to the extremely high cost of applying this strategy to millions of tokens, we use GPT-5-nano for compaction and GPT-5 to provide the final answer.

Coding agents. We compare against commonly used coding agents like OpenCode ( Anomaly, 2026 ) and Claude Code ( Anthropic, 2025 ) . We consider two variants, one where the context is offloaded to a file, and another where it is directly used as the initial prompt. Closed source agents like Claude Code are designed around a corresponding model, so we use Claude Opus 4.1 with Claude Code v2.0.0 (released around the same time as the GPT-5 model we use in our main results) for this baseline.

RLM . We implement an RLM with a Python REPL environment, which loads a module for querying a sub-LM and uses a system prompt presented in Appendix C . For the GPT-5 experiments, we use GPT-5-mini for the recursive LMs and GPT-5 for the root LM, as we found this choice to strike a good balance between the capabilities of RLMs and the cost of the recursive calls. We also evaluate several different max recursion depths allowable to the RLM, from 0-3. Max recursion depth 0 is an RLM without sub-calling capabilities. Max recursion depth 1 allows sub-calling LLMs, while max depth >1 allows sub-calling RLMs. We notate a RLM with max recursion depth N N using a model as RLM(model, depth= N N ), e.g. RLM(GPT-5, depth=2), and assume depth=1 if not stated otherwise.

Fine-tuning. To create RLM-Qwen3-8B , we fine-tune Qwen3-8B on 1,000 filtered trajectories of Qwen3-Coder-480B-A35B as an RLM with Qwen3-8B sub-calls on LongBenchPro ( Chen et al., 2026 ) tasks. We use sampling parameters described in Qwen Team (2025a) , and evaluate the fine-tuned RLM-Qwen3-8B as an RLM. The key insight for training is that being an effective sub-call model is roughly similar to being a general purpose reasoning model, so we can make the training much more tractable at small scale by focusing on improving the root model’s ability to manipulate the REPL and to launch recursive calls. We provide more training details in Appendix A .

## 4 Results and Discussion

Table 1 reports our main evaluation results. We additionally explore how vanilla frontier model and RLM performance degrade as input contexts grow in Figure 1 .

Observation 1: RLMs can scale to the 10M+ token regime and can outperform base LMs and existing task-agnostic agent scaffolds on long context tasks . Across all tasks, RLMs demonstrate strong performance on prompts well beyond the effective context window of a frontier LM, outperforming base models and common long-context scaffolds by up to 2 × 2\times the performance while maintaining comparable or cheaper average token costs. Notably, RLMs scale well beyond the base models’ context window. For instance, on BrowseComp-Plus (1K), a linearly extrapolated cost for GPT-5-mini ingesting 6-11M input tokens is $ 1.50 − $ 2.75 \$1.50-\$2.75 , while RLM(GPT-5, depth=1) has an average cost of $ 0.99 \$0.99 and outperforms both the compaction and retrieval baselines by over 29 % 29\% .

Furthermore, on tasks where processing costs scale with the input context, RLMs make significant improvements over the base model, even on tasks within the model’s context window. On OOLONG, the RLM(depth=1) with GPT-5 and Qwen3-Coder outperform the base model by 28.4 % 28.4\% and 33.3 % 33.3\% respectively. On OOLONG-Pairs, both GPT-5 and Qwen3-Coder make little progress with F1 scores of ≤ 0.1 % \leq 0.1\% , while the RLM(depth=1) using these models achieve F1 scores of 58.0 % 58.0\% and 23.1 % 23.1\% respectively, highlighting the capability of RLMs to handle extremely information-dense tasks.

Observation 2: The REPL is necessary for handling long inputs, while the recursive sub-calling of RLMs provides strong benefits on information-dense inputs. A key characteristic of RLMs is offloading the context as a variable in an environment ℰ \mathcal{E} that the model can interact with. In particular, RLM(depth=0) and coding agents like Claude Code and OpenCode are able to scale beyond the context limit of the model and outperform other task-agnostic baselines on most long context settings. On CodeQA in particular with Qwen3-Coder-480B-A35B, the no-sub-calling RLM(depth=0) is able to outperform all sub-calling variants of the RLM.

On information-dense tasks like OOLONG or OOLONG-Pairs, we observed several cases where programmatic recursive LM sub-calling is necessary. In § 5 , we see RLM(Qwen3-Coder) perform the necessary semantic transformation line-by-line through recursive sub-calls, while the ablation without sub-calls is forced to use keyword heuristics to solve these tasks. On OOLONG-Pairs in particular, the higher recursive depth variants of the RLM for GPT-5 outperform all other methods including Claude Code and OpenCode by a large margin.

Observation 3: LM performance degrades as a function of input length and problem complexity, while RLM performance scales better. The benchmarks S-NIAH, OOLONG, and OOLONG-Pairs contain a fixed number of tasks over contexts with lengths ranging from 2 13 2^{13} to 2 20 2^{20} . Each benchmark can be categorized by different processing complexity of the input context with respect to length (roughly constant, linear, and quadratic respectively). In Figure 1 , we directly compare an RLM(GPT-5, depth=1) to base GPT-5, and find that GPT-5 performance degrades significantly faster for more complex tasks, which aligns with the findings of Goldman et al. (2025) , while RLM performance degrades at a slower rate. For context lengths beyond 2 14 2^{14} , the RLM consistently outperforms GPT-5.

Furthermore, RLM costs scale proportionally to the complexity of the task, while still remaining in the same order of magnitude of cost as GPT-5 (see Figure 16 in Appendix F ). In § 5 , we explore the choices that the RLM makes that cause these differences in cost.

Observation 4: The inference cost of RLMs remains comparable to other methods, and in some cases base LM calls. On average, we find in Table 1 that the inference cost of RLMs is cheaper or comparable to most other baselines, including standard coding agents. Furthermore, in Figure 11 in Appendix F , we find that the median RLM run is cheaper than the median base model run, but more expensive on average due to outlier trajectories where the RLM struggles to find an answer.

We additionally report runtime numbers of each method in Figures 12 , 13 in Appendix F , but we note several important caveats. Unlike API costs, these numbers are heavily dependent on implementation details such as the machine used, API request latency, and the asynchrony of LM calls. In our implementation of the baselines and RLMs, all LM calls are blocking / sequential. Nevertheless, similar to costs, we observe a wide range of runtimes, especially for RLMs.

Observation 5: Beyond long-context, RLMs enable longer reasoning capabilities. In Table 2 , we report RLM performance on LongCoT-mini ( Motwani et al., 2026 ) , a challenging long reasoning benchmark where frontier models solve compositional problems containing interdependent subproblems. We compare with the best model reported in the paper, GPT-5.2, and find that RLM(GPT-5.2, depth=1) uses the REPL to outperform the base model. Furthermore, when providing explicit hints on how to decompose tasks, we find the RLM is able to reliably generate a graph of the problem, solving each node using sub-calls as it programmatically traverses the reasoning graph. It outperforms the base model on all domains and by a 69.5 % 69.5\% performance increase overall.

Observation 6: Training RLMs on one domain can improve general downstream RLM performance, as well as efficiency. Training also exhibits length generalization. Certain behaviors in RLM trajectories are common among different domains, such as probing the input and recursively sub-calling on shorter contexts. In Figure 3 (a), we find that RLM-Qwen3-8B , a Qwen3-8B model that we fine-tuned on RLM(Qwen3-Coder-480B-A35B) trajectories on a small, unrelated set of tasks (LongBenchPro; Chen et al. 2026 ) considerably outperforms the base Qwen3-8B as a RLM across all tasks. Furthermore, its inference costs are much lower and more than 3 × 3\times faster (see Figure 6 in Appendix A ) due to better decision making and fewer mistakes as a RLM. Furthermore, we find that training RLMs exhibits length generalization; in Figure 3 (b), we train Qwen3-4B-Instruct-0527 as an RLM(depth=1) on MRCRv2 ( Vodrahalli et al., 2024 ) , a synthetic long-context task where the model must count and reproduce instances of a body of text in a corpus. By purely training through reinforcement learning with verifiable rewards (RLVR) on a smaller split, we find that RLM(Qwen3-4B-Instruct-0527) is able to generalize to the longer, more difficult split.

## 5 Analyses of RLM Trajectories

RLMs exhibit interesting context and problem decomposition behavior. We discuss observable behavior in small and large LLMs as RLMs to understand how we can steer and improve their performance and efficiency through training and prompt tuning.

Observed RLM decomposition patterns. Current models as RLMs attempt to probe, then decompose a task into sub-tasks for recursive sub-calls to solve. In many cases such as on BrowseComp-Plus, the LM uses model priors to programmatically narrow the search space of sub-calls. RLMs are also able to output beyond their context window by stitching together sub-LM calls inside the REPL, which is required to solve tasks like OOLONG-Pairs. We detail particular trajectories in Appendix E .

First decomposition and errors in RLM trajectories. RLMs defer essentially unbounded-length reasoning chains to sub-LM calls. The choice of decomposition can greatly affect task performance, especially for information-dense problems. In Figure 4 (a), we ablate how sensitive RLM behavior is to in-context decomposition examples in its system prompt on OOLONG. We find that in-context RLM trajectories greatly improve both overall performance and the initial decomposition attempt made by the RLM, even if the example is unrelated to the actual task. Furthermore, while RLMs frequently recover from an initially incorrect decomposition pattern, we find that the first decomposition attempt is important for overall performance. In Figure 4 (b), we plot how many RLM(depth=1) trajectories in Table 1 contains syntax errors. We find that RLM(Qwen3-Coder) trajectories contain significantly more syntax errors, even for correct trajectories, compared to RLM(GPT-5). These errors explain why higher recursion depths for RLM(Qwen3-Coder) perform worse on average : Qwen3-Coder-480B-A35B often makes syntax errors that result in failed outputs, and having sub-RLM calls propagates this issue to sub-calls. We include additional analysis for erroneous RLM behavior in Appendix F.1 .

## 6 Related Works

Long-Context LM Systems. There have primarily been two orthogonal directions for long-context management in language model systems: 1) directly changing the architecture of and retraining the base LM to handle longer contexts ( Press et al., 2022 ; Gu et al., 2022 ; Munkhdalai et al., 2024 ) , and 2) building a scaffold around the LM that implicitly handles the context – RLMs focus on the latter. One popular class of such strategies is lossy context management ( Chen et al., 2023 ) , which uses compaction or truncation to compress the input context at the cost of potentially losing fine-grained information. For example, ReSum ( Wu et al., 2025 ) adds a summarization tool to periodically compress the context of a multi-turn agent. Another class of strategies implement an explicit memory hierarchy in the agent scaffold ( Packer et al., 2024 ; Chhikara et al., 2025 ; Zhang et al., 2025 ) . RLMs differ from these works in that all context window management is implicitly handled by the LM itself.

Task Decomposition through sub-LM calls. Many LM-based agents ( Guo et al., 2024 ; Anthropic, 2025 ) use multiple, well-placed LM calls to solve a problem; however, many of these calls are placed based on human-engineered workflows. Several methods like ViperGPT ( Surís et al., 2023 ) , THREAD ( Schroeder et al., 2025 ) , ReDel ( Zhu et al., 2024 ) , Context Folding ( Sun et al., 2025 ) , and AgentFold ( Ye et al., 2025 ) have explored deferring the choice of sub-LM calls to the LM. These techniques emphasize task decomposition through recursive LM calls, but are unable to handle long context inputs beyond the length of the base LM. DisCIPL ( Grand et al., 2025 ) generates programs with sub-LM calls, but these programs are generated in a single-step and cannot recover from generation mistakes. RLMs, on the other hand, are enabled by an extremely simple intuition (i.e., placing the prompt in the external environment) to symbolically manipulate arbitrarily long strings and to iteratively refine their recursion via execution feedback from the persistent REPL.

## 7 Limitations and Future Work

While RLMs show strong performance on tasks beyond the context window limitations of existing LMs at reasonable inference costs, evaluations for more difficult and natural long-context processing tasks and the best mechanisms for implementing guardrails for RLMs both remain highly under-explored. Broadly, RLMs add a layer of complexity on top of existing LMs that may lead to unintentional side-effects like exploding sub-call costs, which we leave for future work to solve. We also note that future strategies involving asynchronous sub-calls and sandboxed REPLs can potentially significantly reduce the runtime and inference cost of RLMs, but further contribute to this complexity. We include additional limitations and negative results in Appendix B .

Lastly, we focused our experiments on evaluating RLMs using existing frontier models, but show initial evidence on a Qwen3-8B model that explicit training as a RLM provides very rapid performance improvements, even outside the training domain. We hypothesize that RLM trajectories can be viewed as a form of reasoning ( OpenAI et al., 2024 ; DeepSeek-AI et al., 2025 ) , which can be trained by bootstrapping existing models ( Zelikman et al., 2022 ; Zelikman et al., 2024 ) . We hope that training native RLMs can be treated as a new axis of scale to improve LM performance on general and long-horizon tasks.

## 8 Conclusion

We introduced Recursive Language Models (RLMs), a general inference framework for language models that offloads the input context and enables language models to recursively sub-query language models before providing an output. We explored an instantiation of this framework that offloads the context into a Python REPL environment as a variable in memory, enabling the LM to reason over its context in code and recursive LM calls, rather than purely in token space. Our results across multiple settings and models demonstrated that RLMs are an effective task-agnostic paradigm for both long-context problems and general reasoning. Building on our small fine-tuning experiments, we are excited to see future work that explicitly trains models to reason as RLMs, which could result in another axis of scale for the next generation of language model systems.

## Appendix A Additional Training Details