Dream-RSI: Recursive Self-Improvement through Evolving Worlds — part 1/3

Cleaned-text code points [0, 15919). Bibliography omitted here; source.md includes it.

\uselogo

# Dream-RSI: Recursive Self-Improvement through Evolving Worlds

###### Abstract

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce Dream-RSI , a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, Dream-RSI secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings. github.com/zhengkid/Dream-RSI | dream-rsi.com

## 1 Introduction

Recursive self-improvement (RSI) has emerged as an ambitious goal for autonomous AI systems ( Liu et al., 2026c ) . A common mechanism underlying RSI is an iterative discovery loop wherein agents generate candidate solutions, evaluate outcomes, incorporate feedback, and refine future iterations. Such discovery loops have driven substantial progress across scientific and algorithmic domains, including algorithm design ( Novikov et al., 2025 ; Romera-Paredes et al., 2024 ) , open-ended mathematical optimization ( Georgiev et al., 2025 ; Anthropic, 2026 ) , systems design ( Jaber and Jaber, 2026 ; Cao et al., 2026 ) , and agent self-improvement ( Zhang et al., 2026b ; Zhang et al., 2026c ; Lee et al., 2026 ; Zheng et al., 2026a ) , with these discoveries increasingly feeding into the development of more capable AI systems. As agent capabilities improve and self-improvement targets become challenging, discovery increasingly requires long-horizon exploration over vast search spaces, often spanning thousands of proposal–evaluation cycles ( Ye et al., 2026 ; OpenAI, 2026 ) . At this scale, the ability to orchestrate exploration becomes critical ( Zheng et al., 2026b ) . Poor exploration can waste substantial computation and time, severely limiting the efficiency and scalability of RSI.

Existing approaches have largely relied on manually designed exploration strategies that remain largely fixed throughout discovery ( Novikov et al., 2025 ; Yan et al., 2026b ; Du et al., 2026 ; Jiang et al., 2026 ; Ye et al., 2026 ) . Fixed strategies cannot improve from accumulated discovery experience and may repeatedly allocate computation to ineffective search directions. Recent work therefore seeks to optimize exploration policies online during discovery ( Liu et al., 2026a ) , but doing so faces two fundamental bottlenecks. First, feedback is delayed and expensive at the meta level: unlike evaluating an individual candidate, assessing an exploration policy requires observing how it shapes the subsequent discovery process over many proposal–evaluation cycles. Second, the meta-policy space is vast: a newly proposed policy may perform poorly, so many alternatives may need to be tried. Together, these challenges make meta-level improvement particularly costly: each policy may require a long online rollout before receiving useful feedback, making it difficult to efficiently close the self-improvement loop at the exploration layer.

To address these bottlenecks, our key intuition is simple: a fast and inexpensive simulator of discovery would allow many exploration policies to be evaluated before costly online deployment. Surprisingly, completed discovery histories already provide such a simulator. While prior work treats past discovery history merely as static textual context ( Hu et al., 2025 ; Ouyang et al., 2026b ) or training data for weight fine-tuning ( Yuksekgonul et al., 2026 ; Wang et al., 2025 ) , a completed discovery process inherently records a structured tree of past exploration decisions and their realized code-execution outcomes. Drawing an analogy to model-based reinforcement learning and World Models ( Ha and Schmidhuber, 2018 ; Hafner et al., 2023 ) (§ 2 ), once organized into a discovery tree, this history can serve as a replay simulator 1 1 1 We use the terms replay simulator and worlds interchangeably. . As illustrated in Figure 2 , an alternative exploration strategy can navigate this pre-recorded tree to traverse different subsets of recorded branches, in different orders, with different parallel groupings and stopping decisions. Because all execution outcomes are already saved in the tree, evaluating a new strategy requires only reading past records without rerunning the underlying discovery agent or evaluator. This transforms meta-policy improvement from an expensive online trial-and-error process into a fast, simulation-based “dreaming” procedure.

Building on this insight, we introduce Dream-RSI , a framework for scalable and recursively self-improving meta-exploration in agent-driven discovery. We first make exploration explicit and programmable through a lightweight orchestration layer that controls branching, parallel exploration, and stopping while leaving the underlying coding agent unchanged. Rather than keeping this policy fixed, Dream-RSI establishes a closed-loop self-improvement mechanism across three core stages (Figure 1 ): (1) Online Exploration , where the current policy guides real-world discovery and logs historical execution traces; (2) Simulator Construction , where recorded discovery trees are converted into a reusable replay simulator pool; and (3) Dreaming-based Policy Improvement , where candidate policies are evaluated via low-cost "dreaming" over the simulator. The updated policy is then redeployed online to generate new discovery experience and expand the simulator pool, closing a RSI loop at the meta-exploration layer.

Empirically, we evaluate Dream-RSI across 8 scientific discovery tasks spanning three distinct domains: algorithm engineering, mathematical optimization, and GPU kernel engineering. In algorithm engineering (Lasso path solver), Dream-RSI outperforms standard libraries like sklearn and strong baselines while reducing agent calls by up to 162 × 162\times over SimpleTES and 1.7 × 1.7\times over fixed-exploration baselines. In mathematical optimization (sum-difference, autocorrelation, circle packing), it matches or surpasses strong baselines within 1 ​ k 1\text{k} generations, yielding over 50 × 50\times budget savings compared to SimpleTES . In GPU kernel engineering (KernelBench), it either reaches target execution speeds using 1.79 × 1.79\times – 2.43 × 2.43\times fewer generations or improves kernel performance by up to 2.09 × 2.09\times under identical budget constraints.

In summary, our main contributions are as follows: 1) History as Replay Simulator: We conceptualize completed discovery histories as replay simulators. This makes delayed exploration feedback reusable for efficient meta-exploration policy evaluation.; 2) Meta-Layer RSI Loop ( Dream-RSI ): We introduce Dream-RSI , establishing a recursive self-improvement loop that continuously collects discovery histories through online exploration, constructs replay simulators from history to refine meta-exploration strategies via dreaming, and redeploys the upgraded policy online; 3) Empirical Validation: We conduct experiments to demonstrate that Dream-RSI improves both discovery effectiveness and efficiency in several settings.

## 2 Motivation: Discovery History as a Replay Simulator

Consider an agent navigating toward a goal in an unfamiliar environment. During its first traversal, the agent may follow inefficient routes, encounter dead ends, backtrack, and gradually construct a map of the surrounding space. Once recorded, however, this experience becomes reusable: the resulting map supports planning without requiring the agent to physically revisit every location. A new navigation policy can instead reason over the accumulated map, avoid known dead ends, reconsider earlier decisions, and compare alternative routes before acting ( Gupta et al., 2017 ) .

This idea parallels model-based reinforcement learning ( Sutton, 1990 ; M. Moerland et al., 2023 ) . A model captures how an environment evolves in response to an agent’s actions, allowing policies to be trained or evaluated through simulated experience rather than repeated interaction with the real environment ( Ha and Schmidhuber, 2018 ) . The Dreamer family ( Hafner et al., 2019 ; Hafner et al., 2020 ; Hafner et al., 2023 ; Hafner et al., 2025 ) demonstrates this principle particularly clearly: an agent learns a compact dynamics model from collected experience and improves its policy by imagining trajectories within that model.

Long-horizon discovery admits an analogous structure. An exploration policy decides which directions to pursue, which candidates to refine, which branches to explore in parallel, and when to terminate. Executing the policy online produces a structured discovery history containing the explored branches, decision points, computational costs, and realized outcomes. As illustrated in Figure 2 , this history can subsequently be treated as an empirical replay simulator : a grounded model of the portion of the discovery space that has already been observed.

Within this replay simulator, alternative exploration policies induce different trajectories through the recorded discovery tree. A policy may select a different subset of branches, prioritize them in a different order, issue different requests in parallel, or stop at an earlier point. Evaluating such a trajectory requires only revealing the outcomes already stored along the selected branches, rather than rerunning the underlying coding agent and evaluator. Consequently, a single expensive online discovery run can support many inexpensive evaluations of alternative exploration strategies.

## 3 Dream-RSI : Recursive Self-Improvement through Evolving Worlds

As shown in Figure 1 , Dream-RSI alternates between online exploration and offline “dreaming” to improve an executable exploration policy that allocates discovery computation. During the online phase, the policy guides a fixed discovery agent , while a fixed evaluator scores the resulting candidates and provides diagnostic feedback. The resulting discovery tree serves as a replay world in which alternative policies can be evaluated using recorded outcomes. A fixed LLM-based policy-development agent uses this feedback to revise the exploration policy code, and the best evaluated version is deployed for the next online rollout. Only the exploration-policy code changes; the underlying models, evaluator, and execution interfaces remain fixed.

#### Discovery trees and the shared decision interface.

A discovery tree is rooted at r r , which represents the initial workspace state. Each non-root node v v has exactly one primary parent , either the root or a previously created node. This parent identifies where the attempt in v v begins: the discovery agent resumes the parent’s saved workspace and uses its accumulated observations as context to produce a new attempt. Node v v preserves this inherited history and records the outcome of the new generation–evaluation attempt, including the resulting filesystem snapshot, generated artifact, evaluation diagnostics, and score s v s_{v} . Scores follow a fixed task-scoring protocol, with larger values indicating better quality.

In both online execution and offline replay, the exploration policy observes a tree 𝒯 \mathcal{T} , initially containing only the root, and selects the nodes from which to continue exploration. The eligible nodes form the set A ⁡ ( 𝒯 ) = { r } ∪ { v ∈ 𝒯 : v ​ is a leaf } A(\mathcal{T})=\{r\}\cup\{v\in\mathcal{T}:v\text{ is a leaf}\} , where leaves are determined from the currently observed tree. Let W ≥ 1 W\geq 1 be the number of parallel workers, each of which can execute one generation–evaluation request at a time (e.g. concurrent API calls). The exploration policy’s action is a batch C ∈ A ⁡ ( 𝒯 , W ) C\in A(\mathcal{T};W) , where A ⁡ ( 𝒯 , W ) = { C ⊆ A ⁡ ( 𝒯 ) : | C | ≤ W } A(\mathcal{T};W)=\{C\subseteq A(\mathcal{T}):|C|\leq W\} is the feasible batch set. Each selected node specifies the starting point of one attempt, so the batch determines both where exploration continues and how many attempts are scheduled in parallel. Both the online and offline phases use this same decision interface but differ in the transition that follows a selected batch.

#### Online rollout.

Let t = 1 , 2 , … t=1,2,\ldots index the outer iterations, starting from an initial policy π 1 \pi_{1} and an empty history ℋ 0 = ( ) \mathcal{H}_{0}=() . At iteration t t , policy π t \pi_{t} guides a new online rollout with access to the completed discovery history ℋ t − 1 \mathcal{H}_{t-1} . This history provides context for exploration but remains separate from the new tree being constructed. The policy code stays fixed throughout the rollout.

Let 𝒯 t k \mathcal{T}_{t}^{k} denote the new discovery tree after k k completed decision rounds, with 𝒯 t 0 = { r } \mathcal{T}_{t}^{0}=\{r\} . The rollout allows at most K 1 K_{1} rounds. At round k ≤ K 1 k\leq K_{1} , the exploration policy chooses a node batch C t k ∈ A ⁡ ( 𝒯 t k , W ) C_{t}^{k}\in A(\mathcal{T}_{t}^{k};W) and each node v ∈ C t k v\in C_{t}^{k} is assigned to a worker. The discovery agent uses v v ’s saved workspace and available context to produce a new candidate, and the evaluator assesses the result. These attempts run in parallel, each producing one new child of its selected parent. Attaching the completed children to the current tree yields 𝒯 t k + 1 \mathcal{T}_{t}^{k+1} , while all previously recorded nodes remain unchanged. This transition is stochastic because the discovery agent may generate different outcomes from the same starting workspace. For the next round, the newly created child becomes the selectable leaf of an extended branch, while the root remains selectable for opening further branches. The rollout ends when the policy selects an empty batch or completes K 1 K_{1} decision rounds. After the rollout terminates, its final tree is recorded as 𝒯 t \mathcal{T}_{t} and appended to the history, giving ℋ t = ℋ t − 1 ∪ { 𝒯 t } \mathcal{H}_{t}=\mathcal{H}_{t-1}\cup\{\mathcal{T}_{t}\} . The method then enters the offline phase using this expanded collection of replay worlds.

#### Offline evaluation.

During the offline phase of outer iteration t t , the history ℋ t \mathcal{H}_{t} remains fixed while the method constructs and evaluates M ≥ 1 M\geq 1 policy versions π t 0 , … , π t M − 1 \pi_{t}^{0},\ldots,\pi_{t}^{M-1} , starting with π t 0 = π t \pi_{t}^{0}=\pi_{t} . Each version is evaluated separately on every historical tree 𝒯 i \mathcal{T}_{i} , i = 1 , … , t i=1,\ldots,t , before the next version is developed from the resulting feedback. We use m m to index policy versions, i i to index replay worlds, and k k to count decision rounds within one policy–world evaluation. The outer index t t is fixed throughout this phase and is suppressed in the notation for replay trajectories and scores.