Dream-RSI: Recursive Self-Improvement through Evolving Worlds — part 2/3
Cleaned-text code points [15919, 31871). Bibliography omitted here; source.md includes it.
For each policy–tree pair ( m , i ) (m,i) , replay resets the policy’s per-rollout state and starts from 𝒯 i m , 0 = { r } \mathcal{T}_{i}^{m,0}=\{r\} . Here, 𝒯 i m , k ⊆ 𝒯 i \mathcal{T}_{i}^{m,k}\subseteq\mathcal{T}_{i} denotes the subtree revealed after k k completed rounds. The full recorded tree 𝒯 i \mathcal{T}_{i} remains fixed; only the portion observed by the policy evolves. At each decision, π t m \pi_{t}^{m} selects a batch C i m , k ∈ A ( 𝒯 i m , k , W ) C_{i}^{m,k}\in A(\mathcal{T}_{i}^{m,k};W) using the revealed observations. Unlike online execution, replay returns recorded children of the selected nodes deterministically rather than generating new candidates. After the exploration policy takes a nonempty batch C i m , k C_{i}^{m,k} , the next observed tree is 𝒯 i m , k + 1 = 𝒯 i m , k ∪ ⋃ v ∈ C i m , k Child ( v , 𝒯 i , 𝒯 i m , k ) \mathcal{T}_{i}^{m,k+1}=\mathcal{T}_{i}^{m,k}\cup\bigcup_{v\in C_{i}^{m,k}}\operatorname{Child}(v;\mathcal{T}_{i},\mathcal{T}_{i}^{m,k}) where Child ( v , 𝒯 i , 𝒯 i m , k ) \operatorname{Child}(v;\mathcal{T}_{i},\mathcal{T}_{i}^{m,k}) denotes the node set containing unobserved children of v v on tree 𝒯 i \mathcal{T}_{i} given the current observed tree T i m , k T_{i}^{m,k} . For v ≠ r v\neq r , Child ( v , 𝒯 i , 𝒯 i m , k ) \operatorname{Child}(v;\mathcal{T}_{i},\mathcal{T}_{i}^{m,k}) is v v ’s unique recorded child, if one exists. Since v v is a leaf of 𝒯 i m , k \mathcal{T}_{i}^{m,k} , that child is still unrevealed. For v = r v=r , replay returns the earliest-created child of r r outside 𝒯 i m , k \mathcal{T}_{i}^{m,k} , opening one previously unrevealed branch. In either case, Child ( v , 𝒯 i , 𝒯 i m , k ) = ∅ \operatorname{Child}(v;\mathcal{T}_{i},\mathcal{T}_{i}^{m,k})=\emptyset when no recorded continuation remains. The newly revealed nodes expose their stored observations before the policy makes its next decision.
Replay allows at most K 2 K_{2} decision rounds where each nonempty batch counts as one round, and terminates when the policy selects C i m , k = ∅ C_{i}^{m,k}=\emptyset , the round limit k = K 2 k=K_{2} is reached, or 𝒯 i m , k = 𝒯 i \mathcal{T}_{i}^{m,k}=\mathcal{T}_{i} , meaning that all recorded nodes have been revealed. Let k i m , ⋆ ∈ { 0 , … , K 2 } k_{i}^{m,\star}\in\{0,\ldots,K_{2}\} denote the number of completed rounds at termination, yielding the final subtree 𝒯 i m , k i m , ⋆ ⊆ 𝒯 i \mathcal{T}_{i}^{m,k_{i}^{m,\star}}\subseteq\mathcal{T}_{i} .
Thus, replay evaluates how far to pursue each opened branch, how to group attempts into parallel batches, and when to open another branch or stop. These decisions may differ across policies, but each branch is traversed in its recorded parent–child order, and no outcomes beyond 𝒯 i \mathcal{T}_{i} are generated.
#### Replay objective.
The replay objective balances discovery quality, execution cost, and parallelism. Let N i m = | 𝒯 i m , k i m , ⋆ | − 1 N_{i}^{m}=|\mathcal{T}_{i}^{m,k_{i}^{m,\star}}|-1 be the number of revealed non-root nodes. Although replay itself does not execute new discovery attempts, N i m N_{i}^{m} counts the generation–evaluation requests represented by its trajectory. For fixed coefficients β 1 , β 2 ≥ 0 \beta_{1},\beta_{2}\geq 0 , the replay score is V i m = max v ∈ 𝒯 i m , k i m , ⋆ s v ⏟ discovery quality − β 1 N i m ⏟ execution cost + β 2 N i m max { 1 , k i m , ⋆ } ⏟ parallelism bonus . V_{i}^{m}=\underbrace{\max_{v\in\mathcal{T}_{i}^{m,k_{i}^{m,\star}}}s_{v}}_{\text{discovery quality}}-\underbrace{\beta_{1}N_{i}^{m}}_{\text{execution cost}}+\underbrace{\beta_{2}\frac{N_{i}^{m}}{\max\{1,k_{i}^{m,\star}\}}}_{\text{parallelism bonus}}. (1) The first term measures the best solution quality attained during replay. The second penalizes the number of attempted generations. For a nonempty replay, the third rewards the average number of attempts executed per decision round, favoring policies that batch useful continuations rather than execute them sequentially.
#### Policy improvement and selection.
The evaluation score of policy version π t m \pi_{t}^{m} is its average replay score across the fixed history, V m = 1 t ∑ i = 1 t V i m V^{m}=\frac{1}{t}\sum_{i=1}^{t}V_{i}^{m} . The offline phase begins by evaluating the current policy π t 0 = π t \pi_{t}^{0}=\pi_{t} . For each m = 0 , … , M − 1 m=0,\ldots,M-1 , the policy-development agent examines the replay trajectories and scores of π t m \pi_{t}^{m} , together with feedback from earlier revisions, to identify successful decisions and recurring failures. It then revises the executable policy code to produce π t m + 1 \pi_{t}^{m+1} , which is evaluated on the same t t replay worlds. Replay feedback is available to the development agent between revisions.
After M M revisions, the next online policy is selected from all M M evaluated versions as π t + 1 = π t m ⋆ \pi_{t+1}=\pi_{t}^{m^{\star}} , where m ⋆ ∈ arg max m ∈ { 0 , … , M − 1 } V m m^{\star}\in\operatorname*{arg\,max}_{m\in\{0,\ldots,M-1\}}V^{m} . Because the candidate set includes the current policy, this selection satisfies V m ⋆ ≥ V 0 V^{m^{\star}}\geq V^{0} . Thus, the selected policy π t + 1 \pi_{t+1} is no worse than the current policy π t \pi_{t} in average replay score on the fixed history ℋ t \mathcal{H}_{t} . The selected policy is then deployed online to collect 𝒯 t + 1 \mathcal{T}_{t+1} , expanding the history available for the next offline improvement phase.
## 4 Experiments
We evaluate Dream-RSI across three scientific discovery domains: algorithm engineering, kernel optimization and math optimization. Our primary controlled baseline is Recursive Fixed Exploration, which uses the same underlying discovery setting and initialization but keeps the exploration policy fixed across recursive discovery rounds. We additionally compare against task-specific domain baselines.
Across all tasks, Dream-RSI and Recursive Fixed Exploration use the same discovery agent, evaluator, initialization, and resource constraints. Both methods start from the same manually designed exploration policy. This exploration policy follows a simple parallel refining strategy: it launches multiple independent exploration workspaces in parallel, with each workspace maintaining its own local discovery trajectory and repeatedly refining its current candidate based on the history accumulated within that workspace. The two methods therefore follow the same exploration policy in the first discovery round. In subsequent rounds, while Recursive Fixed Exploration keeps its exploration policy static, Dream-RSI progressively refines the policy by dreaming over a replay simulator conditioned on accumulated global discovery history, subsequently deploying the updated policy in each new round. The discovery cost is quantified by the total cumulative number of discovery-agent calls.
Specifically, we evaluate Gemini-3.1 Pro and Gemini-3.7-Flash across multiple recursive discovery rounds via the Gemini CLI 2 2 2 https://geminicli.com/ . Under Recursive Fixed Exploration, each round for Gemini-3.1 Pro executes 10 parallel workspaces with up to 11 refinement steps ( 10 × 11 = 110 10\times 11=110 discovery-agent calls), whereas Gemini-3.7-Flash operates 32 parallel workspaces with up to 20 refinement steps ( 32 × 20 = 640 32\times 20=640 calls). Dream-RSI maintains identical per-round budgets, aligning with the baseline in Round 1 while progressively updating its policy in subsequent rounds. Further details on recursive rounds, task setups, resource budgets, and evaluation protocols follow below.
(a) Final performance.
(b) Recursive Discovery Dynamics.
### 4.1 Algorithm Engineering
In this task, we consider Lasso Regularization Path as our algorithm-engineering task, a fundamental computational primitive in high-dimensional statistics that is widely used in model selection and cross-validation across domains such as genomics and finance. We follow the benchmark setting of SimpleTES ( Ye et al., 2026 ) , where the goal is to discover efficient implementations of the complete Lasso regularization path while preserving numerical correctness. During discovery, we use the same 17 synthetic instances as SimpleTES, which cover diverse problem regimes in terms of dimensionality, sparsity, feature correlation, and active-set structure. To evaluate whether the discovered algorithms generalize beyond the search distribution, we additionally evaluate them on six held-out downstream datasets spanning both biological and non-biological domains.
#### Baselines and Setup.
We compare against standard Lasso solvers sklearn ( Pedregosa et al., 2011 ) and glmnet ( Friedman et al., 2010 ) , as well as SimpleTES ( Ye et al., 2026 ) , which uses GPT-OSS-120B with a reported budget of 51,200 generations. We additionally include Recursive Fixed Exploration as our controlled baseline. Specifically, we run both Recursive Fixed Exploration and Dream-RSI for 5 rounds.
#### Main Results.
Figure 3 (a) summarizes the Lasso discovery results. Across both discovery-agent backbones, Dream-RSI achieves a better downstream quality–compute trade-off than Recursive Fixed Exploration. With Gemini-3.1 Pro, it reduces the average runtime across the six held-out datasets from 3587.1 ms to 2931.0 ms while using only 317 discovery-agent calls, compared with 550 calls for fixed exploration. With Gemini-3.7-Flash, Dream-RSI further reduces the average runtime from 2516.7 ms to 2350.6 ms using 1879 calls instead of 3200. Despite using substantially less discovery compute, the resulting solvers also outperform the standard sklearn and glmnet implementations on all six held-out datasets. Compared with SimpleTES, which uses 51,200 generations, Dream-RSI achieves lower average downstream runtime with roughly two orders of magnitude fewer discovery-agent calls. Notably, the program discovered by Gemini-3.1-Pro appears particularly well suited to large-scale matrices such as RCV1. In contrast, Gemini-3.7-Flash discovers a more general-purpose program that performs consistently across different problem scales.
#### Recursive Discovery Dynamics.
Figure 3 (b) illustrates the trajectory of downstream performance across recursive discovery rounds relative to cumulative discovery compute. By design, both methods share identical search behavior in the initial round. In subsequent rounds, Recursive Fixed Exploration maintains a static exploration policy, whereas Dream-RSI progressively refines and redeploys its policy via dreaming over accumulated discovery history. Consequently, the two trajectories diverge markedly: Dream-RSI consistently achieves superior downstream performance while requiring substantially lower cumulative compute across both Gemini-3.1-Pro and Gemini-3.7-Flash.
#### Discovered Solver Analysis.
We further analyze the discovered solver, with its implementation provided in the Appendix C . Unlike SimpleTES, which switches between LARS and coordinate descent according to problem dimensions, the discovered solver introduces adaptivity within the active-set optimization itself. It combines strong-rule screening with Cauchy–Schwarz-based KKT pruning, selectively recomputing exact gradients only when the bound cannot certify a feature and falling back to a full refresh when pruning becomes ineffective. This adaptive verification scheme is further integrated with efficient active-set bookkeeping, lazy Gram-matrix construction, and hardware-aware implementation.
### 4.2 Mathematics Optimization
We further evaluate Dream-RSI on three mathematical discovery tasks spanning discrete combinatorial optimization, geometric optimization, and functional optimization: the Sum–Difference Problem , Circle Packing , and Autocorrelation Inequalities . The goal of these problems is to discover high-quality solutions that optimize task-specific mathematical objectives under their respective constraints. Formal definitions of the three tasks are provided in Appendix.
We use Gemini-3.1 Pro via the Gemini CLI as the discovery agent for both Recursive Fixed Exploration and Dream-RSI for 10 rounds. For each task, the agent iteratively proposes and evaluates candidate constructions or optimization procedures according to the task-specific objective. We compare against a broad set of existing automated discovery systems, including AlphaEvolve ( Novikov et al., 2025 ) , AlphaEvolveV2 ( Georgiev et al., 2025 ) , OpenEvolve ( Sharma, 2025 ) , CodeEvolve ( Assumpção et al., 2025 ) , ShinkaEvolve ( Lange et al., 2026 ) , TTS-Discovery ( Yuksekgonul et al., 2026 ) , ThetaEvolve ( Wang et al., 2025 ) , EvoX ( Liu et al., 2026a ) , and SimpleTES ( Ye et al., 2026 ) .
#### Results.
Table 1 summarizes the results across the three mathematical discovery tasks. Dream-RSI achieves a Sum–Difference score of 1.145427 1.145427 , outperforming SimpleTES and Recursive Fixed Exploration. On Circle Packing, it reaches 2.635983 2.635983 , matching the strongest reported result among the compared methods. For Autocorrelation, Dream-RSI obtains 1.456375 1.456375 , remaining competitive with existing discovery systems. Notably, SimpleTES achieves state-of-the-art performance on Autocorrelation Inequalities, but requires 51,200 generations, significantly more than the fewer than 1,000 generations used by our approach. Overall, these results show that our Dream-RSI generalize well on mathematics optimization.
### 4.3 Kernel Engineering
We further evaluate Dream-RSI on GPU kernel engineering , where the goal is to automatically discover high-performance implementations of kernels while preserving numerical correctness. Unlike mathematical discovery, kernel engineering requires reasoning jointly about algorithmic structure, memory access, parallelization, and hardware-specific optimizations, providing a substantially different testbed for evaluating whether our Dream-RSI generalizes across discovery domains.
We consider four representative kernel-engineering tasks from KernelBench ( Ouyang et al., 2025 ) : VGG16 , LayerNorm , ConvDiv , and ConvMax . Candidate implementations are evaluated by their execution performance, measured as inverse runtime ( 1 / ms 1/\mathrm{ms} ), subject to correctness checks against the reference implementation. We use Gemini-3.1 Pro as the coding agent and compare Dream-RSI with Recursive Fixed Exploration under the same evaluation protocol and initialization.
#### Results.
Figure 4 shows the discovery trajectories as the number of generations increases. On VGG16 and LayerNorm, Dream-RSI reaches comparable final performance using 2.43 × 2.43\times and 1.79 × 1.79\times fewer generations, respectively. On ConvDiv and ConvMax, under comparable discovery budgets, Dream-RSI achieves 2.09 × 2.09\times and 1.44 × 1.44\times higher performance, respectively. These results show that adapting the exploration policy across recursive rounds can improve the efficiency and effectiveness of long-horizon discovery.
## 5 Further Analysis
### 5.1 Analysis of Historical Inductive Biases in Long-Horizon Discovery
We further investigate how the nature of the historical inductive bias affects long-horizon discovery. A natural alternative for utilizing history is to abstract prior trajectories into high-level directional insights, which are directly injected into the prompt as explicit semantic guidance for subsequent rounds. To evaluate the efficacy of this prompt-level semantic guidance, we apply it to both Recursive Fixed Exploration and Dream-RSI . As illustrated in Figure 5 , explicit directional guidance consistently underperforms its unguided counterpart across both paradigms under equivalent discovery budgets. These results suggest that in long-horizon discovery—where multiple parallel threads are deployed for exploration—imposing strong semantic inductive biases regarding future search directions tends to over-constrain the search space and impede diverse exploration.