Dream-RSI: Recursive Self-Improvement through Evolving Worlds — part 3/3

Cleaned-text code points [31871, 39002). Bibliography omitted here; source.md includes it.

### 5.2 Analysis of Evolution of Exploration Behavior

(a) Round-best performance

(b) Exploration effort

Figure 6 illustrates how the learned exploration policy evolves across recursive rounds on ConvDiv. As shown, the exploration policy exhibits a clear adaptive pattern: as performance improves, it initially conserves discovery compute (e.g., reducing the number of evaluated attempts from 110 to 50). When progress subsequently plateaus, it increases exploration effort again, coinciding with further performance gains..

## 6 Related Work

#### AI-Driven Scientific and Algorithmic Discovery.

LLM-based discovery systems iteratively generate, evaluate, and refine candidate solutions using prior artifacts and feedback, as in AlphaEvolve ( Novikov et al., 2025 ) , OpenEvolve ( Sharma, 2025 ) , CodeEvolve ( Assumpção et al., 2025 ) , ShinkaEvolve ( Lange et al., 2026 ) , PACEvolve ( Yan et al., 2026b ) , DeltaEvolve ( Jiang et al., 2026 ) and MLEvolve Du et al. (2026) . More recent work emphasizes the importance of exploration itself: SkyDiscover provides adaptive discovery infrastructure ( Liu et al., 2026b ) , SwarmResearch dynamically orchestrates multiple search branches ( Virk et al., 2026 ) , and EvoX ( Liu et al., 2026a ) explicitly optimizes search strategies rather than only candidate solutions. This shift makes exploration a meta-level optimization problem, but useful supervision for exploration strategies is expensive and delayed because their quality often becomes apparent only after long discovery rollouts.

#### Self-Evolving Agents.

A broader line of work studies agents that improve their own components during interaction. Prior methods evolve model weights ( Huang et al., 2026c ; Huang et al., 2026a ) , agent harnesses ( Lee et al., 2026 ; Zhang et al., 2026b ) , contexts ( Zhang et al., 2026d ) , skills ( Zhang et al., 2026a ; Ouyang et al., 2026a ; Wu et al., 2026b ) , model behavior through test-time learning ( Wang et al., 2025 ; Yuksekgonul et al., 2026 ; Yan et al., 2026a ; Wu et al., 2026a ) , rubrics ( Xiong et al., 2026 ) , environments ( Huang et al., 2026b ) and other applications ( Dai et al., 2026 ) . Most operate at the object level , improving components used for task execution or reasoning. Recent work has begun to optimize meta-level mechanisms, including search strategies and self-improvement procedures ( Liu et al., 2026a ; Yan et al., 2026a ; Wang et al., 2026 ; Zhang et al., 2026c ; Kim et al., 2026 ) . However, such meta-level strategies are difficult to improve because their quality is often revealed only after costly long-horizon rollouts. Dream-RSI makes this meta-level optimization recursive and off-policy by turning accumulated discovery history into replay simulators, allowing exploration controllers to be repeatedly evaluated, improved, and redeployed without rerunning the underlying discovery process.

#### Memory, History, and Experience Reuse.

Prior work reuses agent experience as search history, context, memory, reusable skills, or training signals. DeltaEvolve structures evolutionary history through semantic deltas ( Jiang et al., 2026 ) ; SwarmResearch and MLEvolve use cross-branch or retrospective information to guide subsequent search ( Virk et al., 2026 ; Du et al., 2026 ) ; and other work improves how agents access and retain experience through evolving contexts, broader harness state, libraries, or skills ( Zhang et al., 2026d ; Lee et al., 2026 ; Xu et al., 2026 ; Ouyang et al., 2026a ) . We take a different view: rather than using exploration history only as context or memory for the next decision, we organize it as a replay simulator in which many alternative exploration controllers can be evaluated cheaply. This turns previously collected discovery experience into reusable feedback for meta-level optimization, alleviating the scarcity and high cost of training signals for improving exploration strategies.

## 7 Conclusion

We presented Dream-RSI , a framework for recursive self-improvement of exploration in recursive self improvement. By converting accumulated discovery history from static context into an active, replayable simulator, Dream-RSI addresses the core bottleneck of meta-optimization: delayed and expensive feedback, which is especially severe in long-horizon discovery settings. By ‘dreaming’ within replay simulators constructed from historical discovery trees, Dream-RSI evaluates candidate exploration policies rapidly and at negligible execution cost. The improved policies are then redeployed online to drive further discovery and expand the simulator pool, closing the recursive self-improvement loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

## Appendix A Detailed Task Description

## Appendix B Prompts

For reproducibility, we provide the prompts used for online exploration and replay-based exploration-policy improvement. Variables enclosed by dollar signs or braces are instantiated by the runtime system before execution.

### B.1 Exploration Prompt

The following prompt is used to guide the discovery agent during online exploration. It requires the agent to inspect the complete available discovery history before proposing a new solution, explicitly reason about both successful and failed attempts, and avoid repeatedly exploiting a locally saturated direction.

### B.2 Replay-Based Policy Improvement Prompt

The following prompt is used by the controller-development agent during historical replay. The agent modifies the exploration policy using feedback obtained from replay over previously collected discovery trajectories while remaining restricted to prefix-observable information.

## Appendix C Discovered Programs

We provide the complete implementation of the Lasso-path solver discovered by Dream-RSI . As discussed in Section 4.1 , the solver combines strong-rule screening with adaptive Cauchy–Schwarz KKT pruning, disjoint active-set bookkeeping, lazy Gram-matrix construction, and hardware-aware optimizations.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .