Self-Improvements in Modern Agentic Systems: A Survey — part 10/14

Cleaned-text code points [137592, 152731). Bibliography omitted here; source.md includes it.

Existing systems approach this challenge by advancing tool creation across three dimensions: (1) Synthesis Triggers. Tool creation can be driven either by immediate needs or open-ended curiosity. While systems like ATLASS and PyVision emphasize on-demand invention—synthesizing and refining tools when current retrieved APIs fail ( Haque et al., 2025 ; Zhao et al., 2025b ) , agents like FRIDAY and STELLA shift toward self-directed exploration. They autonomously propose curricula or discover domain-specific resources (e.g., bioinformatics) to proactively accumulate tools outside a fixed task list ( Wu et al., 2024b ; Jin et al., 2025 ) . (2) Lifecycle Automation. Synthesizing raw code is only the first step; rendering it executable is the actual bottleneck. TOOLMAKER addresses this by automating the end-to-end tool lifecycle, extracting logic from scientific papers, installing dependencies, debugging in a closed loop, and producing robust callable interfaces ( Wölflein et al., 2025 ) . This transforms abstract function generation into tangible deployability ( Cai et al., 2024 ; Yuan et al., 2024a ; Qian et al., 2023 ) . (3) Standardized Integration. To ensure that explosive tool growth does not destabilize existing routing policies, recent systems adopt protocol-driven architectures. Frameworks like Alita and Code2MCP automate the conversion of code repositories into standardized services via the Model Context Protocol (MCP) ( Qiu et al., 2025c ; Qiu et al., 2025b ; Ouyang et al., 2025a ) . Similarly, AgentOrchestra enforces a rigorous pipeline of intent parsing, validation, and formal registration before a new tool enters the active pool ( Zhang et al., 2025j ) . This ensures that autonomous creation remains fully compatible with the agent’s broader governance and retrieval constraints.

### 6.4 Full Scaffolding

Full scaffolding self-improvement represents the deepest level of architectural intervention. Rather than merely tuning isolated components (e.g., prompts, tools, or memory), the agent treats its entire operational logic and codebase as a mutable substrate, enabling fundamental structural reorganization. Formally, recalling the agent state 𝒜 t = ( θ t , Σ t ) \mathcal{A}_{t}=(\theta_{t},\Sigma_{t}) , full scaffolding improvement corresponds to the most general scaffolding-only transition: Σ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 Σ ⁡ ( Σ t ; 𝒮 t ) , \qquad\Sigma_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\Sigma}(\Sigma_{t};\mathcal{S}_{t}), (22) where 𝒮 t \mathcal{S}_{t} denotes the learning signal extracted from the agent’s own execution traces and evaluations (e.g., task success/failure, unit tests, self-critique, cost signals). Crucially, unlike settings with a fixed external meta-optimizer, the improvement procedure is itself implemented within the current scaffolding, making the update self-referential : Σ t + 1 = ℐ Σ t ​ ( Σ t , 𝒮 t ) , \Sigma_{t+1}=\mathcal{I}_{\Sigma_{t}}(\Sigma_{t};\mathcal{S}_{t}), (23) where ℐ Σ t \mathcal{I}_{\Sigma_{t}} highlights that the improver can evolve together with the agent it improves. In principle, these self-referential loops can discover unanticipated update strategies and enhance the agent’s intrinsic evolvability ( Zhang et al., 2026c ; Gerhart and Kirschner, 2007 ; Hendrikse et al., 2007 ; Dawkins, 2019 ) . While fully open-ended recursive self-improvement remains a grand challenge, current systems successfully operationalize this concept within defined objectives, benchmarks, and safety protocols.

Within this paradigm, agents’ scaffolding is typically represented as conventional computer programs with mild constraints. Formally, let ⟨ Σ t ⟩ \langle\Sigma_{t}\rangle denote a serializable encoding of the program that implements the agent, and let e ​ x ​ e ​ c ​ ( ⋅ ) exec(\cdot) denote execution in an environment (e.g., an interpreter, compiler, and sandbox). A full-scaffolding update can be viewed as producing a candidate program via executing the current agent-as-improver on its own code: ⟨ Σ ~ t + 1 ⟩ = e ​ x ​ e ​ c ​ ( ⟨ Σ t ⟩ , 𝒮 t ) , \langle\tilde{\Sigma}_{t+1}\rangle=exec(\langle\Sigma_{t}\rangle;\mathcal{S}_{t}), (24) often materialized as a patch Δ t \Delta_{t} applied to the current scaffolding, Σ ~ t + 1 = Σ t ⊕ Δ t \tilde{\Sigma}_{t+1}=\Sigma_{t}\oplus\Delta_{t} . In practice, a verifier (e.g., unit tests, regression suites, safety checks) gates the update: Σ t + 1 = { Σ ~ t + 1 , 𝒱 ⁡ ( Σ ~ t + 1 ) = 1 , Σ t , otherwise . \Sigma_{t+1}=\begin{cases}\tilde{\Sigma}_{t+1},&\mathcal{V}(\tilde{\Sigma}_{t+1})=1,\\ \Sigma_{t},&\text{otherwise}.\end{cases} (25)

The universal representability of computer programs (i.e., Turing completeness) enables self-improving agents to explore an extensive space of self-improving strategies, bounded in principle only by the theoretical limits of computation. Early work on applying a computer program to improve itself dates back to self-referential program-search and meta-evolution frameworks ( Schmidhuber, 1987 ) .

AlphaEvolve ( Novikov et al., 2025 ) is designed as a coding agent for open scientific problems. It adopts an evolutionary paradigm, continuously receiving feedback from one or more evaluators to iteratively improve its algorithms, thereby substantially accelerating new scientific discoveries and optimizing complex technical stacks. Similarly, ShinkaEvolve ( Lange et al., 2026 ) is an LLM-driven program-evolution framework that enables efficient, open-ended program discovery and optimization across multiple tasks with very few evaluation samples, via exploration–exploitation–balanced parent sampling, novelty-based rejection sampling for code, and bandit-based adaptive selection of an integrated ensemble of LLMs.

ADAS ( Hu et al., 2025 ) searches within a design space for an agentic system that maximizes a given evaluation function using a dedicated search algorithm. EvoFlow continuously evolves a set of heterogeneous workflows that achieve favorable cost–performance trade-offs while maintaining structural diversity, yielding a Pareto set in an online manner. Self-Taught Optimizer ( Zelikman et al., 2024 ) abstracts an agent’s scaffolding as an improver: given its own program and a utility function, it repeatedly queries an LM to generate multiple candidate improved versions and selects the highest-scoring one as the output, forming a recursive self-improvement loop. Agent Symbolic Learning ( Ou et al., 2025 ) views a language agent as a “symbolic network” whose weights are instantiated by prompts, tools, and their composition; it simulates backpropagation and gradient descent through natural-language losses/gradients, and uses a symbolic optimizer to jointly update prompts, tools, and pipelines, enabling sustained self-learning and evolution in real deployments.

Another representative line is the open-ended evolutionary framework inspired by the Gödel machine. Yin et al. (2025c) proposes a Gödel-machine-inspired self-referential agent framework, Gödel Agent, implemented via monkey patching. It autonomously performs self-awareness, self-modification, and recursive self-improvement, aiming to search the entire agent design space. Inspired by Darwinian evolution and open-ended research, Zhang et al. (2026c) proposed the self-evolving coding agent Darwin Gödel Machine (DGM), which maintains an archive of newly generated coding agents and grows a continually expanding tree of diverse, high-quality agents through open-ended exploration, allowing parallel exploration of multiple paths in the search space. Subsequently, Wang et al. (2026a) proposed the Huxley-Gödel Machine (HGM), motivated by Huxley’s notion of clades; it introduces a clade-level metaproductivity (CMP) metric to guide evolution as an approximation to the Gödel-machine ideal. Live-SWE-Agent ( Xia et al., 2025 ) is the first real-time software agent that can autonomously and continuously evolve on the fly at runtime; by evolving its own scaffolding, especially its tool components, it achieves desired performance on software issue-solving tasks.

## 7 Applications

The preceding sections organized self-improving agents by what is updated, namely model parameters or scaffolding, and by the signals that drive these updates. This section examines how these mechanisms appear across representative domains. Across applications, a common pattern is the use of sandboxed or otherwise controlled environments that provide feedback while limiting the cost of failure. Software engineering relies on compilers, tests, and continuous integration pipelines; web automation uses simulated or instrumented browsers; games provide resettable environments with explicit rules and outcomes; scientific discovery uses executable workflows, domain tools, and simulators; embodied AI relies on robotic simulators and limited real-world rollouts; and general computer control uses virtualized desktops or isolated operating-system environments. The fidelity, scalability, and cost of these environments shape each domain’s dominant bottlenecks, improvement targets, and iteration modes, as summarized in Table 5 .

### 7.1 Software Engineering

Software engineering (SWE) is a useful testbed for self-improving agents because it provides dense and automatable feedback. Compilers, unit tests, linters, and continuous integration pipelines can turn many agent actions into checkable outcomes, making improvement easier to measure than in domains that rely on subjective evaluation. This property naturally supports both branches of our taxonomy: (i) FM improvement , where executable feedback is used to update model parameters θ \theta , and (ii) scaffolding improvement , where the model is fixed but the scaffold Σ \Sigma is revised. In SWE, the scaffold may include prompts, memory, tools, control logic, and, in some cases, the agent’s own source code.

##### Evaluation substrate vs. self-improvement.

It is important to distinguish evaluation environments from self-improving methods. SWE-bench provides a standardized suite of real GitHub issues, repositories, and test-based validation, enabling quantitative measurement of progress ( Jimenez et al., 2024 ) . Several related benchmarks have since been proposed ( Deng et al., 2025 ; Aleithan et al., 2024 ; Yang et al., 2025a ) . However, systems such as SWE-agent ( Yang et al., 2024 ) and Agentless ( Xia et al., 2024 ) are better viewed as non-self-improving baselines under our definition. They may iterate within a task instance, but they do not, by default, perform persistent cross-interaction updates to θ \theta or Σ \Sigma . Their relevance here is methodological: they show that scaffold design can substantially affect performance, motivating the question of whether such scaffolds can be improved automatically rather than hand-engineered.

##### Scaffolding improvement via self-modifying software agents.

A distinctive feature of SWE is that the agent itself is software, which makes direct scaffold or source-code modification feasible. The Darwin G"odel Machine operationalizes this idea by iteratively rewriting its own codebase and validating candidate modifications on coding benchmarks, producing a tree archive of improved descendants ( Zhang et al., 2026c ) . Building on this self-modification paradigm, the Huxley–G"odel Machine searches the space of self-modifications using an estimate of descendant performance, aiming to optimize long-term improvement potential rather than only immediate benchmark gains ( Wang et al., 2026a ) .

Live-SWE-agent moves scaffold improvement from offline search to runtime self-evolution. Starting from a minimal scaffold, the agent edits and extends its own scaffold while solving real SWE tasks, so the current issue can inform future behavior ( Xia et al., 2025 ) . SE-Agent studies a complementary trajectory-level mechanism, using cross-trajectory revision, recombination, and refinement to escape local optima in reasoning and action sequences ( Guo et al., 2025 ) . Viewed through Σ = ( p , m , T , g ) \Sigma=(p,m,T,g) , these methods update increasingly rich scaffold components, from prompts and tool routines to full system code. They also highlight a SWE-specific lesson: performance depends not only on the base model, but also on the agent’s executable interface to the codebase and its ability to preserve reusable problem-solving strategies across issues.

##### FM improvement from executable feedback.

SWE also provides a direct route to FM improvement because test outcomes can serve as rewards or supervision. SWE-RL scales reinforcement learning to real-world software engineering tasks by using software-evolution data and executable signals to improve the underlying model ( Wei et al., 2025c ) . Related work studies reinforcement learning in long-context and multi-turn SWE settings, with the goal of training models that can sustain extended tool-using interaction while remaining grounded in verification feedback ( Golubev et al., 2025 ) . A practical limitation is that execution-based rewards can be sparse, noisy, or expensive, due to flaky tests, incomplete coverage, or costly environment setup. One complementary direction strengthens the executable signal itself by expanding the test suite: curiosity-driven planning can steer an LLM to generate tests that reach under-explored behaviors ( Amayuelas et al., 2026 ) . Execution-free feedback can therefore complement tests. SWE-RM explores reward-model-based scoring as a substitute for, or supplement to, unit tests, and analyzes which verifier properties transfer to reinforcement learning improvements ( Shum et al., 2025 ) .

##### Synthesis and open challenges.

SWE exposes a useful contrast between the two improvement families. Scaffolding improvement is fast, modular, and model-agnostic, but it may overfit to benchmark tooling, repository conventions, or interaction protocols. FM improvement can internalize skills into θ \theta and may transfer more broadly, but it is more expensive and more exposed to reward hacking and evaluation artifacts. Robust SWE self-improvement therefore requires (i) stronger evaluation against benchmark-specific search and verifier overfitting, such as benchmark mutation and stress testing, as well as (ii) hybrid loops in which scaffold-level discoveries are distilled into the base model only when they transfer beyond a fixed verifier or benchmark protocol ( Garg et al., 2026 ) .

### 7.2 Web Navigation and Automation

Web navigation and automation is a challenging domain for self-improving agents because the environment is dynamic, long-horizon, and often weakly verified. Web interfaces change frequently, the same user intent may be realized through different layouts or DOM structures, and failures often become visible only late in a trajectory. These properties make one-shot prompt engineering brittle and motivate improvement loops that persist across tasks. In our taxonomy, this domain supports both FM improvement, which updates θ \theta from interaction data and feedback, and scaffolding improvement, which keeps θ \theta fixed while updating Σ \Sigma .

##### Evaluation and data substrate.