Self-Improvements in Modern Agentic Systems: A Survey — part 2/14

Cleaned-text code points [15376, 30571). Bibliography omitted here; source.md includes it.

Lenat’s Automated Mathematician (AM) system exemplified this approach by using heuristics to propose, extend, and evaluate candidate concepts, effectively searching over symbolic programs that encode mathematical ideas ( Lenat1, 1984 ) . EURISKO pushed further by representing heuristics themselves as manipulable Lisp objects, allowing the system to generate, modify, and test its own problem-solving strategies ( Lenat, 1983 ; Lenat and Brown, 1984 ) . However, as emphasized by Schmidhuber (1987) , the practical success of systems like EURISKO depended heavily on the user serving as an external evaluation signal, interpreting outputs and manually pruning unproductive heuristic drift, rather than on the system possessing a truly autonomous closed-loop credit assignment mechanism. This historical limitation underscores a challenge that persists in modern agentic systems, namely the need for robust internal evaluation signals and verification procedures to sustain autonomous recursive self-improvement.

### 2.3 Connectionism and the Emergence of Meta-Learning (1980s–2000s)

The resurgence of connectionism shifted the emphasis from hand-coded heuristics to learning dynamics as the substrate of improvement. Self-improvement was consequently reframed as explicitly optimizing the learning process itself, encompassing the optimizer, inductive biases, and adaptation rules. Schmidhuber (1987) introduced self-referential learning frameworks in which evolutionary processes could optimize not only candidate solutions but also the learning procedures that generate them, making the learning system itself the object of search. Along a complementary line, Bengio et al. (1991) demonstrated that synaptic learning rules need not be fixed; by parameterizing these rules, the optimizer itself becomes a learnable object.

Schmidhuber introduced fast-weight programmers (FWPs) as a broad class of networks ( Schmidhuber, 1991a ; Schmidhuber, 1992 ; Schmidhuber, 1993 ) . A representative early instance—retrospectively identified as the 1991 unnormalized linear Transformer (ULTRA) ( Schlag et al., 2021 ) —features a “slow” neural network that learns by gradient descent to program the “fast weights” of another network through additive outer-product updates. The explicitly self-referential version followed in work on self-referential weight matrices, where a network learns to modify its own fast weights while receiving feedback signals such as errors or rewards as inputs, making the learning dynamics themselves part of what is optimized ( Schmidhuber, 1993 ) . This line is important for self-improvement because it moves self-modification into a continuous program space, rather than restricting it to symbolic code rewriting. Recent work has revisited this direction in scalable self-referential neural architectures, including modern self-referential weight matrices that learn to modify themselves ( Irie et al., 2022 ) , recurrent fast-weight programmers and their self-referential extensions ( Irie et al., 2023 ; Irie et al., 2021 ) , and self-referential networks that meta-learn continual learning algorithms ( Irie et al., 2025 ) . Such mechanisms suggest a possible additional meta-level for self-improving foundation models, where the foundation model itself may become part of the modifiable substrate rather than remaining only a frozen component.

This period also consolidated learning to learn as a framing for self-improvement. In reinforcement learning (RL), the 1994 paper introduced an RL machine as an early recursive self-improving agent that alter parts of their own learning strategy to shift inductive bias over a single lifelong interaction stream ( Schmidhuber, 1994 ) . The same underlying idea was later described using related terminology, including environment-independent reinforcement acceleration and the success-story algorithm ( Wiering and Schmidhuber, 1996 ; Schmidhuber et al., 1996 ; Schmidhuber and Zhao, 1996 ; Schmidhuber et al., 1997 ) . Thrun and Pratt likewise emphasized learning reusable inductive biases across task distributions ( Thrun and Pratt, 1998 ) . Related neuroevolution work such as NeuroEvolution of Augmenting Topologies (NEAT) improved neural architectures by expanding network topology under performance-driven selection ( Stanley and Miikkulainen, 2002 ) . AIXI formalized an uncomputable upper bound on optimal sequential decision-making in computable environments ( Hutter, 2000 ) . Furthermore, Schmidhuber (2003b) introduced the Gödel Machine, a fully self-referential, self-improving machine that is theoretically optimal. A Gödel Machine operates in a single-life reinforcement learning environment ( Schmidhuber, 1994 ) and iteratively searches for candidate self-modifications, adopting a modification only when it can prove that doing so will improve its expected cumulative future reward. To establish this improvement, the proof must reason about future rewards while accounting for all possible subsequent self-modifications. The machine is fully self-referential in the sense that every component of the Gödel Machine is itself subject to modification, including the self-modification generator and the theorem prover.

### 2.4 Formal and Architecture-Level Self-Improvement (2000s-2020s)

From the mid-2000s onward, research followed two main tracks: (i) analyzing the theoretical limits of self-improvement, and (ii) building practical mechanisms to automate architecture design. Philosophical analyses began to explore the trajectory of these systems, debating the existential risks of the technological singularity ( Subramanian, 2010 ) . In theory, Orseau and Ring pointed out that when agents interact with limited resources and irreversible actions, they face significant risks such as self-deception, reward tampering, and fatal errors ( Orseau and Ring, 2011 ) . Related research formalized how agents can safely model their environments and themselves without generating logical paradoxes. For instance, Fallenstein et al. ( Fallenstein et al., 2015 ) introduced reflective oracles to reason about probabilistic programs that call themselves, while logical induction offered a mathematical approach to updating beliefs under self-referential uncertainty ( Garrabrant et al., 2020 ) . Complementary work on proof-producing reflection in higher-order logic studied how reflective reasoning can be embedded in formal proof systems, providing a technical route for agents or theorem provers to reason about their own reasoning steps ( Fallenstein and Kumar, 2015 ) .

Together, these frameworks clarified the gap between ideal models and practical deployments, demonstrating the necessity of safe and constrained self-modification. Early attempts to bridge this gap computationally, such as the endeavor to implement Gödel machines, sought architectures where agents could mathematically prove the optimality of their own code updates ( Steunebrink and Schmidhuber, 2012 ) . At the architecture level, Nivel et al. proposed bounded recursive self-improvement, where reflective mechanisms and value-driven scheduling support self-modification under designer-imposed constraints ( Nivel et al., 2013 ) . In engineering, automated design has matured into Neural Architecture Search (NAS), where controller learning proposes architectures that can directly optimize task performance ( Zoph and Le, 2017 ) . Meanwhile, scalable algorithms such as self-play in reinforcement learning demonstrated how agents can generate increasingly difficult curricula from their own behavior, thus achieving continuous capability growth without human supervision ( Silver et al., 2017b ) . Ultimately, these theoretical and engineering advances have laid the foundation for modern agent systems centered on foundation models.

### 2.5 Scalable Foundation Models and Agentic Systems (2020s–Present)

The classical vision of autonomous agents—capable of perceiving, acting, and learning in open-ended environments—found a scalable, modern substrate in foundation models (FMs). Powered by the immense knowledge compressed during large-scale pretraining and the flexibility of a unified natural-language interface, FMs function as the cognitive engines of contemporary agentic systems. Crucially, the emergence of these systems fundamentally reshaped the paradigm of self-improvement, decoupling it into two distinct but complementary mechanisms: rapid, training-free adaptation and persistent parameter optimization.

Within this modern paradigm, the most immediate form of self-improvement occurs without computationally expensive weight updates. Because scalable sequence models facilitate broad generalization, agents can achieve rapid, training-free adaptation by dynamically revising their plans, prompts, and memory representations on the fly. This short-horizon self-improvement relies heavily on in-context learning, which effectively functions as a form of meta-learning. Mechanistically, this adaptation is mediated by the attention mechanism’s key-value cache, acting as an associative memory that generates transient, context-dependent weight changes during sequence processing. This functional equivalence directly connects modern in-context learning to the 1991 fast-weight programmers (formally identified as unnormalized linear Transformers), in which networks similarly learned to induce rapid parameter changes during inference without requiring standard gradient descent ( Schmidhuber, 1991a ; Schmidhuber, 1992 ; Schmidhuber, 1993 ; Schlag et al., 2021 ) .

Complementing this fast, in-context adaptation are slow improvement loops designed to permanently internalize new capabilities. Reinforcement learning from human feedback (RLHF) and its variants achieved this by converting interaction traces and preference signals into persistent parameter updates ( Ouyang et al., 2022 ) . Parallel methodologies advanced partially self-supervised alignment by enabling models to critique their own outputs and provide AI feedback ( Bai et al., 2022 ) . Meanwhile, frameworks like Reasoning and Acting (ReAct) and Reflexion build these loops on top of contextual execution, converting execution errors into natural-language reflections ( Yao et al., 2022 ; Shinn et al., 2023 ) . Recent systems pushed this further by maintaining skill libraries, curricula, and evaluation harnesses, spanning open-ended embodied learning ( Wang et al., 2023 ) , software-engineering agents with instrumented interfaces ( Yang et al., 2024 ; Zhang et al., 2026c ; Wang et al., 2026a ) , and explicitly self-improving computer-use agents that iterate data, reward models, and policies over generations ( Xiao et al., 2025 ; Sun et al., 2025b ) . Overall, the modern era reframes self-improvement as nested loops across parameters and scaffolding, renewing classical questions about control, evaluation, and safety at scale.

## 3 Definitions

### 3.1 Formulation of Agentic Systems

We begin by clarifying the formal definition of an agent. The core of an autonomous agent lies in perceiving its environment, updating its internal state, and performing actions to achieve a specific goal. While such systems have driven decades of AI research, this survey focuses exclusively on a contemporary class: foundation-model-based agents , which utilize large foundation models as their central cognitive engines. Crucially, they leverage natural language as a unified interface to integrate perception, reasoning, and tool manipulation. Because a foundation model fundamentally operates as a stateless inference engine, achieving autonomy requires coupling this cognitive core with a persistent, interactive scaffold. Therefore, we formally define the configuration of a foundation-model-based agent at time step t t as: 𝒜 t = ( θ t , Σ t ) , \mathcal{A}_{t}\;=\;(\theta_{t},\;\Sigma_{t}), (1) where θ t \theta_{t} encapsulates the neural parameters of the foundation model, and Σ t \Sigma_{t} denotes the agent’s dynamic operational scaffold. The scaffold Σ t \Sigma_{t} specifies how the foundation model is conditioned, grounded, and connected to the external world. Concretely, it can be decomposed as Σ t := ( p t , m t , 𝒯 t , g t ) , \Sigma_{t}:=(p_{t},\,m_{t},\,\mathcal{T}_{t},\,g_{t}), (2) where p t p_{t} denotes structured prompts or system instructions, m t m_{t} denotes memory mechanisms and their retrieval and update policies, 𝒯 t \mathcal{T}_{t} denotes the set of external tools together with their invocation interfaces, and g t g_{t} denotes additional control logic such as routing, scheduling, or safety constraints. The interaction between these components and the model’s internal parameters is illustrated in Figure 5 , which provides a schematic overview of an FM-based agent within our proposed formalism. Together, θ t \theta_{t} and Σ t \Sigma_{t} determine how the agent reasons, plans, and acts.

During execution, to bridge this intrinsic configuration with the external environment, the agent maintains an ephemeral execution state X t X_{t} (e.g., key-value caches, intermediate plans, or short-term working memory) that evolves as it processes an interaction stream. To connect this structural definition to observable behavior, it is beneficial to clarify how the agent’s configuration generates its action selection mechanism. Although the foundation model parameters θ t \theta_{t} implement a general generative distribution, the agent’s realized behavior is jointly determined by both θ t \theta_{t} and its operational scaffold Σ t \Sigma_{t} . Accordingly, we denote the induced policy of the agent based on the foundation model as: π θ t , Σ t ​ ( A t ∣ X t ) , \pi_{\theta_{t},\Sigma_{t}}(A_{t}\mid X_{t}), (3) where A t A_{t} denotes the action produced by the system at time step t t . Here, the conditioning on X t X_{t} captures the transient context of the ongoing interaction. While X t X_{t} may strongly influence immediate behavior, it is inherently ephemeral. It is typically discarded or reset once an immediate goal is reached or an external task boundary is crossed, and therefore does not constitute part of the agent’s intrinsic architecture. In contrast, the parameters ( θ t , Σ t ) (\theta_{t},\Sigma_{t}) represent the agent’s intrinsic configuration over time. While a standard agent may adapt to novel situations by dynamically updating its transient state X t X_{t} (e.g., in-context examples), its underlying capabilities remain bounded by its fixed initial setup.

### 3.2 Formal Definition of Self-Improvement

Building upon the formal configuration 𝒜 t = ( θ t , Σ t ) \mathcal{A}_{t}=(\theta_{t},\Sigma_{t}) , we formalize self-improvement in foundation-model-based agents (SI-FMA) as a process of persistent, endogenous adaptation. Rooted in the foundational principles of explicitly self-referential meta-learning ( Schmidhuber, 1987 ; Schmidhuber, 1993 ; Schmidhuber et al., 1997 ) , a self-improving agent actively leverages signals induced by its own execution—such as interaction outcomes, critiques, verification results, or proposed edits—to durably modify its underlying computational components.

##### Self-improvement as a self-induced operator.