Self-Improvements in Modern Agentic Systems: A Survey — part 3/14
Cleaned-text code points [30571, 45610). Bibliography omitted here; source.md includes it.
We conceptualize self-improvement through a self-induced operator 𝒰 \mathcal{U} that updates the agent’s intrinsic configuration. Let π θ t , Σ t \pi_{\theta_{t},\Sigma_{t}} denote the agent’s induced policy. We formalize this process by factorizing the self-improvement step into an execution phase and an update phase: 𝒜 t + 1 = 𝒰 ( 𝒜 1 : t , ℰ ( π θ t , Σ t ; Σ t , 𝒞 t ) ) , \mathcal{A}_{t+1}\;=\;\mathcal{U}\Big(\mathcal{A}_{1:t},\;\mathcal{E}\big(\pi_{\theta_{t},\Sigma_{t}};\Sigma_{t},\mathcal{C}_{t}\big)\Big), (4) where ℰ \mathcal{E} denotes an agent-executed procedure that produces a learning signal (e.g., interaction trajectories, reflections, critiques, or proposed edits) by running the induced policy against a task context 𝒞 t \mathcal{C}_{t} (e.g., a task distribution, a user interaction stream, or a self-play environment). The scaffold Σ t \Sigma_{t} is included as an explicit argument to ℰ \mathcal{E} to permit direct self-inspection (e.g., critiquing prompt templates or auditing tool configurations).
The system-level update rule 𝒰 \mathcal{U} then applies this self-generated signal to modify the agent’s intrinsic components ( θ \theta or Σ \Sigma ). Unlike the routine evolution of the transient state X t X_{t} (e.g., merely accumulating dialogue history or working memory), the operator 𝒰 \mathcal{U} commits durable changes. Crucially, in line with lifelong meta-learning principles, while these updates are not strictly irreversible and may be undone, they allow the agent to consolidate successful strategies into an increasingly stable policy over its interaction stream ( Schmidhuber, 1994 ; Schmidhuber, 1995 ; Wiering and Schmidhuber, 1996 ; Schmidhuber et al., 1996 ) .
##### Two modes of self-reference in SI-FMA.
SI-FMA may be self-referential in multiple, qualitatively distinct senses. In the first mode, the agent’s policy is executed to indirectly induce improvement by generating experience or auxiliary artifacts that serve as learning signals. Self-generated trajectories, evaluations, preferences, or synthetic labels give rise to a learning objective that is subsequently consumed by an update rule, such as an external optimization procedure acting on the foundation model parameters θ t \theta_{t} . This mode realizes self-improvement at the distributional level: the agent’s behavior shapes the data and supervision that determine its future parameters.
In the second mode, the agent’s policy is executed to directly implement improvements by modifying components of its own operational definition. Through its actions, the agent may edit prompt templates, manage or reorganize memory, reconfigure tool interfaces, or alter control logic, thereby explicitly updating the scaffolding Σ t \Sigma_{t} that governs future execution. In this case, self-improvement is realized through direct, action-level self-modification.
These two modes correspond to distinct but complementary paradigms. Foundation model improvement realizes self-improvement through self-induced learning in parameter space, whereas scaffolding improvement realizes self-improvement through direct self-modification in execution space. Together, they capture the principal mechanisms by which FM-based agents may alter themselves over time.
##### Foundation model improvement.
In foundation model improvement, self-improvement targets the parameters of the underlying foundation model while holding the agent-level scaffolding fixed. Starting from an induced policy π θ t , Σ t \pi_{\theta_{t},\Sigma_{t}} , the agent is repeatedly executed in the environment, generating self-induced experience through its own interaction trajectories.
We abstract this process by a self-induced update operator that maps the agent’s current configuration to an updated set of model parameters: θ t + 1 = 𝒰 θ ( θ 1 : t , ℰ ( π θ t , Σ t ; Σ t , 𝒞 t ) ) , Σ t + 1 = Σ t , \theta_{t+1}=\mathcal{U}_{\theta}\bigl(\theta_{1:t},\;\mathcal{E}(\pi_{\theta_{t},\Sigma_{t}};\Sigma_{t},\mathcal{C}_{t})\bigr),\quad\Sigma_{t+1}=\Sigma_{t}, (5) where 𝒞 t \mathcal{C}_{t} is the task or deployment context defined above, ℰ \mathcal{E} denotes the execution of the agent under its induced policy in 𝒞 t \mathcal{C}_{t} , producing learning signals such as rewards, preference comparisons, synthetic labels, critiques, or verification outcomes, and 𝒰 θ \mathcal{U}_{\theta} denotes a parameter-learning procedure that updates the foundation model parameters based on these signals.
Importantly, while the construction of these learning signals may involve rule-based evaluators, learned reward models, or auxiliary model invocations, the underlying data distribution is induced by the agent’s own policy. The operator 𝒰 θ \mathcal{U}_{\theta} may instantiate policy-gradient methods, offline or online reinforcement learning, preference optimization, or other parameter-learning algorithms. Foundation model improvement typically operates on longer time scales, incurs substantial computational cost, and leads to stable, global changes in the agent’s internal representations, generalization behavior, and capabilities.
##### Scaffolding improvement.
In scaffolding improvement , self-improvement targets the agent-level structures surrounding the foundation model while keeping the model parameters θ \theta fixed. The scaffolding Σ t \Sigma_{t} governs how histories are converted into the conditioning context for the foundation model, how tool calls are specified and executed, how memory is stored and retrieved, and more generally how token sequences are constrained such that they correspond to valid, executable actions. Through these mechanisms, Σ t \Sigma_{t} shapes the agent’s effective observation and action semantics, as well as the construction and evolution of the agent state X t X_{t} during execution.
Formally, scaffolding improvement can be expressed as a self-induced update in which an agent-executed procedure produces a meta-level signal that is then applied as an intrinsic update: Σ t + 1 = 𝒰 Σ ( Σ 1 : t , ℰ ( π θ t , Σ t ; Σ t , 𝒞 t ) ) , θ t + 1 = θ t , \Sigma_{t+1}\;=\;\mathcal{U}_{\Sigma}\Big(\Sigma_{1:t},\;\mathcal{E}\big(\pi_{\theta_{t},\Sigma_{t}};\Sigma_{t},\mathcal{C}_{t}\big)\Big),\qquad\theta_{t+1}=\theta_{t}, (6) where ℰ \mathcal{E} denotes the execution of the agent (possibly under a meta-objective) to generate update-relevant artifacts, such as proposed prompt edits, memory reorganizations, tool-interface changes, or new control routines, and 𝒰 Σ \mathcal{U}_{\Sigma} denotes the system-level mechanism that commits these artifacts as structural scaffolding updates. Unlike the transient evolution of X t X_{t} , the update above is intended to endure beyond individual task boundaries.
These updates modify the induced policy π θ , Σ \pi_{\theta,\Sigma} by reshaping (i) the conditioning context supplied to the foundation model, (ii) the effective action space via admissibility constraints and tool schemas, and (iii) the execution semantics by which token sequences are parsed and grounded into environment actions. Compared to foundation model improvement, scaffolding improvement is typically faster, more reversible, and more context-dependent, yet it can produce substantial changes in the agent’s effective behavior and problem-solving strategies.
##### Skill as a reusable update.
The two modes of self-reference above realize self-improvement either indirectly, through self-generated signals that an update rule consolidates into the parameters θ \theta , or directly, through actions that edit the scaffold Σ \Sigma ; a skill cuts across this distinction. We model a skill as a reusable instance of the self-induced update operator 𝒰 \mathcal{U} : a named update to the agent’s own configuration 𝒜 t \mathcal{A}_{t} that it retains and reuses. Acquiring a skill serializes this update into one of 𝒜 t \mathcal{A}_{t} ’s substrates: a tool and its calling convention ( 𝒯 \mathcal{T} ), an instruction or workflow ( p p ), a memory entry ( m m ), consolidated weights ( θ \theta ), or control logic ( g g ). The skill’s identity is the update it encodes; the substrate only names where it is stored. This is what makes “skill” orthogonal to the substrate axis of our taxonomy. A skill library is then a structured store of, and retrieval policy over, these serialized updates; the same store-and-retrieve structure recurs across substrates, which is why it surfaces in both tool routing and memory organization.
Reusability takes two forms. A skill may be invoked repeatedly, as a retained routine called many times across tasks, or applied once in the manner of an installer: a single update whose value lies in the persistent change it leaves behind, but which remains a portable artifact, reusable across agents and sessions. Either way, a skill is a first-class, serialized operator, in contrast to an ad hoc one-off action that leaves no retained trace.
We further distinguish two scopes according to what an invoked skill acts on. An object-level skill acts on the task or world state: invoking it runs a (typically multi-step) routine through the execution operator ℰ \mathcal{E} to carry out a sub-task (e.g., a learned collect-wood routine), the agentic analog of a temporally extended option in hierarchical reinforcement learning ( Sutton et al., 1999 ; Bacon et al., 2017 ) . Here the 𝒰 \mathcal{U} step lies in acquisition, which writes the routine into its substrate (e.g., 𝒯 \mathcal{T} ), not in invocation. A meta-level skill instead acts on the agent’s own configuration 𝒜 t \mathcal{A}_{t} : invoking it edits a component of Σ t \Sigma_{t} or triggers a parameter update (e.g., writing a new tool, refactoring a prompt, consolidating experience into memory, or patching one’s own scaffold). The installer-like skills above are inherently meta-level. For self-improvement, the meta-level scope is the central one. Because a meta-level skill both acts on 𝒜 t \mathcal{A}_{t} and is itself serialized back into 𝒜 t \mathcal{A}_{t} , the operator can become part of its own operand. Once such a skill is in turn improved, we recover the self-referential loop in which the improver evolves together with the system it improves ( Schmidhuber, 1987 ; Schmidhuber, 1993 ; Schmidhuber, 2003b ) .
Later sections describe how skills are serialized in recent works: as tools (Section 6.3 ), as memory workflows (Section 6.2 ); the application sections illustrate how acquired skills are invoked and reused across tasks.
### 3.3 Connections to Related Learning Paradigms
Because the SI-FMA paradigm is deeply rooted in established learning paradigms, examining it through the lens of these foundational theories offers critical insights. We map SI-FMA onto classical frameworks to clarify where self-improving FM-based agents inherit existing assumptions, where they recombine familiar mechanisms, and where agent architecture makes these mechanisms operationally distinct in practice.
Relation to Reinforcement Learning (RL). Classical reinforcement learning provides a natural reference frame for SI-FMA: it formalizes how an agent improves its policy through interaction, and recent work on Agentic RL ( Zhang et al., 2026a ) extends this view to foundation-model-based agents. Mapping SI-FMA onto this frame clarifies which channels of self-improvement inherit standard RL machinery and which lie outside its formulation. Specifically, updates to θ \theta correspond to standard policy optimization under a fixed decision process, whereas updates to Σ \Sigma lie outside the standard RL formulation, reshaping the decision process in which the policy operates.
• Parameter updates ( θ \theta ) . When improvement targets the foundation model parameters using interaction trajectories, the process aligns with standard policy optimization. The foundation model acts as a large-scale policy network π θ \pi_{\theta} , and techniques such as RLHF ( Christiano et al., 2017 ; Ouyang et al., 2022 ) or self-play optimize π θ \pi_{\theta} via algorithms like Proximal Policy Optimization (PPO) ( Schulman et al., 2017 ) or Direct Preference Optimization (DPO) ( Rafailov et al., 2023 ) .
• Scaffolding updates ( Σ \Sigma ) . When improvement targets the scaffolding, the agent performs a form of structural meta-learning that has no direct counterpart in classical RL. First, whereas classical RL assumes a fixed action space and state representation, updating Σ \Sigma (e.g., adding tools or modifying memory) dynamically alters the effective state-action space and the observation processing logic, thereby reshaping the underlying Markov decision process (MDP) itself. Second, the optimization target itself differs in kind: Σ \Sigma comprises discrete, structured artifacts such as prompts, memory entries, and routing rules, which are typically updated through search, generation, or symbolic edits rather than gradient descent, and which remain explicit and inspectable rather than absorbed into network weights.
SI-FMA also departs from classical RL along a second axis: the source of the learning signal. While classical RL relies on external scalar rewards, self-improving agents increasingly utilize self-generated supervision, where the agent acts as its own critic to synthesize feedback signals from its own trajectories, judgments, or verifier modules.
Relation to Online Learning. Online learning ( Littlestone, 1988 ; Hoi et al., 2021 ) studies sequential decision or prediction under a data stream ( Qi et al., 2025 ) , where the learner updates each round and performance is assessed through cumulative loss or regret. SI-FMA intersects with this view when an agent is updated repeatedly during deployment and evaluation tracks an improvement trajectory rather than a single endpoint, though it is not restricted to the online setting—many self-improvement pipelines operate in batched or offline regimes. Where the paradigms meet, the two channels relate to online learning asymmetrically: θ \theta updates recover the classical view of updating a hypothesis under a non-stationary stream, whereas Σ \Sigma updates enact a system-level analogue that shifts adaptation to explicit, inspectable components.
• Parameter updates ( θ \theta ). These inherit the standard stability challenges of online learning, including distribution shift and catastrophic forgetting. Systems mitigate these risks through controlled update operators such as replay or rehearsal ( Robins, 1995 ) , regularization ( Kirkpatrick et al., 2017 ; Ramesh et al., 2026 ) , parameter-efficient tuning ( Hu et al., 2022 ) , and explicit versioning with rollback.
• Scaffolding updates ( Σ \Sigma ). These provide a channel for rapid adaptation through prompting policies, memory read-write rules, tool routing, and orchestration logic. Externalizing adaptation in this way improves transparency and control, but does not eliminate forgetting and introduces its own failure modes such as memory poisoning, drift in tool semantics, and brittle template dependence.