Self-Improvements in Modern Agentic Systems: A Survey — part 4/14

Cleaned-text code points [45610, 60111). Bibliography omitted here; source.md includes it.
Relation to Active Learning. Active learning studies how a learner selects queries under a labeling budget to maximize information gain, typically by requesting labels from an external oracle ( Ren et al., 2021 ) . SI-FMA intersects with this view when an agent actively controls its data-acquisition process—for example, by targeting frequent failure modes, seeking environments that maximize verifier disagreement, or explicitly requesting human feedback. However, because modern agents often rely on self-generated verification or critiques rather than oracle labels, this behavior transcends standard active learning. It connects SI-FMA directly to classical artificial-curiosity frameworks, in which systems are explicitly designed to actively construct experiments that maximize learning progress, Bayesian surprise, or compression progress ( Schmidhuber, 1991b ; Storck et al., 1995 ; Schmidhuber, 2006b ; Schmidhuber, 2010 ; Herrmann and Schmidhuber, 2026 ) . This perspective places self-improvement within a broader family of curiosity-driven exploration mechanisms, emphasizing that the key design choices are the query policy, the intrinsic objective, and the trustworthiness of the resulting feedback.

## 4 A Taxonomy of Existing Approaches

Building on the definitions introduced in Section 3 , we now introduce a taxonomy to organize existing approaches to self-improvement in FM-based agents. This section serves as a methodological classification that provides a common reference frame for comparing a rapidly growing and heterogeneous literature. In the remainder of this section, we use 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 t  a  r  g  e  t ⁡ ( ⋅ , 𝒮 t ) \operatorname{\mathtt{IMPROVE}}_{target}(\,\cdot\,;\mathcal{S}_{t}) to denote an abstract self-improvement procedure in the sense of SI-FMA. Here 𝒮 t \mathcal{S}_{t} denotes an update signal produced through the agent’s own execution (the operator ℰ \mathcal{E} in Section 3 ), such as interaction trajectories, critiques, preferences, or other self-generated artifacts. This notation organizes existing approaches first by the target of modification, separating foundation-model updates from scaffolding updates. Within each target, approaches are further distinguished by the form of the self-induced learning signal that drives the update. For scaffolding improvement, we additionally group methods by the scaffold component being modified, such as prompts, memory, tools, or full scaffolding.

### 4.1 Foundation Model Improvement

Foundation model improvement targets the parameters of the underlying foundation model while leaving the agent-level scaffold unchanged: θ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 θ ( θ 1 : t ; 𝒮 t ) , Σ t + 1 = Σ t . \theta_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\theta}(\theta_{1:t};\mathcal{S}_{t}),\qquad\Sigma_{t+1}=\Sigma_{t}. (7) In this paradigm, the agent’s own execution under its induced policy π θ t , Σ t \pi_{\theta_{t},\Sigma_{t}} generates learning signals that are subsequently consumed by a parameter-update procedure. By committing the update directly into θ t \theta_{t} , the agent internalizes the learned capabilities. This allows the adaptation cost to be amortized across future interactions, though such parametric updates typically operate on longer time scales and incur higher computational overhead. θ 1 : t \theta_{1:t} denotes the parameter history, enabling validation and rollback (e.g., reverting to a prior checkpoint) when a proposed update degrades performance or violates constraints.

##### Classification by signal form.

As detailed in Section 5 , we further categorize foundation model improvement according to the form of the self-induced signal 𝒮 t \mathcal{S}_{t} : • Intrinsic Generative Demonstrations (primary update signal: 𝒟 t \mathcal{D}_{t} ): the agent synthesizes explicit training instances (e.g., demonstrations or augmented datasets) that are used for supervised-style learning ( Wang et al., 2022 ; Lu et al., 2024b ; Zhao et al., 2024b ; Shi et al., 2025a ) .

• Intrinsic Evaluative Feedback (primary update signal: e t e_{t} ): the agent constructs supervisory signals such as scalar rewards, preference pairs, or critiques to guide optimization and alignment ( Gong et al., 2026 ; Huang et al., 2025c ; Bai et al., 2022 ; Bensal et al., 2025 ; Zhang et al., 2025c ) .

• Extrinsic Exploratory Experience (primary update signal: τ t \tau_{t} ): the agent learns from interaction trajectories and environment-grounded outcomes produced during execution ( Wei et al., 2025a ; Dong et al., 2025 ; Da et al., 2025 ; Sun et al., 2025b ) .

### 4.2 Scaffolding Improvement

Scaffolding improvement targets the agent’s operational scaffold while keeping the foundation model parameters fixed: Σ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 Σ ( Σ 1 : t ; 𝒮 t ) , θ t + 1 = θ t . \Sigma_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\Sigma}(\Sigma_{1:t};\mathcal{S}_{t}),\qquad\theta_{t+1}=\theta_{t}. (8) Here, 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 \operatorname{\mathtt{IMPROVE}} commits structural changes into Σ t = ( p t , m t , 𝒯 t , g t ) \Sigma_{t}=(p_{t},m_{t},\mathcal{T}_{t},g_{t}) , thereby reshaping how the frozen model is conditioned, grounded, and constrained during all subsequent executions. This paradigm typically yields fast, reversible, and task-specific adaptation without the risks of catastrophic forgetting.

##### Classification by scaffold component.

Crucially, each of the following modifications represents a structural update to the agent’s intrinsic configuration, distinguishing it from transient shifts in working memory. As detailed in Section 6 , we further decompose scaffolding improvement according to which component of Σ t \Sigma_{t} is targeted: • Prompt Optimization ( p t → p t + 1 p_{t}\rightarrow p_{t+1} ): edits to structured prompts or in-context exemplars, p t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 p ( p 1 : t ; 𝒮 t ) p_{t+1}=\operatorname{\mathtt{IMPROVE}}_{p}(p_{1:t};\mathcal{S}_{t}) ( Yuksekgonul et al., 2024 ; Guo et al., 2024 ; Fernando et al., 2024 ; Zhou et al., 2022 ) .

• Memory Evolution ( m t → m t + 1 m_{t}\rightarrow m_{t+1} ): updates to how experience is stored, consolidated, and retrieved, m t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 m ( m 1 : t ; 𝒮 t ) m_{t+1}=\operatorname{\mathtt{IMPROVE}}_{m}(m_{1:t};\mathcal{S}_{t}) ( Chhikara et al., 2025 ; Wang et al., 2025i ; Zhang et al., 2025e ; Zhong et al., 2024 ) .

• Tool Governance ( 𝒯 t → 𝒯 t + 1 \mathcal{T}_{t}\rightarrow\mathcal{T}_{t+1} ): refinement or expansion of the agent’s action space through tool creation or selection, 𝒯 t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 𝒯 ( 𝒯 1 : t ; 𝒮 t ) \mathcal{T}_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\mathcal{T}}(\mathcal{T}_{1:t};\mathcal{S}_{t}) ( Haque et al., 2025 ; Wang et al., 2025e ; Dong et al., 2025 ; Zhang et al., 2025j ) .

• Full Scaffolding Update ( Σ t → Σ t + 1 \Sigma_{t}\rightarrow\Sigma_{t+1} ): holistic reconfiguration of the agent’s operational architecture, Σ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 Σ ( Σ 1 : t ; 𝒮 t ) \Sigma_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\Sigma}(\Sigma_{1:t};\mathcal{S}_{t}) ( Hu et al., 2025 ; Zhang et al., 2026c ; Novikov et al., 2025 ; Xia et al., 2025 ) .

## 5 Foundation Model Improvement

Foundation-model-based self-improvement constitutes one of the most direct pathways for enhancing an agent’s intrinsic capabilities. This approach targets the agent’s core, namely the parameter set θ t \theta_{t} of its underlying FM, and updates these parameters to internalize new behaviors and reasoning patterns ( Wang et al., 2022 ; Bai et al., 2022 ; Shafayat et al., 2025 ; Bensal et al., 2025 ; Dong et al., 2025 ; Wang et al., 2025h ; Zhou et al., 2025b ) . Such updates are typically achieved through gradient-based optimization, resulting in stable changes to the model weights. In this sense, the FM is treated as a continually learnable system that stores improvements in its parametric memory, rather than relying solely on transient execution states. Because the model parameters compress the agent’s learned representations and behavioral priors, updating them allows the agent to fundamentally refine its policy, reduce systematic errors, and improve alignment with target objectives. In practice, the agent effectively serves as its own source of supervision: it uses its own execution to generate training signals—such as demonstrations, evaluations, or interaction trajectories—and applies learning algorithms to these self-induced signals to systematically improve its capabilities over successive updates.

To systematically analyze FM-based self-improvement, we classify existing methods by the nature of how the learning signal 𝒮 t \mathcal{S}_{t} is used to update the foundation-model parameters under our formalism. Consistent with the history-aware operator in Section 3 , we write the transition as 𝒜 t + 1 = IMPROVE ( 𝒜 1 : t ; 𝒮 t ) \mathcal{A}_{t+1}=\text{IMPROVE}(\mathcal{A}_{1:t};\mathcal{S}_{t}) with θ t + 1 = IMPROVE θ ( θ 1 : t ; 𝒮 t ) \theta_{t+1}=\text{IMPROVE}_{\theta}(\theta_{1:t};\mathcal{S}_{t}) and Σ t + 1 = Σ t \Sigma_{t+1}=\Sigma_{t} , allowing validation and rollback to prior checkpoints when necessary. This taxonomy yields three principal subcategories: (i) Intrinsic generative demonstrations (§ 5.1 ), where the learning signal is instantiated as 𝒮 t ≈ 𝒟 t \mathcal{S}_{t}\approx\mathcal{D}_{t} : the FM-based agent generates explicit examples, demonstrations, or task–solution pairs that are used for parameter updates. (ii) Intrinsic evaluative feedback (§ 5.2 ), where the learning signal is instantiated as 𝒮 t ≈ e t \mathcal{S}_{t}\approx e_{t} : the FM-based agent or an internal evaluator judges candidate behavior through scores, preferences, confidence estimates, consistency signals, critiques, or revisions, which are then used to update the foundation model. (iii) Extrinsic exploratory experience (§ 5.3 ), where the learning signal is instantiated as 𝒮 t ≈ τ t \mathcal{S}_{t}\approx\tau_{t} : the agent collects trajectories, rewards, observations, or executable outcomes through interaction with external or simulated environments. These subcategories are not mutually exclusive in deployed systems. Many practical pipelines mix intrinsic demonstrations, intrinsic evaluations, and exploratory experience within a single improvement cycle; for clarity, we categorize methods by the dominant signal source and objective used in their update operator. Algorithm 1 summarizes the generic update loop of foundation-model improvement under self-generated learning signals.

### 5.1 Intrinsic Generative Demonstrations

The remarkable capabilities of modern foundation models are fundamentally driven by their massive parameter scales, which inherently demand vast quantities of high-quality training data to continuously optimize and align ( Zhao et al., 2024b ; Gan and Liu, 2025 ) . The methods in this category fall into the group of internal generative processes, where FMs act simultaneously as the cognitive learner and the data synthesizer ( Zhao et al., 2024b ; Zweiger et al., 2025 ) . Leveraging the semantic priors already compressed within their weights, FM-based agents can autonomously construct intrinsic generative demonstrations in forms such as instruction-response pairs, reasoning trajectories, and execution logs, without requiring new external observations ( Zhao et al., 2024b ; Shi et al., 2025a ; Simonds and Yoshiyama, 2025 ) . By casting the agent as its own data provider, this paradigm shifts the bottleneck of parameter updates from the costly acquisition of human annotations to the design of effective generation strategies and internal filtering mechanisms ( Qin et al., 2025 ) .

Formally, as illustrated in Fig. 6 (a), under foundation-model improvement we treat these intrinsic generative demonstrations as the learning signal, i.e., 𝒮 t ≈ 𝒟 t \mathcal{S}_{t}\approx\mathcal{D}_{t} . At iteration t t , the agent state 𝒜 t = ( θ t , Σ t ) \mathcal{A}_{t}=(\theta_{t},\Sigma_{t}) induces a generative distribution 𝒫 gen  ( x , y ∣ 𝒜 t ) \mathcal{P}_{\mathrm{gen}}(x,y\mid\mathcal{A}_{t}) and samples an intrinsic dataset 𝒟 t gen = { ( x i , y i ) } i = 1 n t ∼ 𝒫 gen ( ⋅ ∣ 𝒜 t ) . \mathcal{D}^{\mathrm{gen}}_{t}=\{(x_{i},y_{i})\}_{i=1}^{n_{t}}\sim\mathcal{P}_{\mathrm{gen}}(\cdot\mid\mathcal{A}_{t}). (9) Optionally, a quality-control operator Φ t \Phi_{t} filters or weights examples ( Jiang et al., 2025 ) to yield 𝒟 ~ t gen = Φ t  ( 𝒟 t gen ) \tilde{\mathcal{D}}^{\mathrm{gen}}_{t}=\Phi_{t}(\mathcal{D}^{\mathrm{gen}}_{t}) . The effective training set (and hence the learning signal) is then 𝒟 t = 𝒟 t base ∪ 𝒟 ~ t gen \mathcal{D}_{t}=\mathcal{D}^{\mathrm{base}}_{t}\cup\tilde{\mathcal{D}}^{\mathrm{gen}}_{t} , where 𝒟 t base \mathcal{D}^{\mathrm{base}}_{t} denotes any existing data.

The parameter update can be written compactly as θ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 θ ( θ 1 : t ; 𝒟 t ) \theta_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\theta}(\theta_{1:t};\mathcal{D}_{t}) , where 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 θ \operatorname{\mathtt{IMPROVE}}_{\theta} denotes the outcome of optimizing an empirical objective on 𝒟 t \mathcal{D}_{t} (typically initialized at the current active checkpoint θ t \theta_{t} ). We write this objective as ℒ ⁡ ( θ , 𝒟 t ) \mathcal{L}(\theta;\mathcal{D}_{t}) , with Ω ⁡ ( θ , θ 0 ) \Omega(\theta,\theta_{0}) an optional regularizer around an initialization θ 0 \theta_{0} and λ ≥ 0 \lambda\geq 0 its coefficient: θ t + 1 = arg ⁡ min θ ⁡ ℒ ⁡ ( θ , 𝒟 t ) + λ  Ω  ( θ , θ 0 ) . \theta_{t+1}=\arg\min_{\theta}\;\mathcal{L}(\theta;\mathcal{D}_{t})+\lambda\,\Omega(\theta,\theta_{0}). (10) In practice, 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 θ \operatorname{\mathtt{IMPROVE}}_{\theta} is implemented via iterative fine-tuning with a gradient-based optimizer. Let θ t ( 0 ) = θ t \theta_{t}^{(0)}=\theta_{t} , and let ℬ t ( k ) \mathcal{B}_{t}^{(k)} denote a minibatch drawn from 𝒟 t \mathcal{D}_{t} according to the weighting induced by Φ t \Phi_{t} . Using step size η k \eta_{k} at inner step k k and the gradient operator ∇ θ \nabla_{\theta} with respect to θ \theta , for k = 0 , … , K t − 1 k=0,\dots,K_{t}-1 , θ t ( k + 1 ) = θ t ( k ) − η k  ∇ θ ℒ  ( θ t ( k ) , ℬ t ( k ) ) , \theta_{t}^{(k+1)}=\theta_{t}^{(k)}-\eta_{k}\,\nabla_{\theta}\mathcal{L}\big(\theta_{t}^{(k)};\mathcal{B}_{t}^{(k)}\big), (11) and the outer-loop update is θ t + 1 = θ t ( K t ) \theta_{t+1}=\theta_{t}^{(K_{t})} . To systematically explore this approach, we consider three key aspects: the strategy used to generate these intrinsic demonstrations, the format of the generated data, and the challenges that arise in closing the self-improvement loop.

##### Generation strategies.