Self-Improvements in Modern Agentic Systems: A Survey — part 9/14

Cleaned-text code points [122524, 137592). Bibliography omitted here; source.md includes it.

Given the memory state m t := ( object t , structure t ) m_{t}:=(\text{object}_{t},\text{structure}_{t}) , memory processing refers to the family of signal-driven operations that act on m t m_{t} throughout an agent’s lifetime. Formally, we view memory improvement as m t + 1 = IMPROVE m ​ ( m t , 𝒮 t ) m_{t+1}=\text{IMPROVE}_{m}(m_{t};\mathcal{S}_{t}) . Unlike traditional agents that rely on static, hard-coded indexing, self-improving agents use the learning signal 𝒮 t \mathcal{S}_{t} (e.g., task outcomes, utility, or internal critiques) to dynamically adjust what to write, how to retrieve, and when to forget ( Xu et al., 2025c ; Liang et al., 2026 ) . Concretely, we instantiate IMPROVE m \text{IMPROVE}_{m} as an adaptive CRUD (Create, Read, Update, Delete) operation family:

Rather than appending raw logs exhaustively, memory creation is a selective distillation process guided by 𝒮 t \mathcal{S}_{t} . Recent systems typically implement this through three recurring patterns: (1) Semantic compression , which transforms raw interactions into structured metadata, summaries, or reusable schemas for efficient indexing ( Salama et al., 2025 ; Wang et al., 2025i ; Ouyang et al., 2025b ) ; (2) Context-aware discrete decisions (e.g., add, update, delete, or no-op), conditioned on retrieved neighbors to prevent redundant or conflicting entries ( Chhikara et al., 2025 ; Xu et al., 2025f ) ; and (3) Controlled boundary insertion , which dynamically optimizes the write policy for downstream utility closer to generation time ( Zhang et al., 2025e ) . Across all designs, creation is bounded by a trade-off: over-writing inflates retrieval noise, whereas under-writing sacrifices long-term capability.

Memory reading dictates the quality of an agent’s downstream reasoning. To maintain precision as the memory bank scales, self-improving systems typically employ four key mechanisms: (1) Hybrid heuristics , which blend semantic relevance with recency and importance scores to stabilize recall ( Park et al., 2023 ) ; (2) Structure-aware retrieval , which performs coarse-to-fine traversal over hierarchical or graph topologies to isolate reusable abstractions from low-level traces ( Zhang et al., 2025d ) ; (3) Retrieval gating , which dynamically decides whether to query memory and how much context is sufficient, thereby optimizing token cost and minimizing distraction ( Wang et al., 2025a ) ; and (4) Retrieval-driven adaptation , where historical trajectories are fetched as actionable cases to guide behavior without requiring parametric model updates ( Zhou et al., 2025a ) . Ultimately, ineffective reading policies will hinder performance: retrieving irrelevant noise or missing critical details will directly cause subsequent planning and execution to fail.

Memory updating is the primary mechanism for self-correction and knowledge evolution. Guided by feedback ( 𝒮 t \mathcal{S}_{t} ), agents typically implement updates through four operational patterns: (1) Scheduled review and attenuation , which periodically strengthens high-utility items and decays obsolete ones to stabilize memory growth ( Liang et al., 2026 ) ; (2) Local refresh , which dynamically updates topological neighbors during new insertions to maintain contextual consistency ( Xu et al., 2025f ) ; (3) Iterative distillation , which synthesizes repeated successes into compact, reusable abstractions via continuous selection and replacement ( Suzgun et al., 2025 ; Zhang et al., 2025i ) ; and (4) Offline aggregation , which shifts computationally expensive compression away from the online execution loop while preserving recall quality ( Fang et al., 2026 ) . Without effective update policies, agents suffer from memory decay: outdated facts remain, knowledge structures break down, and improper merging erases important details.

Memory deletion acts as a systematic noise reduction and resource optimization mechanism. Guided by signal ( 𝒮 t \mathcal{S}_{t} ), agents dynamically identify and remove obsolete or redundant entries through three main patterns: (1) Multi-stage pruning , which filters low-value items at write time and periodically clears entries based on access frequency and relevance ( Zhang et al., 2025g ) ; (2) Consensus-based eviction , which uses collaborative voting in distributed setups to prevent the accidental deletion of critical shared knowledge ( Bach, 2025 ) ; and (3) Tiered eviction , which applies operating-system-inspired rules across different memory layers to bound size while preserving long-term consistency ( Kang et al., 2025 ) . Improper deletion policies create a direct dilemma: over-pruning loses critical knowledge, while under-pruning floods the system with outdated noise and slows down retrieval.

##### Putting it together: a signal-driven memory loop.

The dimensions of memory objects, structure, and processing culminate in a unified, signal-driven lifecycle (Figure 8 ). This loop proceeds in a continuous cycle: (i) Observe & Detect saliency from new interactions; (ii) Create compact objects; (iii) Organize them into the chosen topological structure; (iv) Read on demand via adaptive retrieval; (v) Plan & Act based on fetched context; (vi) Evaluate outcomes to derive the learning signal 𝒮 t \mathcal{S}_{t} ; and (vii) Update/Delete entries to consolidate high-value knowledge and prune noise. Together, these stages elevate memory from a passive cache to a self-governing engine that sustains open-ended autonomy.

### 6.3 Tool

Basic agents have been constrained by their reliance on static, manually curated toolkits, lacking the autonomy to adapt, discover, or integrate new resources when confronted with novel challenges ( Shapiro et al., 2023 ; Zhang et al., 2025a ; Sang et al., 2025 ; Huang et al., 2025b ) . The transition from a basic executor to a genuinely self-improving agent necessitates a shift away from simple, pre-defined tool use toward Tool Governance Metacognition . This dynamic paradigm empowers the agent to autonomously reason about the necessity, utility, and reliability of its tools, thereby continually advancing its capability boundaries rather than merely operating within them ( Liu and van der Schaar, 2025 ; Yin et al., 2025b ) .

Formally, tool-based improvement can be written as 𝒯 t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 𝒯 ⁡ ( 𝒯 t ; 𝒮 t ) , \mathcal{T}_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\mathcal{T}}(\mathcal{T}_{t};\mathcal{S}_{t}), (21) where 𝒮 t \mathcal{S}_{t} is the learning signal produced by tool governance metacognition. This metacognition represents the architectural shift essential for a self-improving agent, establishing a dynamic and self-directed mechanism for managing its operational tools ( Wang et al., 2025b ) . It is instantiated through three core dimensions: Dynamic Tool Routing, which orchestrates the efficient selection and combination of tools from a growing pool; Iterative Tool Refinement, which adapts and debugs existing instruments to overcome execution failures or environmental shifts; and Autonomous Tool Creation, which invents novel tools to fundamentally expand the agent’s capability boundaries. This governance structure moves tool use from a static look-up process to a generative, self-improving cycle.

#### 6.3.1 Dynamic Tool Routing

Dynamic tool routing concerns how an agent selects, sequences, and coordinates tools in a heterogeneous environment. As tool pools grow, routing becomes a primary bottleneck for self-improvement. A large tool set increases coverage but also increases error modes, including misrouting, compounding execution failures, and wasted compute. Routing methods therefore trade off coverage, reliability, and decision cost. Most existing systems can be understood through three routing paradigms, distinguished by what they treat as the “retrieval unit” and how they incorporate feedback over time.

##### Retrieval- and graph-based routing.

Systems in this paradigm tackle the routing bottleneck by optimizing either the retrieval space or the representation of dependencies. To optimize the retrieval space and maintain scalability, systems dynamically manage the granularity and scope of the tool pool. For instance, while MemTool prevents routing degradation by actively pruning the tool set into a lightweight operational memory ( Lumer et al., 2025a ) , TAR expands the retrieval unit itself, dynamically routing between atomic APIs and entire competent agents to handle higher-level tasks ( Lumer et al., 2025b ) . Crucially for self-improvement, this retrieval process is not static; it evolves by leveraging past successes as a learning signal. Systems like VOYAGER and MetaAgent index reusable tool-use trajectories into procedural memory, allowing the routing policy to continuously refine itself based on structural knowledge from prior executions ( Wang et al., 2023 ; Qian and Liu, 2025 ) . Conversely, to address the compositionality limitation of flat retrieval, recent works encode tool transitions as topological structures. Rather than one-shot relevance, systems like ToolNet and OrchDAG model tool dependencies as directed graphs, enabling routing that accounts for multi-step feasibility and preconditions ( Liu et al., 2024c ; Lu et al., 2025 ) . Building on this, MassTool combines learned semantic matching with these graph structures, achieving high-precision navigation even within massive and complex tool topologies ( Lin et al., 2025 ) .

##### Policy-learning routing.

Policy-learning routing internalizes tool choice as sequential decision making. Instead of retrieving a tool from an external index, the agent learns invocation behavior from training signals, which can improve robustness in multi-turn settings where success depends on state, history, and error recovery. The main trade-off is data and feedback. Learned routing can overfit to training environments and can be brittle when tool interfaces or distributions drift. To establish a reliable routing baseline, systems like AUTOACT, MCP-Flow, Tool-Star, and DeepEyesV2 bootstrap policies through supervised fine-tuning on synthetic or mined trajectories ( Qiao et al., 2024 ; Wang et al., 2025f ; Dong et al., 2025 ; Hong et al., 2026 ) . However, to continually improve long-horizon behavior, routing policies must transcend static supervision. Works like AGENTFLOW and SPORT achieve this by applying sparse rewards or preference signals to shape planning and exploration, while AutoTIR and DeepAgent explicitly align tool selection with multi-objective compliance and action-level attribution ( Li et al., 2026 ; Li et al., 2025a ; Wei et al., 2025b ; Li et al., 2025b ) . Pushing this paradigm to its limit, ToolGen collapses retrieval, selection, and invocation into a single generative process via tool-token unification. While this elegantly reduces pipeline complexity, it places a heavy burden on the learned policy to remain calibrated under distribution shifts ( Wang et al., 2025e ) .

##### Proactive and interactive routing.

A third paradigm treats routing as an interactive process rather than a one-shot decision. This perspective is motivated by two recurring failure sources in self-improving agents. First, user intent is often underspecified, and misrouting can be avoided by clarification. Second, execution failures are common, and routing must support local repair without restarting the entire plan. Proactive routing therefore trades additional interaction and compute for higher reliability. Rather than failing passively, these systems embed metacognitive interventions to handle uncertainty. When confronting underspecified intents or functional gaps, agents like MCP-Zero and ASKTOACT proactively initiate tool discovery or prompt for clarification, transforming ambiguity into a learning signal for self-correction ( Fei et al., 2025 ; Zhang et al., 2025l ) . Conversely, when facing execution failures, dynamic repair becomes essential. Tool-Planner facilitates this by clustering tools into interchangeable kits, enabling localized, API-level repair without the prohibitive cost of global replanning ( Liu et al., 2025d ) . Similarly, ToolACE-R dynamically calibrates its revision effort based on task difficulty, balancing routing reliability against practical compute budgets in real-world deployments ( Zeng et al., 2025 ) .

#### 6.3.2 Iterative Tool Refinement

Iterative tool refinement is the mechanism by which agents turn fragile programs into dependable skills. It is central to self-improvement because tool errors are not merely execution-time failures. If unreliable tools are stored and reused, they can corrupt the agent’s future behavior through repeated retrieval and compounding errors. Refinement therefore serves both as debugging and as a gatekeeping process that controls what enters the structural skill repository ( Ravi et al., 2025 ; Dolcetti et al., 2025 ; Petrovic et al., 2025 ) .

A canonical refinement loop alternates between generation, execution, and revision. VOYAGER establishes this baseline by executing generated code, capturing error traces and environment feedback as learning signals ( 𝒮 t \mathcal{S}_{t} ), and feeding them back into subsequent revisions until a verifier confirms success ( Wang et al., 2023 ) . Building on this foundation, recent systems strengthen the refinement process across three dimensions: (1) Critique Specialization. To enhance diagnostic accuracy, systems move beyond generic self-reflection. For instance, STELLA deploys a dedicated critic agent to assess intermediate results and provide targeted feedback, mitigating the tendency of monolithic models to overlook domain-specific failure modes ( Jin et al., 2025 ) . (2) API Abstraction. Rather than merely revising raw action traces, several works focus on abstracting robust subroutines. SkillWeaver distills interactions into reusable web tools refined through execution feedback ( Zheng et al., 2025 ) , while PyVision dynamically refines Python programs to stabilize multimodal tool use against noisy perception ( Zhao et al., 2025b ) . This abstraction significantly improves skill transfer across tasks. (3) Interface Alignment. Addressing the semantic gap between agents and tools, systems like DRAFT iteratively refine tool documentation rather than the underlying code. This targets the critical observation that many execution failures stem from mismatches between natural language instructions and tool affordances, rather than from flawed implementations ( Qu et al., 2025 ) .

#### 6.3.3 Autonomous Tool Creation

Autonomous tool creation expands the agent’s capability boundary by synthesizing new executable functions when existing tools are insufficient. For self-improving agents, creation is most valuable when it converts one-off problem solving into reusable procedural knowledge. It also introduces its own risks. Newly created tools must be validated, documented, and integrated without destabilizing routing policies, otherwise tool growth can increase brittleness rather than autonomy ( Gao et al., 2026 ; Qiu et al., 2025c ) .