Self-Improvements in Modern Agentic Systems: A Survey — part 7/14
Cleaned-text code points [91590, 107255). Bibliography omitted here; source.md includes it.
Formally, as established in Section 3 , scaffolding improvement corresponds to transitions that modify the operational scaffold while explicitly keeping the foundation-model parameters frozen ( θ t + 1 = θ t \theta_{t+1}=\theta_{t} ). Driven by an execution-derived learning signal 𝒮 t \mathcal{S}_{t} (e.g., task outcomes, critique, or execution errors), the update is formalized as Σ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 Σ ( Σ 1 : t ; 𝒮 t ) \Sigma_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\Sigma}(\Sigma_{1:t};\mathcal{S}_{t}) . By maintaining a version history ( Σ 1 : t \Sigma_{1:t} ), the system inherently supports validation and rollback against harmful modifications across all components. We structure the discussion by increasing depth of architectural intervention. Concretely, we instantiate the scaffolding as Σ t := ( p t , m t , 𝒯 t , g t ) \Sigma_{t}:=(p_{t},m_{t},\mathcal{T}_{t},g_{t}) , where p t p_{t} denotes the prompt template, m t m_{t} the internal memory, 𝒯 t \mathcal{T}_{t} the external tool set, and g t g_{t} the control logic. Algorithm 2 summarizes the generic improvement loop across these components:
• Prompt-based improvement (Section 6.1 ) targets the agent’s input and instruction layer, formalized as p t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 p ( p 1 : t ; 𝒮 t ) p_{t+1}=\operatorname{\mathtt{IMPROVE}}_{p}(p_{1:t};\mathcal{S}_{t}) . As the primary semantic interface through which the agent perceives its tasks, optimizing the prompt provides a direct way to refine task communication, objectives, and constraints without altering the internal parameters of the foundation model.
• Memory-based improvement (Section 6.2 ) equips the agent with an evolving internal cognitive resource, updated via m t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 m ( m 1 : t ; 𝒮 t ) m_{t+1}=\operatorname{\mathtt{IMPROVE}}_{m}(m_{1:t};\mathcal{S}_{t}) . By dynamically storing, pruning, and retrieving past experiences, the agent shifts from memoryless execution to cumulative learning, supporting long-horizon reasoning and trustworthy adaptation.
• Tool-based improvement (Section 6.3 ) strengthens the agent’s execution interface through the update 𝒯 t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 𝒯 ( 𝒯 1 : t ; 𝒮 t ) \mathcal{T}_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\mathcal{T}}(\mathcal{T}_{1:t};\mathcal{S}_{t}) . By autonomously refining and managing external callable modules (e.g., web search, code interpreters), the agent translates internal decisions into precise, executable actions, effectively extending its capabilities beyond the native limits of the foundation model.
• Full scaffolding improvement (Section 6.4 ) represents the most profound level of architectural intervention, formalized as Σ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 Σ ( Σ 1 : t ; 𝒮 t ) \Sigma_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\Sigma}(\Sigma_{1:t};\mathcal{S}_{t}) . By treating the entire codebase and operational logic as a mutable substrate, the agent dynamically reconfigures how its perception, reasoning, and execution faculties are integrated into a coherent whole.
Crucially, these intervention types are compositional rather than mutually exclusive: a single update can edit multiple scaffold components simultaneously, and full-scaffolding methods naturally subsume component-level edits while adding archive-based exploration and stronger acceptance tests.
### 6.1 Prompt
The prompt serves as the agent’s core behavioral prior, defining how the foundation model parses its environment. Prompt optimization is therefore a central and highly accessible form of scaffolding improvement. While early methods relied on manual heuristic tuning, the field is rapidly developed toward automated, signal-driven improvement loops. Central to this development is the transition from scalar-based feedback to rich, structured linguistic critiques. By leveraging natural-language gradients, these systems now facilitate targeted, iterative updates that mirror the precision of gradient-based optimization in higher-dimensional strategy spaces.
In this section, prompt primarily refers to structural instruction layers that are reused across interactions, such as system prompts or stable policy templates. A closely related line of work optimizes context construction, including exemplar selection, retrieval assembly, and the maintenance of long-term playbooks. Although both are scaffold-level updates, they differ in target: optimizing a system prompt alters the agent’s core behavioral prior, whereas context optimization refines the dynamic conditioning of specific interactions. When discussing representative systems, we will explicitly indicate which object is updated.
As shown in Fig. 7 , we categorize prompt refinement methods based on the form and richness of the learning signal 𝒮 t \mathcal{S}_{t} . This yields four paradigms: (1) Scalar-Feedback Optimization , where 𝒮 t \mathcal{S}_{t} is a scalar performance score such as accuracy or reward; (2) Qualitative-Feedback Refinement , where 𝒮 t \mathcal{S}_{t} is a natural-language critique or suggestion for providing interpretable revision guidance; (3) Population-Based Evolution , where 𝒮 t \mathcal{S}_{t} consists of population-level fitness evaluation and selection signals over a set of candidate prompts; and (4) Textual Gradient Optimization , where 𝒮 t \mathcal{S}_{t} is structured directional guidance, often expressed as a textual gradient that specifies how the prompt should be revised. As listed in Table 2 , we further summarize representative systems and trade-offs across paradigms.
#### 6.1.1 Scalar-Feedback Optimization
Early approaches to prompt optimization usually rely on task-level quantitative metrics to evaluate candidates ( Zhou et al., 2022 ) . The core objective can be formally defined as finding an optimal prompt p ∗ p^{*} from a discrete space of possible prompts 𝒫 \mathcal{P} that maximizes a scalar performance score (e.g., task accuracy or reward): p ∗ = arg max p ∈ 𝒫 f ( p ) , p^{*}=\arg\max_{p\in\mathcal{P}}f(p), (17) where f ( p ) f(p) represents the evaluation function. In the context of our scaffolding improvement framework, this scalar score is used to constitutes the learning signal ( 𝒮 t = f ( p t ) \mathcal{S}_{t}=f(p_{t}) ). Considering 𝒮 t \mathcal{S}_{t} is non-directional, providing only a magnitude of success without explanatory context, these methods typically rely on structured search algorithms to navigate the discrete text space. APE pioneered this paradigm by using an LLM to propose instruction candidates, subsequently selecting the best one based on empirical evaluation scores via simple search ( Zhou et al., 2022 ) . Building on this, OPRO formalized a more contextualized search by constructing a meta-prompt that includes a trajectory of previously evaluated prompts alongside their scalar scores, indirectly guiding the LLM to propose higher-scoring candidates in subsequent iterations ( Yang et al., 2023 ) .
Besides, researchers have adapted advanced derivative-free optimization algorithms to operate strictly on scalar feedback. For instance, RL-based frameworks like RLPrompt treat prompt generation as a discrete reinforcement learning problem, directly optimizing a reward signal derived from downstream task accuracy ( Deng et al., 2022 ) . Similarly, to improve the sample efficiency of exploring the discrete text space, methods like InstructZero map prompts into a continuous latent space, leveraging Bayesian Optimization to predict and maximize scalar task rewards efficiently ( Chen et al., 2024 ) . Recent systems like BPO (Black-Box Prompt Optimization) further extend this score-driven paradigm to human preference alignment, using scalar preference scores to optimize user inputs without altering the underlying model weights ( Cheng et al., 2024a ) .
#### 6.1.2 Qualitative-Feedback Refinement
Building upon scalar-driven methods, a more nuanced approach focuses on iterative refinement using qualitative, interpretable feedback. Instead of a single score, the learning signal ( 𝒮 t \mathcal{S}_{t} ) manifests as a textual critique, error analysis, or natural-language suggestion, denoted as c t c_{t} . This richer signal allows for a targeted revision process, formally modeled as an iterative loop where each new prompt p t + 1 p_{t+1} is a function of the previous prompt p t p_{t} and its corresponding qualitative feedback c t c_{t} : p t + 1 = Refine ( p t , c t ) , p_{t+1}=\text{Refine}(p_{t},c_{t}), (18) where c t c_{t} is typically generated by an evaluator or the LLM itself based on historical execution: c t = Critique ( Output ( p t ) ) c_{t}=\text{Critique}(\text{Output}(p_{t})) . While foundational paradigms like Self-Refine ( Madaan et al., 2023 ) and multi-agent debate ( Du et al., 2024 ) demonstrated the power of self-critique for transient output correction, recent scaffolding improvements persist this qualitative feedback to update the agent’s prompting policy or instructional context. One prominent direction leverages error-driven qualitative analysis. Reflexion, for example, enables an agent to generate “verbal introspection” upon task failure, storing these qualitative reflections in memory to explicitly guide and constrain subsequent prompting attempts ( Shinn et al., 2023 ) . Expanding on this diagnostic capability, MAPS introduces an automated, LLM-tailored prompt optimization framework that explicitly induces and validates reusable natural-language rules from failure cases. By iteratively injecting these qualitative insights into the prompt, MAPS substantially optimizes the policy for complex tasks like unit test generation ( Gao et al., 2025a ) . Similarly, inspired by Chain of Hindsight (CoH), agents can learn from textual contrasts by reviewing a history of past attempts accompanied by qualitative evaluations ( Liu et al., 2024a ) . Another critical direction involves evolving the instructional context. To manage the growing complexity of text-based critiques, ACE introduces an agentic context engineering framework with a modular Generator–Reflector–Curator pipeline. ACE treats prompts and memory as evolving text playbooks, actively curating qualitative feedback while mitigating brevity bias and context collapse ( Zhang et al., 2025i ) . In specific domain applications, systems like Scrable utilize continuous qualitative evaluation to iteratively refine the structural system prompt for customer review generation, halting only when the textual quality reaches a predefined standard ( Azov et al., 2024 ) .
#### 6.1.3 Population-Based Evolution
To introduce more structured exploration and mitigate the risk of converging to local optima, researchers have adapted principles from evolutionary biology. In this paradigm, the learning signal 𝒮 t \mathcal{S}_{t} manifests as population-level fitness evaluations and selection pressures over a diverse pool of candidate prompts. Formally, these methods treat prompts as “genes” within a population P t = { p t ( i ) } i = 1 N P_{t}=\{p_{t}^{(i)}\}_{i=1}^{N} at generation t t . The evolutionary update leverages LLMs to apply semantic operators rather than simple string manipulations, following three key steps: 1. Selection: A subset of prompts survives based on a fitness evaluation function, which translates task performance into selection pressure ( 𝒮 t \mathcal{S}_{t} ).
2. Crossover: The LLM intelligently merges the semantic strengths of two parent prompts, generating offspring: p child = Crossover ( p t ( i ) , p t ( j ) ) p_{\text{child}}=\text{Crossover}(p_{t}^{(i)},p_{t}^{(j)}) .
3. Mutation: The LLM introduces semantic variations to explore new instructional phrasing: p ′ = Mutate ( p ) p^{\prime}=\text{Mutate}(p) .
The subsequent generation, P t + 1 P_{t+1} , is formed from the fittest individuals and their offspring. Foundational frameworks like EvoPrompt ( Guo et al., 2024 ) pioneered this by explicitly guiding LLMs to act as evolutionary operators, yielding crossover and mutation steps that are semantically meaningful and far surpass classical random character edits. Expanding the depth of this standard evolutionary search, frameworks like Promptbreeder ( Fernando et al., 2024 ) introduces a profound self-referential mechanism where the LLM evolves not only the task prompts but also the "mutation prompts" (the instructions governing how new offspring are generated). Addressing scenarios where scalar fitness scores are unavailable, DEEVO creatively structures 𝒮 t \mathcal{S}_{t} through multi-agent debates, utilizing the win-rate of LLM-driven argumentation as the evolutionary fitness signal ( Nair et al., 2025 ) . Crucially, demonstrating the compositional nature of scaffolding improvement, recent work integrates population-based search with qualitative feedback. In reflective prompt evolution frameworks like GEPA ( Agrawal et al., 2026 ) , the evolutionary process is guided not merely by a scalar fitness score, but by a meta-level reflection step. After evaluating a generation, a reflector LLM analyzes successes and failures to generate textual critiques. This qualitative feedback explicitly informs the next generation’s mutation and crossover operators, making the search highly targeted and sample-efficient. The success of such hybrid approaches highlights the distinct advantage of replacing opaque scalar rewards with directional, semantic guidance—directly paving the way for the formalized textual gradient methods discussed next.
#### 6.1.4 Textual Gradient Optimization
The most recent paradigm is distinguished by a formal, mathematically-inspired treatment of the feedback signal, drawing direct analogies to gradient descent in continuous optimization. Rather than relying on heuristic edits or scalar search, the learning signal ( 𝒮 t \mathcal{S}_{t} ) is formalized as a textual gradient —a structured, directional feedback message that explicitly diagnoses why an output is incorrect and prescribes a precise revision vector. This optimization process can be expressed with an analogous update rule: p t + 1 = p t ⊕ g ( p t ) , p_{t+1}=p_{t}\oplus g(p_{t}), (19) where the textual gradient g ( p t ) g(p_{t}) serves directly as our learning signal ( 𝒮 t = g ( p t ) \mathcal{S}_{t}=g(p_{t}) ). The ⊕ \oplus operator denotes a textual update step, where an LLM acts as the optimizer to semantically apply the gradient’s guidance and revise the prompt. While the foundational concept of a “textual gradient” was introduced in Automatic Prompt Optimization (APO) ( Pryzant et al., 2023 ) , this direction has been significantly advanced by framing agentic workflows as computational graphs. TextGrad, for instance, operationalized this by introducing automatic differentiation via text, allowing qualitative gradients to backpropagate through complex, multi-component language systems ( Yuksekgonul et al., 2024 ) . Concurrently, semantic backpropagation frameworks have further formalized this process, executing first-order-like optimization on text nodes to achieve principled prompt refinement ( Wang et al., 2024b ) . The sophistication of this paradigm is profoundly underscored by meta-level frameworks like MetaTextGrad. Rather than solely optimizing task prompts, it uses an LLM to dynamically optimize the “optimizer prompts” (the instructions dictating how the textual gradients are computed and applied), effectively allowing the agent to self-improve its own improvement process ( Xu et al., 2025a ) . Looking forward, this formal feedback mechanism could bridge the gap between scaffolding and parameter-based learning: a textual gradient might be translated into low-rank parameter updates, seamlessly integrating prompt engineering with model fine-tuning.
### 6.2 Memory