Self-Improvements in Modern Agentic Systems: A Survey — part 11/14

Cleaned-text code points [152731, 168231). Bibliography omitted here; source.md includes it.

A first line of work builds reproducible environments and demonstrations, which are prerequisites for systematic self-improvement. Mind2Web provides diverse instruction-following trajectories on real websites and supports supervised learning or imitation as an initial improvement step ( Deng et al., 2023 ) . WebArena offers a self-hostable environment with long-horizon tasks and execution-based success criteria ( Zhou et al., 2024 ) . VisualWebArena extends evaluation to visually grounded web tasks and exposes failures in perception and grounding ( Koh et al., 2024 ) . WorkArena targets enterprise-style workflows and highlights the gap between current agents and routine knowledge-worker automation ( Drouin et al., 2024b ) . BrowserGym further unifies evaluation across web-agent benchmarks, reducing experimental fragmentation and enabling more controlled comparisons of training and scaffold design ( Drouin et al., 2024a ; de Chezelles et al., 2025 ) . These resources are not self-improving methods themselves, but they provide the data and measurement substrate needed to study improvement across interactions.

##### Scaffolding improvement with memory and grounding.

In web environments, many improvements come from better scaffolding because robust grounding remains a central bottleneck. SeeAct couples visual understanding with action grounding on live websites, reducing brittle action selection in long-horizon execution ( Zheng et al., 2024 ) . WebCoach is more directly aligned with self-improvement: it equips browsing agents with cross-session memory curated from new trajectories, allowing the agent to avoid repeated mistakes without retraining the base model ( Liu et al., 2025a ) . ReAP similarly stores and reuses reflections over successful and failed trajectories to improve later web navigation ( Azam et al., 2025 ) . These approaches instantiate scaffolding improvement by updating memory content, retrieval policies, and grounding-related structural artifacts that transfer across tasks.

##### FM improvement from interaction feedback.

A complementary line updates θ \theta using web interaction signals. Patel et al. (2024) study self-improvement on WebArena by fine-tuning on mixtures of model-generated data and show that iterative training can improve web-agent capability. OpenWebVoyager proposes an exploration-and-feedback loop on real websites, improving its policy by learning from high-quality trajectories over multiple iterations ( He et al., 2024b ) . Several works adopt reinforcement learning for long-horizon browser control. WebRL introduces a self-evolving online curriculum that generates new tasks from failures and combines it with outcome-supervised reward modeling ( Qi et al., 2025 ) . WebAgent-R1 trains web agents via end-to-end multi-turn reinforcement learning with binary success rewards and reports gains on WebArena-Lite ( Wei et al., 2025d ) . Beyond navigation benchmarks, Agent Q studies learning from both successful and unsuccessful trajectories in WebShop and shows that such experience can improve generalization in multi-step web interaction ( Putta et al., 2024 ) . These FM-oriented methods also reveal a recurring difficulty: web feedback is often sparse, so improvement depends heavily on reward modeling, curriculum design, and the stability of the evaluation environment.

##### Synthesis and open challenges.

Web automation faces a core challenge of self-improvement. This challenge comes from non-stationarity and weak observability. Page layouts and interaction flows often change over time. These changes can quickly weaken gains that were measured on static snapshots. Scaffolding improvement can help systems update grounding, state abstraction, and recovery behavior more quickly. Yet it can also overfit to specific websites. It may also increase safety risks, such as prompt injection or unintended high-impact actions ( Chen et al., 2026 ) . FM improvement can support more transferable navigation skills. However, it depends on stable training signals and careful treatment of stale data. Progress therefore requires drift-aware evaluations. It also requires sandboxed protocols that separate safe exploration from irreversible actions. Finally, it requires grounding-centered improvements that transfer across layouts, DOM structures, and interaction conventions. These improvements should avoid memorizing site-specific patterns.

### 7.3 Games and Strategic Reasoning

Games remain a central testbed for self-improving agents because they provide repeatable interaction, well-defined objectives, and scalable feedback ( Stanić et al., 2023 ) . Even when state spaces are large and long-horizon planning is required, games offer closed training loops in which agents can generate experience through self-play ( Silver et al., 2018 ; Silver et al., 2017a ; Schrittwieser et al., 2020 ; OpenAI et al., 2019 ) . This makes the domain a natural fit for our taxonomy. Many game agents improve by updating model or policy parameters θ \theta through reinforcement learning, while others improve by evolving the scaffold Σ \Sigma , such as curriculum logic, planning routines, or memory structures that store reusable skills and experience.

##### FM improvement through self-play and outcome feedback.

Recent work extends self-play to foundation models by using strategic games as sources of interaction data and outcome-based learning signals. SPAG trains language models through a two-player adversarial game and improves the policy via self-play reinforcement learning ( Cheng et al., 2024b ) . SPIRAL studies multi-turn zero-sum self-play and shows that self-generated competitive interactions can provide a scalable signal for improving reasoning without human-labeled data ( Liu et al., 2026 ) . In adversarial game suites, SCO-PAL performs step-level policy optimization from game interaction data and identifies self-play as an effective opponent-selection strategy for strategic reasoning ( Zhang et al., 2025m ) . Self-play reinforcement learning with limited initial data further examines how self-generated interactions can bootstrap improvement when external supervision is scarce ( Fang et al., 2025c ) .

Imperfect-information and multi-agent settings introduce additional challenges in credit assignment and payoff estimation ( Zhuge et al., 2023 ; Stanić et al., 2023 ) . MARSHAL proposes an end-to-end reinforcement learning framework for multi-agent self-play across cooperative and competitive strategic games, and reports transfer beyond games to multi-agent reasoning benchmarks ( Yuan et al., 2026 ) . In negotiation-heavy environments, DipLLM studies fine-tuning for equilibrium-oriented policies in Diplomacy, providing a parameter-updating route for language-mediated strategic interaction ( Xu et al., 2025e ) . For high-variance payoff domains, SPRL uses reward shaping from self-play payoffs to stabilize training in complex strategic games ( Qi et al., 2026 ) . Social deduction games provide another setting in which language agents can improve long-horizon strategic behavior through reinforcement learning from interaction outcomes ( Xu et al., 2024b ) .

##### Scaffolding improvement through curricula and reusable skills.

Games also support scaffolding improvement because skills, procedures, and curricula can be represented explicitly and reused across interactions. Voyager exemplifies this paradigm in Minecraft by maintaining an expanding library of executable skills and an automatic curriculum that persists across tasks ( Wang et al., 2023 ) . Odyssey extends this direction with a structured open-world skill library and a planner–critic loop that improves long-horizon execution through skill reuse and prerequisite checking ( Liu et al., 2025c ) .

Beyond open-world games, several methods construct reusable skills from interaction trajectories. Skill Set Optimization extracts high-reward subtrajectories, converts them into transferable skills, and prunes ineffective ones, yielding continual in-context policy improvement in game-like environments ( Nottingham et al., 2024 ) . ExpeL shows that agents can accumulate experiences, distill them into natural-language lessons, and reuse them as in-context demonstrations for later decision making ( Zhao et al., 2024a ) . In strategic and negotiation-heavy games, Richelieu maintains memory and reflection across self-play interactions, using accumulated experience to revise planning and negotiation behavior without weight updates at every step ( Guan et al., 2024 ) . Recent self-evolving strategic systems similarly emphasize structural artifacts and long-term memory as the substrate for iterative strategy refinement across repeated games ( Belle et al., 2025 ) .

##### Synthesis and open challenges.

Games offer unusually clear conditions for studying self-improvement. Their environments can be simulated, and feedback can be collected at scale. At the same time, games reveal risks that may be less visible in software engineering or web automation. Self-play can create brittle competence. A system may exploit a simulator or a narrow opponent distribution. It may then fail when the population shifts or when the rules change. In multi-agent games, improvement may also become non-monotonic. Strategic cycling can make progress difficult to measure. Evaluation against a fixed set of opponents may therefore give a misleading view of performance. Language-based strategy raises further safety concerns. These concerns include persuasion and deception. Win-loss objectives alone cannot govern such behaviors. Progress in this domain requires population-based evaluation that captures non-transitivity. It also requires training designs that promote robustness under rule and opponent shifts. Explicit constraints are also needed for language-mediated interaction.

### 7.4 Scientific Discovery

Scientific discovery has long motivated self-improving and curiosity-driven agents. Earlier work on artificial scientists and artificial curiosity framed learning agents as systems that invent informative experiments, reduce uncertainty, maximize learning progress or information gain, and iteratively improve their world models through self-generated interaction ( Schmidhuber, 1990 ; Schmidhuber, 1991c ; Storck et al., 1995 ; Schmidhuber, 1997 ; Schmidhuber, 2003a ; Schmidhuber, 2013 ; Schmidhuber, 2015 ) . Contemporary FM-based scientific-discovery agents extend this line with large-scale language modeling, tool use, literature grounding, and automated workflow orchestration. In the FM era, the domain has become a high-impact setting for self-improving agents because it spans the full research lifecycle, including hypothesis generation, writing, experiment design, execution, analysis, and communication ( Lu et al., 2024a ; Yamada et al., 2025 ; Xiong et al., 2025 ) . Scientific discovery also differs from software engineering and web automation. Scientific environments are fragmented across subfields, rely on heterogeneous tools and data formats, and often provide feedback that is delayed, costly, or only partially automated. As a result, self-improvement in this domain is often realized as a research workflow that accumulates reusable artifacts across projects, rather than as a single response-level correction.

##### Evaluation substrate and feedback structure.

Scientific discovery systems rely on diverse feedback signals. In computational research, agents can run code, simulations, and ablations. They can receive feedback from metrics, error traces, and experimental outcomes. In experimental science, feedback is often delayed and noisy. It is also constrained by reproducibility, cost, and safety. These properties shape which improvement pathway is feasible. FM improvement becomes more practical when reliable executable signals are available. Scaffolding improvement becomes more central when the main bottleneck lies in tool access, protocol selection, and evidence management across heterogeneous resources. More broadly, scientific discovery has long motivated intrinsic feedback criteria beyond immediate task success. These criteria include learning progress, information gain, compression progress, and informative experiment design ( Storck et al., 1995 ; Schmidhuber, 2007 ; Schmidhuber, 2010 ; Schmidhuber, 2013 ) .

##### Scaffolding improvement through tool expansion and workflow evolution.

A prominent trend frames self-improvement as a tool-augmented research loop. In this loop, the agent proposes hypotheses, invokes specialized tools, critiques results, and iterates. ChemCrow shows that language-model control over a broad set of chemistry tools can improve reliability and capability coverage. Under our definition, ChemCrow becomes self-improving when its tool repertoire or selection policy is updated across tasks ( M. Bran et al., 2024 ) . SciAgents emphasizes structured knowledge and multi-agent role specialization. It uses ontological graphs and in-situ learning to refine hypotheses across fragmented literature and data sources ( Ghafarollahi and Buehler, 2024 ) . HoneyComb moves closer to explicit scaffold evolution. It constructs and refines domain tools and maintains a curated scientific knowledge base. These updates correspond to structural changes in the tool layer and retrieval policy ( Zhang et al., 2024a ) .

A second line focuses on end-to-end research orchestration. The AI Scientist and AI-Scientist-v2 implement iterative pipelines that generate ideas, write and debug code, run experiments, analyze results, and draft manuscripts, with repeated refinement driven by internal critique and experimental outcomes ( Lu et al., 2024a ; Yamada et al., 2025 ) . AI co-scientist studies a complementary setting in which multi-agent debate and evolution are aligned with scientist-provided objectives and constraints; improvement appears as stronger hypothesis proposals under evidence tracking rather than as next-token likelihood optimization ( Gottweis et al., 2025 ) . In our taxonomy, these systems primarily instantiate scaffolding improvement by revising planning strategies, experimental protocols, evaluation rubrics, and tool-use policies.

##### Model improvement through closed-loop experimentation.

Parameter-level improvement is also possible in scientific discovery, though it often targets either scientific surrogate models or the agent policy itself. Self-driving laboratories and closed-loop experiment design provide a mature template: iterative experimentation updates internal models of the objective landscape, which then guide the next experiments ( Tobias and Wahab, 2025 ) . When reliable supervision is available, the agent policy can also be updated from experimental or computational outcomes. Coscientist integrates web search, code execution, and laboratory automation to design, plan, and run chemistry experiments ( Boiko et al., 2023 ) . ORGANA and LLM-RDF further illustrate LLM-centered orchestration of experimental workflows and instrument-facing actions, opening a path toward learning from experimental success and failure when those signals can be formalized ( Darvish et al., 2024 ; Ruan et al., 2024b ) . These directions are promising, but they also show that parameter-level improvement depends critically on feedback validity and on controls that prevent weak proxies or spurious correlations from being mistaken for scientific progress.

##### Synthesis and open challenges.