Self-Improvements in Modern Agentic Systems: A Survey — part 12/14
Cleaned-text code points [168231, 184194). Bibliography omitted here; source.md includes it.
Scientific discovery highlights three domain-specific challenges for self-improving agents. The first challenge is evaluation. Systems cannot easily verify novelty, correctness, or reproducibility on their own. Apparent improvements may instead reflect the exploitation of weak proxies. The second challenge is evidence management under heterogeneity. Improvements must persist across tools, data formats, and rapidly evolving literature. Stable scaffold evolution is therefore as important as policy learning. The third challenge is safety and governance. Experimental actions can be irreversible or hazardous. Scientific writing can also amplify misinformation when the evidence is weak. Progress in this domain therefore depends on reproducibility-centered evaluation, evidence tracking, and standardized experimental and computational protocols. It also depends on governance mechanisms that bound risk when agents propose or execute real-world scientific actions.
### 7.5 Embodied AI and Robotics
Embodied AI studies agents that perceive, reason, and act through a physical or simulated body. Compared with software engineering and web automation, this domain adds continuous state and action spaces, partial observability, safety-critical exploration, and the sim-to-real gap. These properties make self-improvement valuable but difficult. In practice, improvement is often realized as open-ended skill acquisition, where the agent accumulates reusable competence across tasks, embodiments, and environments.
##### Evaluation substrate and feedback structure.
Embodied evaluation is typically conducted in simulation suites and real-robot trials, with feedback ranging from dense rewards to sparse success criteria and human interventions. Benchmarks such as RLBench, ManiSkill2, and Meta-World enable controlled comparison of improvement loops in simulation ( James et al., 2020 ; Gu et al., 2023 ; Yu et al., 2020 ) . High-throughput simulators such as Isaac Gym reduce iteration cost and support large-scale training and ablations ( Makoviychuk et al., 2021 ) . Since evaluation protocols are discussed more generally in Section 8 , we focus here on how embodied settings instantiate self-improvement mechanisms.
##### FM improvement through autonomous practice and data flywheels.
A core route to embodied self-improvement is iterative data collection followed by policy updating, forming a data flywheel across interactions. RoboCat embodies this mechanism by training a multi-task, multi-embodiment manipulation policy and then using the trained policy to generate additional data for subsequent training iterations ( Bousmalis et al., 2024 ) . MEDAL++ proposes a nearly autonomous reinforcement learning loop in which the robot learns to both perform and undo tasks, enabling reset-free practice and improving success rates with minimal human supervision ( Sharma et al., 2023 ) . AutoRT scales real-world experience acquisition by using foundation models to propose diverse instructions and orchestrate robot fleets in the wild, producing large collections of interactions for later policy improvement ( Ahn et al., 2024 ) . Robot-Powered Data Flywheels formalize deployment as continual data collection and foundation-model adaptation, demonstrating improvement of vision-language components through robot-generated in-the-wild data ( Grannen et al., 2025 ) . Self-Improving Embodied Foundation Models further propose a post-training recipe that uses shaped success detection to support autonomous robot practice and skill acquisition beyond imitation data ( Ghasemipour et al., 2025 ) .
##### Scaffolding improvement through curricula, memory, and safety logic.
Embodied systems also benefit from scaffolding improvement because much of the difficulty lies in exploration, recovery, and safe interaction. RoboGen illustrates a generative simulation paradigm that automatically produces tasks, scenes, and supervision signals, effectively evolving the curriculum and data generation process across iterations ( Wang et al., 2024e ) . RACAS ( Ashley et al., 2026 ) accumulates the embodied knowledge of robot control through a structurally self-managed memory. AutoRT can be viewed as a scaffold-level improvement approach when its instruction proposal policy, risk filters, and orchestration logic are iteratively refined to increase coverage while controlling safety violations ( Ahn et al., 2024 ) . Retrieval-driven upskilling and curriculum refinement provide another scaffold-level mechanism, allowing agents to accumulate reusable task recipes and training specifications across deployments ( Zhu et al., 2025a ) . These structural artifacts can transfer across tasks even when the base model remains fixed.
##### Synthesis and open challenges.
Embodied self-improvement faces multiple practical constraints, including safety, non-stationarity, and hardware cost. Real robots may experience sensor drift, rare but severe failures, and irreversible actions. These factors limit simple trial-and-error strategies. They also make the improvement process highly dependent on the design of safe exploration and recovery mechanisms. Simulators can support large-scale training. However, differences between simulation and reality often make improvements achieved in simulation difficult to transfer to real platforms. This gap calls for curriculum learning methods and data collection strategies that are specifically designed for transfer. Evaluation is also challenging. Some improvements may only reflect shortcuts found for specific benchmarks. Real deployment conditions also vary across laboratories and platforms. Progress in this field therefore depends on safety-aware improvement procedures, curriculum learning methods for sim-to-real transfer, and cross-embodiment evaluation practices. These practices are needed to distinguish genuine skill acquisition from overfitting to specific benchmarks.
### 7.6 General Computer Control
General computer control studies agents that operate full operating systems and third-party applications through graphical user interfaces. This setting is broader than web navigation because it requires control over files, windows, dialogs, shortcuts, and application-specific workflows, often across multiple applications within a single task. The main difficulty is environment diversity: the agent must adapt to unseen software and interface conventions. Self-improvement in this domain is therefore best understood as acquiring reusable procedures for learning new applications from interaction.
##### Evaluation substrate.
Studying cross-interaction improvement requires reproducible operating-system environments with execution-based scoring. OSWorld provides a real computer environment that supports task setup and automated evaluation for open-ended computer tasks across operating systems ( Xie et al., 2024 ) . WindowsAgentArena offers a Windows-focused benchmark and scalable OS-level evaluation ( Bonatti et al., 2025 ) . OSWorld-MCP extends evaluation beyond GUI actions by measuring tool invocation ability under the Model Context Protocol ( Jia et al., 2026 ) . Further discussion of evaluation appears in Section 8 .
##### Scaffolding improvement via experience and procedural reuse.
In general computer control, early gains often come from scaffold improvement because agents must handle long horizons and application-specific idiosyncrasies. Agent S introduces experience-augmented hierarchical planning with continual memory updates and retrieval over past trajectories. This enables cross-task gains through reusable procedural knowledge and supports transfer across OS benchmarks when memory and retrieval policies are maintained across interactions ( Agashe et al., 2025 ) . SEAgent emphasizes autonomous mastery of novel software by generating curricula from simple to complex tasks and using a world-state model for step-wise trajectory assessment ( Sun et al., 2025b ) . These components instantiate scaffold evolution through task generation, evaluation heuristics, and reusable exploration routines that persist across episodes even when the base model is unchanged.
##### FM improvement through iterative training and verifier construction.
A complementary line updates θ \theta using automatically collected interaction traces and learned evaluative signals. UI-Genie proposes a self-improving pipeline that co-evolves an agent and a reward model for GUI tasks, addressing outcome verification and scalable data generation through reward-guided exploration ( Xiao et al., 2025 ) . GUI-Reflection trains self-reflection and error correction abilities through iterative online reflection tuning, turning failures into supervision for subsequent parameter updates ( Wu et al., 2025a ) . SEA proposes verifiable trajectory generation and step-wise reinforcement learning for long-horizon computer-use training ( Cheng et al., 2026 ) . ComputerRL studies sustained end-to-end online reinforcement learning on OSWorld and introduces alternating training phases to mitigate optimization pathologies in extended RL ( Lai et al., 2026 ) . PC Agent-E reduces reliance on large-scale human demonstrations by combining a small seed set with synthetic action decisions, providing an economical route to iterative policy improvement ( He et al., 2026 ) .
##### Synthesis and open challenges.
General computer control brings challenges that are not always apparent in purely web-based settings. Safety is a central concern because an agent may delete files, enter passwords, or initiate financial transactions. Any improvement process must therefore include safeguards and conservative recovery strategies. Verification is also difficult. In software engineering, unit tests can often provide clear feedback. In general computer control, success may depend on the operating system state, external accounts, or user-specific context. Another challenge is transfer. Agents need to learn new applications without relying too much on surface-level UI patterns. This calls for exploration strategies and curricula that encourage procedural abstractions. Progress will require reliable state-based verification where possible, cautious policies for irreversible actions, and adaptation mechanisms that support transfer across new applications. These issues also connect to early work on meta-reinforcement learning with self-modifying policies, where agents adapted by collecting long-term statistics about the effects of their own modifications during a lifelong trial ( Schmidhuber, 1994 ; Schmidhuber et al., 1997 ; Schmidhuber et al., 1998 ; Schmidhuber, 1999 ; Schmidhuber, 2006a ) .
## 8 Evaluation
A self-improving agent may update the parameters of its foundation model, or may change its scaffolding, such as prompts, memory, tools, and orchestration logic. Evaluation therefore should treat improvement as a process that unfolds over time. It should also separate real capability gains from artifacts created by the improvement pipeline. For improvement claims to be comparable across studies, the evaluation protocol should specify several elements. These elements include what persists across interactions, which feedback signals drive updates, what operational budgets constrain the agent, and where the boundaries of true capability transfer should be drawn. With these dimensions in view, Section 8.1 discusses metric-based and judge-based measurement. Section 8.2 then surveys the benchmarking landscape by distinguishing mechanism benchmarks from domain benchmarks.
### 8.1 Measuring Improvement
Evaluating agents solely on episodic tasks obscures their long-term evolution. Grounded in lifelong meta-reinforcement learning ( Schmidhuber, 1994 ) , robust validation must extend beyond static benchmark gains to assess whether self-modifications yield compounding, sustained benefits across the agent’s broader operational lifetime ( Schmidhuber et al., 1997 ; Schmidhuber et al., 1998 ; Schmidhuber, 1999 ) . To measure these compounding benefits, this subsection first outlines metric-based reporting requirements, followed by judge-based measurements for tasks lacking executable oracles.
##### Formalizing the Evaluation Objective.
In this section, we first formalize the evaluation of a self-improving agent. Recall that the agent configuration at iteration t t is 𝒜 t = ( θ t , Σ t ) \mathcal{A}_{t}=(\theta_{t},\Sigma_{t}) . Unlike static systems evaluated by a single terminal score, evaluating a self-improving agent requires tracking a performance trajectory over iterations t ∈ { 1 , … , T } t\in\{1,\dots,T\} subject to a cumulative resource budget b t ≤ B max b_{t}\leq B_{\text{max}} . At t t -th iteration, for a task x x drawn from a held-out evaluation distribution 𝒟 eval \mathcal{D}_{\text{eval}} , the agent generates an execution trace τ ∼ 𝒜 t ( x ) \tau\sim\mathcal{A}_{t}(x) . The overall capability at iteration t t is measured by the expected score: m t = 𝔼 x ∼ 𝒟 eval , τ ∼ 𝒜 t ( x ) [ Φ ( x , τ ) ] , m_{t}=\mathbb{E}_{x\sim\mathcal{D}_{\text{eval}},\tau\sim\mathcal{A}_{t}(x)}\left[\Phi(x,\tau)\right], (26) where Φ \Phi acts as the generic evaluator. The difference between metric-based and judge-based measurement lies in the instantiation of Φ \Phi :
• Metric-based measurement (Section 8.1.1 ): Φ metric \Phi_{\text{metric}} acts as a deterministic, executable evaluator. For instance, in software engineering, Φ metric ( x , τ ) ∈ { 0 , 1 } \Phi_{\text{metric}}(x,\tau)\in\{0,1\} directly verifies if the generated code in τ \tau passes the programmatic unit tests defined in x x . It provides objective grounding but is inherently limited to tasks with formal success criteria.
• Judge-based measurement (Section 8.1.2 ): Φ judge \Phi_{\text{judge}} acts as a parameterized evaluator. It approximates the objective by conditioning on a predefined rubric κ \kappa and relies on an auxiliary model θ judge \theta_{\text{judge}} (e.g., LLM-as-a-Judge), formulated as Φ judge ( x , τ , κ , θ judge ) \Phi_{\text{judge}}(x,\tau,\kappa;\theta_{\text{judge}}) . While this enables the evaluation of open-ended and long-horizon tasks, it introduces new vulnerabilities, such as the agent over-optimizing to the judge’s latent biases rather than the ground-truth objective.
#### 8.1.1 Metric-Based Measurement
Report trajectories under a fixed budget. Self-improvement unfolds over time and naturally exhibits plateaus or regressions. Evaluation should therefore report the full performance trajectory ( m t m_{t} ) across update iterations ( t t ) rather than exclusively highlighting a final peak score, bounded by a predefined resource budget ( B max B_{\text{max}} ) ( Xi et al., 2024 ) . The protocol should explicitly specify checkpoints, acceptance criteria, and early-stopping rules; without them, unbounded iterations risk overfitting to the evaluation distribution, thereby artificially inflating peak scores at the cost of general robustness. Furthermore, because trajectory dynamics, such as accumulated memories or code patches, are highly sensitive to initialization, reproducibility demands reporting expected performance and variance across multiple random seeds ( Siegel et al., 2024 ; Aleithan et al., 2024 ) .
Test transfer beyond the improvement signal. To confirm genuine capability gains rather than memorization of the feedback, evaluation should measure performance ( m t m_{t} ) on a held-out distribution ( 𝒟 eval \mathcal{D}_{\text{eval}} ) that does not overlap with the optimization data ( Deng et al., 2023 ; Lu et al., 2024c ) . To prevent agents from overfitting to the improvement process, rigorous protocols usually adopt two specific safeguards: employing private, unreleased task sets (hidden evaluation), or testing on newly generated tasks constructed after the foundation model’s knowledge cutoff (temporally shifted evaluation) ( Gu et al., 2023 ; Mialon et al., 2024 ) .