Self-Improvements in Modern Agentic Systems: A Survey — part 6/14

Cleaned-text code points [75959, 91590). Bibliography omitted here; source.md includes it.

Let R R denote a critique-and-revision operator. Given a candidate output y t y_{t} , the agent produces e t = R ⁡ ( x , y t ) , e_{t}=R(x,y_{t}), (16)

where e t e_{t} is composed of c t c_{t} and y t ∗ y_{t}^{\ast} , c t c_{t} is a critique and y t ∗ y_{t}^{\ast} is a revised output. The pair ( y t , y t ∗ ) (y_{t},y_{t}^{\ast}) can instantiate a preference signal y t ∗ ≻ y t y_{t}^{\ast}\succ y_{t} , while the critique c t c_{t} can be used as explanatory supervision or as a natural-language training signal. “ReST meets react” ( Aksitov et al., 2024 ) combines agentic reasoning and self-training to improve multi-step problem solving. SELF ( Lu et al., 2024b ) uses language feedback to iteratively refine generated answers, transforming model-produced critiques and revisions into signals for self-improvement. RISE ( Qu et al., 2024 ) uses interaction-based recursive self-reflection and fine-tunes based on improved predictions. Reflect, Retry, Reward ( Bensal et al., 2025 ) follows a similar loop in which the model reflects on failures, retries the task, and uses the resulting feedback to guide reinforcement learning. The AlphaAllM approach ( Tian et al., 2024 ) further integrates search and critique, using model-generated evaluations during the search process to construct stronger training signals. These methods exploit the ability of foundation models to provide semantic, actionable, and informative feedback that is richer than simple scalar rewards. A critique can identify missing constraints, unsupported claims, incorrect reasoning steps, or unsafe implications, thus providing a richer signal for parameter updates. This flexibility also introduces failure modes since the model may produce plausible but incorrect critiques, or learn to satisfy the surface form of the critique rather than its underlying goals. Therefore, critique-based feedback is more reliable for parameter-level self-improvement when combined with safeguards such as comparing the original and modified outputs, filtering low-quality critiques, using heterogeneous critique models, or combining internal critiques with external validation when available.

##### Trade-offs and safeguards.

The advantage of intrinsic evaluation feedback lies in reducing reliance on human annotation and converting the agent’s own output into a learning signal that can be directly used by the parameter update algorithm. Its main risk is that the evaluator is often tightly coupled with the policy to be improved, so this loop may reinforce common blind spots, reward outputs that conform to the model’s existing preferences, overfit superficial evaluation criteria, or become biased with repeated use and reinterpretation of the evaluation criteria. Signals based on confidence and consistency increase vulnerability when the model is poorly calibrated or when there are correlations between its samples. Therefore, reliable use of this paradigm requires safeguards such as separating the generator and evaluator at different checkpoints or model families, maintaining external anchors by retaining human annotations or context-based validation, treating disagreements between evaluators as signals of uncertainty, and regularly reviewing the scoring criteria, rewarding the model, and evaluating quality. In practice, intrinsic evaluation feedback can be viewed as a component of a broader improvement loop, supplemented by demonstrations, external validation, and exploratory experiences, rather than as the only source of supervision.

### 5.3 Extrinsic Exploratory Experience

The preceding two subsections focused on intrinsic self-improvement signals, where the agent improves the foundation model using its own generated demonstrations (§ 5.1 ) or evaluative judgments (§ 5.2 ). Extrinsic exploratory experience differs in that the learning signal is grounded in what happens after the agent acts. Under our foundation-model improvement formalism, the learning signal is instantiated as experience, 𝒮 t ≈ τ t \mathcal{S}_{t}\approx\tau_{t} , where τ t \tau_{t} denotes trajectories collected by executing the policy π θ t , Σ t \pi_{\theta_{t},\Sigma_{t}} in a task environment or its learned proxy. The parameter update can be written as θ t + 1 = IMPROVE θ ( θ 1 : t ; τ t ) \theta_{t+1}=\text{IMPROVE}_{\theta}(\theta_{1:t};\tau_{t}) , with Σ t + 1 = Σ t \Sigma_{t+1}=\Sigma_{t} . While this view connects self-improvement back to the classical reinforcement learning framework ( Silver and Sutton, 2025 ; Zhao et al., 2024a ) , experience for foundation-model agents is not just a stream of state-action-reward tuples ( s , a , r , s ′ ) (s,a,r,s^{\prime}) . A trajectory may contain web pages, screenshots, code logs, compiler errors, tool calls, and intermediate reasoning traces. Because these artifacts are readable by foundation models, the same experience can be flexibly reused across reinforcement learning, supervised fine-tuning, preference construction, and failure analysis. However, acquiring and utilizing this rich experience introduces distinct difficulties: interaction can be slow or costly, rewards may be sparse or delayed, verifiers can be gamed, and learned world models ( Schmidhuber, 1990 ; Schmidhuber, 2015 ; Nanbo et al., 2025 ) may produce plausible but counterfactual transitions.

To address these challenges, we organize existing methods by how the exploratory experience is obtained. For interaction with grounded task environments, the agent acts in real, sandboxed, or rule-based task settings, and the learning signal comes directly from the task environment (e.g., state changes, unit tests, or task-specific verifiers). For interaction with simulated proxy environments, a learned world model serves as a proxy for the task environment, generating predicted states, rollouts, or outcomes for policy improvement. Notably, these two modes are not mutually exclusive. In fact, as demonstrated by early controller-world-model architectures ( Schmidhuber, 1990 ) , a system may use grounded interaction to collect data and update its world model, and subsequently use simulated interaction to plan, explore, or improve its policy.

#### 5.3.1 Interaction with Grounded Task Environments.

In this mode, the agent learns by directly acting in a task environment. Typical settings include code interpreters for coding agents, web APIs for web agents, mobile user interfaces (UIs) for GUI agents, and physical environments for embodied agents. The learning signal comes from the environment’s response—such as state changes, execution traces, or unit tests. Training then proceeds by collecting these interaction trajectories and updating the foundation-model policy using reinforcement learning methods like PPO ( Schulman et al., 2017 ) , or preference-based objectives like DPO ( Rafailov et al., 2023 ) when successful and unsuccessful trajectories can be contrasted. A useful way to organize this line of work is by the source of feedback:

(1) Programmatic verifiers provide the clearest form of grounded signal. When a task admits an executable check, the agent can be trained without learning a separate reward model. Code generation is the canonical case, since unit tests provide direct pass or fail feedback for proposed programs. Agent-RLVR ( Da et al., 2025 ) , for example, uses unit-test outcomes to guide policy optimization, contrasting successful programs against failed attempts. The same pattern extends beyond code whenever an output can be checked by an external procedure, including structured query language (SQL) execution, theorem proving, tool use with checkable post-conditions, and structured tasks with deterministic success criteria.

(2) Learned reward models evaluate trajectories collected from the task environment. In this setting, the task environment naturally produces the trajectory, while the learned evaluator only scores its success. WebRL ( Qi et al., 2025 ) trains an outcome-supervised reward model to label web-navigation trajectories automatically, reducing reliance on hand-crafted success criteria or human annotation. UI-Genie ( Xiao et al., 2025 ) couples policy learning with reward-model refinement, using validated trajectories to train the agent and step-level labels from both successful and failed trajectories to improve the evaluator. Related work such as MobileGUI-RL ( Shi et al., 2025b ) adapts Group Relative Policy Optimization (GRPO) to mobile GUI navigation with trajectory-aware advantages and rewards that combine task success with execution efficiency. Across these methods, the feature is that feedback is computed over rich linguistic or visuolinguistic trajectories, which makes it natural to construct rewards, preferences, and step-level supervision from the same interaction record.

A third approach leverages (3) self-generated tasks to acquire grounded experience. In these systems, the agent may propose new tasks or goals, but the learning signal remains extrinsic because candidate solutions are accepted or rejected by execution, verification, or environmental feedback. Absolute Zero ( Zhao et al., 2025a ) uses self-play to generate tasks and solutions in an open-ended environment, while execution-based validation determines which solutions are retained for learning. ETO ( Song et al., 2024c ) learns more conservatively from contrasts between successful and failed trajectories in fixed environments. These methods blur the boundary between intrinsic and extrinsic signals at the level of task selection, but not at the level of supervision. The agent may choose what to try, while the environment determines whether the attempt succeeds.

Finally, standardized platforms have emerged to facilitate research across these grounded interaction modes. For example, AgentGym ( Xi et al., 2024 ) provides unified APIs for interaction, evaluation, and training across multiple agent tasks. Such platforms significantly reduce the engineering cost of iterative experience collection, making it easier to compare reward sources, training objectives, and update procedures under shared environmental feedback.

#### 5.3.2 Interaction with Simulated Proxy Environments

The simulated-interaction mode equips the agent with an internal predictive model of the environment. A typical pipeline first collects interaction trajectories from a task environment, then trains a world model to predict the environment’s response to the agent’s actions ( Schmidhuber, 1990 ; Ha and Schmidhuber, 2018 ) . Abstractly, this can be written as a learned dynamics model W ( s k + 1 , r k ∣ s k , a k ) W(s_{k+1},r_{k}\mid s_{k},a_{k}) , which predicts the next state and reward from the current state and action at step k k of a trajectory. The policy can then obtain additional experience by interacting with this learned proxy instead of repeatedly querying the original task environment. This internal simulation improves sample efficiency and reduces the cost or risk of exploration, especially when direct interaction is slow, expensive, or unsafe ( Ding et al., 2025 ; Richens et al., 2025 ) .

While classical world models successfully predict transitions over both compact state vectors and raw pixel spaces ( Ha and Schmidhuber, 2018 ) , what distinguishes foundation-model agents is the extensive prior knowledge embedded in the learned dynamics. In FM-based systems, the world model is typically a generative language, vision-language, or video model that predicts high-dimensional observations—such as the next web page or video frame—in the exact format consumed by the policy. This design offers two practical advantages. First, large-scale generative pretraining provides a powerful prior over environment dynamics, significantly reducing the task-specific interaction needed to fit a useful proxy environment. Second, because generated observations lie in the same representation space as the policy input, simulated experience can be used directly without a separate representation-alignment step. This holds true for both linguistic web models and pixel-space video models like WMPO ( Zhu et al., 2025b ) . Together, these advantages make FM-based world models effective for planning, trajectory synthesis, and failure analysis.

This pattern is most developed in web navigation, where direct interaction can be slow and search over action sequences can be expensive. WebEvolver ( Fang et al., 2025b ) trains a coevolving world model to predict next web observations and uses simulated rollouts to refine the agent policy. WebSynthesis ( Gao et al., 2025b ) uses a learned web world model for reversible, search-based trajectory synthesis, while WebDreamer ( Gu et al., 2025 ) leverages a web transition model for model-based planning to guide action selection. Beyond web navigation, SPA ( Chen et al., 2025a ) learns explicit state-estimation and transition models through self-play fine-tuning, using them to initialize and stabilize downstream policy optimization. In embodied control, WMPO ( Zhu et al., 2025b ) learns a pixel-space world model and optimizes the policy over imagined rollouts to avoid costly physical trial-and-error. A related direction uses structured memories of prior interaction rather than full generative dynamics. GLoW ( Kim and seung-won hwang, 2026 ) maintains a dual-scale textual world memory—consisting of a global frontier of high-value discoveries and local advantage reflections—to guide a Go-Explore-style agent in text-based games, achieving strong performance with 100–800 × \times fewer real environment interactions than RL baselines.

##### Challenges.

Across extrinsic exploratory experience, foundation-model agents inherit classical difficulties of RL, including sparse and delayed rewards, low-throughput real interaction, overfitting to imperfect proxy environments. They also introduce failure modes specific to this setting. Reward hacking through language is more readily available than in classical RL: a foundation-model agent can satisfy the literal condition of a verifier (e.g., exploiting prompt loopholes in an LLM judge) without solving the underlying task, because the verifier’s specification is itself a linguistic object that the agent can manipulate. Capability regression arises because extensive RL updates on narrow extrinsic rewards can erode the broader competencies the foundation model acquired in pretraining, a tension absent in agents trained from scratch. Hallucinated dynamics pose a distinctive risk for world-model approaches, since generative simulators can fabricate plausible but incorrect transitions that the policy then learns to exploit. Finally, the linguistic, multimodal nature of trajectories creates a trajectory-length and context-window tension unique to this setting: long-horizon experience must be compressed or summarized to fit within the model’s context before it can be used as a learning signal. Current research therefore emphasizes more reliable extrinsic verifiers and reward models, world models with calibrated uncertainty over their own predictions, and training procedures that update the foundation model on extrinsic experience without sacrificing its general capabilities.

## 6 Scaffolding Improvement

This section analyzes the scaffolding improvement paradigm for FM-based agents, where we approach this by decomposing the agent’s operational scaffold into core components. This workflow begins with receiving and interpreting instructions, then contextualizes them using internal knowledge and memory, proceeds to executing actions in the world, and ultimately evolves the overall operational structure.