Self-Improvements in Modern Agentic Systems: A Survey — part 5/14

Cleaned-text code points [60111, 75959). Bibliography omitted here; source.md includes it.

The strategy of data generation is a critical factor for the quality and scalability of a self-improvement loop. The underlying techniques start with a small set of example seeds and then build a larger instruction-output pair corpus. ( Wang et al., 2022 ; Zhao et al., 2024b ) . Methods such as Evol-Instruct ( Xu et al., 2024a ) go beyond simple volume expansion. They drive complexity evolution by using LLM to rewrite instructions and gradually increase the difficulty of the instructions, thereby overcoming the limitations of humans in building highly complex scenes. To improve data quality, more sophisticated methods have added a refining or filtering stage. Huang et al. (2023) employs self-consistency to filter high-confidence inference paths, while Singh et al. (2024) uses external verifiers like unit tests to select only correct solutions from the model’s attempts. In addition to bootstrapping and filtering, some approaches conceptualize data generation as a form of curriculum learning. Simonds and Yoshiyama (2025) create a learning curriculum by recursively decomposing complex problems into simpler sub-problems, and Qin et al. (2025) explicitly introduce mechanisms to expand the sample pool to counteract the risk of diminished output diversity over time. More recently, Test-Time Self-Improvement (TT-SI) ( Acikgoz et al., 2025 ) shifts the paradigm from massive offline generation to highly sample-efficient, on-the-fly adaptation. By detecting weak cases at inference time via uncertainty estimation, the agent self-generates targeted training examples for its specific blind spots and performs targeted low-rank adaptation (LoRA) fine-tuning; this yields immediate, per-instance improvements with a fraction of the data cost.

Each of these strategies has distinct strengths and failure modes. For example, self-consistency filtering rests on the assumption that correct reasoning tends to be self-agreeing. It can break down when the model is confidently wrong, because repeated reasoning may repeatedly converge to the same incorrect conclusion ( Chen et al., 2023 ; Hamidieh et al., 2026 ) . Curriculum-based generation can significantly improve learning performance for complex tasks, but it becomes vulnerable when the chosen decomposition method is flawed or ignores necessary contextual information. For example, this can occur when the subproblems are constructed in a way that no longer preserves the constraints of the original task ( Zhou et al., 2023 ; Xu et al., 2024a ) . Interactive refinement also depends on the model’s ability to detect and correct its own errors. When certain types of errors are beyond the model’s perception range, the improvement process may exacerbate these errors rather than eliminate them ( Shinn et al., 2023 ; Huang et al., 2024a ) . Therefore, choosing an appropriate generative strategy requires matching the method to the model’s current capabilities while avoiding mechanisms that focus on or amplify existing blind spots in the model.

##### Data formats.

These intrinsic demonstrations take various forms, ranging from simple input-output pairs to complex, structured artifacts designed to supervise specific cognitive abilities. Basic forms include simple instruction-response pairs, often with explicit reasoning traces added (e.g., chain-of-thought explanations) to make the model’s intermediate reasoning steps transparent and trainable ( Huang et al., 2023 ) . However, in scenarios involving multi-step decision-making or tool usage, agents often struggle with long-term tasks because suboptimal behaviors accumulate, eventually causing them to deviate from the correct path. To address this issue, the generated data should capture structured trajectories, rather than the final result. For example, tool application programming interface (API) calls or code execution sequences, combined with intermediate environment feedback and validation labels ( Shi et al., 2025a ; Singh et al., 2024 ) , can provide fine-grained supervision, teaching agents how to handle dependencies and recover from errors.

This inherent generative paradigm also extends to multimodal scenarios, where agents synthesize training samples, combining visual inputs such as images or videos with textual descriptions and reasoning to improve performance on tasks including visual reasoning ( Zhang et al., 2025k ) . At a higher level of abstraction, these demonstrations can take the form of metacognitive artifacts. For example, decomposed problem trees ( Simonds and Yoshiyama, 2025 ) and self-editing instructions ( Zweiger et al., 2025 ) , both of which can support complex planning and self-modification. More generally, the choice of data format should be consistent with the target capability, as different formats encode different types of supervisory information. Inference trajectories are suited for logical reasoning, code combined with tests is well-suited for programming, and structured action logs provide supervision for tool usage.

##### Challenges and safeguards.

This intrinsic generative paradigm still faces considerable challenges despite its success. First, recursively training the model on the generated corpus introduces the risk of model collapse and forgetting ( Shumailov et al., 2024b ; Shumailov et al., 2024a ) . If the model generates defective, biased, or erroneous samples, it will internalize these errors during fine-tuning, creating a negative feedback loop that degrades model performance ( Zhao et al., 2024b ; Wang et al., 2022 ) . Insufficient diversity in generated demonstrations also narrows the solution space and leads to pattern collapse during iterations. In fact, agents may get trapped in knowledge bubbles and repeatedly generate data, which reinforces their existing biases and traits instead of generating genuine new capabilities ( Wu et al., 2025c ; Yuan et al., 2025 ) .

To make these inherent generation loops more stable and reliable, several safeguards have been proposed. A simple and effective safeguard is to retain artificially generated benchmark data and accumulate generated demonstration data on top of it, which can prevent collapse under repeated training ( Gerstgrasser et al., 2024 ) . Another work strengthens quality control by using more robust checkers to verify generated samples. These checkers include external validators specifically designed for model-generated content ( Yi et al., 2026 ) and formal systems for inference verification, such as theorem provers ( Leang et al., 2025 ) . Complementary data-centric safeguards focus on selecting and filtering the training corpus to remove low-quality or harmful examples. To address diversity loss and pattern collapse, diversity-aware pooling expansion and selection mechanisms have been shown to maintain exploratory nature during the iterative process ( Qin et al., 2025 ) . Furthermore, shifting from large-scale offline generation to proactive, uncertainty-driven generation during testing naturally mitigates the accumulation of redundant data by focusing computational resources only on out-of-distribution or challenging samples ( Acikgoz et al., 2025 ) . Finally, because automated evaluators and validation processes can be unreliable or drift over time, recent research emphasizes human auditing and human-computer interaction to improve evaluation criteria as practical safeguards ( Dietz et al., 2025 ; Do et al., 2025 ) .

### 5.2 Intrinsic Evaluative Feedback

Just as massive parameter scales demand vast quantities of demonstration data, aligning and refining these capabilities requires high-quality supervisory signals. The bottleneck in traditional supervisory signal acquisition lies in the high cost of manually labeled preferences and human evaluation. To overcome this limitation, the methods in this category drive a paradigm shift by reframing the acquisition of supervision as an internal evaluative process. In this regime, FMs act not only as the output generator and the self-evaluator, but ultimately as the cognitive learner that internalizes its own judgments. Leveraging their language-native capabilities to follow rubrics, compare alternatives, and articulate reasoning, FM-based agents can autonomously produce intrinsic evaluative feedback , such as scalar scores, preference pairs, consistency signals, and natural language critiques, without requiring new environment interaction. Unlike intrinsic generative demonstrations (Section 5.1 ) that provide examples, or extrinsic exploratory experience (Section 5.3 ) derived from environment-grounded outcomes, the dominant learning signal here is the agent’s endogenous judgment of its own candidate behaviors. By treating the agent as a critic of itself, this paradigm shifts the bottleneck of parameter updates to the design of robust internal evaluation rubrics and critique mechanisms.

Formally, let 𝒜 t = ( θ t , Σ t ) \mathcal{A}_{t}=(\theta_{t},\Sigma_{t}) denote the current agent configuration. Given an input or task context x x , the agent samples a set of candidate outputs 𝒴 t ( x ) = { y t ( 1 ) , … , y t ( K ) } , y t ( k ) ∼ π θ t , Σ t ( ⋅ ∣ x ) . \mathcal{Y}_{t}(x)=\{y_{t}^{(1)},\ldots,y_{t}^{(K)}\},\qquad y_{t}^{(k)}\sim\pi_{\theta_{t},\Sigma_{t}}(\cdot\mid x). (12) An intrinsic evaluator ϕ t \phi_{t} , which may be implemented by the current model, an auxiliary judge model, a learned reward model, or a fixed scaffolded critique procedure, maps the task, candidates, and evaluation criteria κ t \kappa_{t} into an evaluative signal: e t = ϕ t ​ ( x , 𝒴 t ​ ( x ) , κ t ) . e_{t}=\phi_{t}(x,\mathcal{Y}_{t}(x);\kappa_{t}). (13) Here, ϕ t \phi_{t} is the source of feedback rather than the target of the update in this subsection. When an auxiliary evaluator is itself trained, we treat it as part of the feedback-construction pipeline; the method is classified here because the resulting evaluative signal is used to update the foundation model. The signal e t e_{t} may instantiate a scalar reward r t r_{t} , a preference relation y + ≻ y − y^{+}\succ y^{-} , a confidence or uncertainty score, a textual critique c t c_{t} , or a refined target y t ∗ y_{t}^{\ast} . Under foundation-model improvement, this feedback becomes the update signal, 𝒮 t ≈ e t \mathcal{S}_{t}\approx e_{t} , and the parameter update is written as θ t + 1 = 𝙸𝙼𝙿𝚁𝙾𝚅𝙴 θ ( θ 1 : t ; e t ) , Σ t + 1 = Σ t . \theta_{t+1}=\operatorname{\mathtt{IMPROVE}}_{\theta}(\theta_{1:t};e_{t}),\qquad\Sigma_{t+1}=\Sigma_{t}. (14) Depending on the form of e t e_{t} , the update may be implemented through reinforcement learning, preference optimization, reward-model training, critique-conditioned fine-tuning, or supervised fine-tuning on revised outputs.

##### Rubric feedback.

A first family of methods derives evaluative feedback by asking a model to judge candidate outputs against explicit criteria. The criteria may be task instructions, grading rubrics, safety principles, constitutional rules, or domain-specific preferences. In this setting, the evaluation standard κ t \kappa_{t} serves as the context for judgment: candidate outputs are scored, ranked, or compared according to how well they satisfy the stated criteria. The evaluator may also produce a brief rationale for its judgment, but the primary learning signal is the score, ranking, or preference relation rather than an open-ended revision.

Constitutional AI is a representative early example of this pattern. Instead of relying only on direct human preference labels, the model critiques and ranks outputs according to a set of written principles, producing AI feedback that can be used to train a preference model and subsequently optimize the policy through reinforcement learning ( Bai et al., 2022 ) . Recent self-improvement methods extend this judge-based loop more directly into parameter-updating systems. Meta-Rewarding trains a model to act, judge its own responses, and further evaluate its own judgments, turning both judgments and meta-judgments into preference pairs for iterative alignment improvement ( Wu et al., 2025b ) . Simonds et al. (2025) show that LLM judges can provide reward signals without reference solutions, enabling reinforcement learning from model-generated judgments in reasoning domains where programmatic rewards are difficult to specify. Self-Evolved Reward Learning further studies a learned reward model that labels additional data for its own improvement and supports reinforcement learning from self-feedback ( Huang et al., 2025a ) . Together, these methods illustrate how explicit principles, rubrics, or judge prompts can transform natural-language evaluation standards into scalable supervision for foundation-model improvement. The main advantage is flexibility, since the same model-based evaluator can be conditioned to assess helpfulness, harmlessness, factuality, reasoning quality, or format compliance. The main limitation is reliability, since ambiguous criteria or biased judges may reward superficial compliance rather than genuine improvement.

##### Consistency feedback.

A second family of methods exploits the stochastic behavior of foundation models to derive feedback signals for self-improvement. When ground-truth labels or reliable external verifiers are unavailable, the agent can generate multiple candidate solutions for the same task and use agreement among them as an intrinsic signal. The underlying assumption is not that consistency guarantees correctness, but that agreement, entropy, or self-certainty can provide a useful weak signal for ranking or weighting candidates.

Concretely, the agent samples 𝒴 t ​ ( x ) = { y t ( 1 ) , … , y t ( K ) } \mathcal{Y}_{t}(x)=\{y_{t}^{(1)},\ldots,y_{t}^{(K)}\} and applies an aggregation operator C C to produce a confidence or reward-like signal: e t = C ⁡ ( 𝒴 t ​ ( x ) ) . e_{t}=C(\mathcal{Y}_{t}(x)). (15) For example, majority voting can identify a consensus answer, predictive entropy can be used as a confidence score, and agreement among independent reasoning paths can be used to construct preference or reward signals. TTRL ( Zuo et al., 2025 ) uses majority voting over multiple generated answers at test time to produce reward signals for reinforcement learning. SRT ( Shafayat et al., 2025 ) similarly investigates whether self-consistency can support self-improvement without ground-truth labels. EMPO ( Zhang et al., 2025h ) encourages reasoning behavior through entropy-based signals, while INTUITOR ( Zhao et al., 2025c ) uses self-certainty as an intrinsic reward. In these methods, the agent does not primarily learn from newly generated demonstrations; rather, it learns from evaluative structure extracted from its own distribution over answers.

The strength of this family is that it can be applied broadly, including in domains where labels are expensive and formal verification is unavailable. However, consistency is only a proxy for correctness. If a model is systematically biased or confidently wrong, repeated sampling may amplify the same error. Confidence-derived feedback also depends on calibration: a model’s expressed certainty may not align with actual correctness. These issues make consistency-based self-improvement methods, while useful, quite fragile, especially when model errors are correlated across different samples. ( Tan et al., 2025a ; Feng et al., 2025 ; Till et al., 2025 ; Hamidieh et al., 2026 ) .

##### Corrective feedback.

A third family of methods treats natural-language critique or modification itself as the core evaluation outcome. Unlike rubric-based judging (whose main output is a score or preference under a stated criterion), error-correcting feedback methods require the model to identify specific flaws in candidate outputs and propose modifications. Therefore, this feedback is corrective, not merely comparative.