Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering — part 10/11

Cleaned-text code points [140229, 155402). Bibliography omitted here; source.md includes it.

The result is not a zero-sum contest between model intelligence and infrastructure intelligence. It is a systems-partitioning problem. Strong harnesses externalize the burdens that benefit from persistence, reuse, and control while leaving stable, fast, and generic competencies inside the model. The optimal partition is not static: as models grow more capable and as externalized infrastructure matures, the boundary will continue to shift—a dynamic explored further in Section 8.1 .

## 8 Future Discussion

The preceding sections examined how memory, skills, and protocols externalize distinct cognitive burdens, and how the harness unifies them into a working agent. Those analyses describe what has already been externalized. This section asks what comes next, following the logic of externalization itself through six connected questions:

• Where is the boundary between parametric and externalized capability heading, and how does multi-modal perception widen that frontier?

• Does the same logic extend from digital agents to embodied systems?

• How can the externalization process become more autonomous?

• What costs and risks accumulate as more is moved outward?

• How do externalized artifacts reshape interaction at ecosystem scale?

• How should the quality of externalization be measured?

The following subsections take up these questions in turn, moving from the shifting boundary of externalization through its embodied extension to the problem of how its benefits and costs should be assessed.

### 8.1 The Expanding Frontier

A recurring lesson of the preceding sections is that the boundary between what stays inside the model and what gets externalized is not fixed. It shifts as models, tasks, and infrastructure co-evolve. Understanding that boundary—and anticipating where it will move next—is therefore a central design question for agent systems.

In one direction, model improvement can pull capability back inward. A model that reliably produces structured output needs less format validation in the harness; one with a larger effective context window may tolerate simpler memory architectures; one with stronger intrinsic tool-use ability may require less elaborate intent-capture logic. Each such advance renders some piece of external infrastructure redundant. In the opposite direction, richer harnesses create new demands on models: operating inside a structured runtime requires respecting schemas, cooperating with permission checks, and coordinating with staged context injection ( Zhang et al., 2025d ; Cheng et al., 2024 ) . The frontier therefore moves in both directions at once, and a central engineering challenge is knowing when to externalize further and when to retract.

Within this shifting landscape, several classes of cognitive work that today remain largely implicit are plausible candidates for further externalization.

##### Planning and goal management.

Current agents typically generate plans through in-context reasoning, producing decompositions that are ephemeral—they exist only in the active generation and are lost once the context resets. Early agent frameworks such as BabyAGI already experimented with persistent task queues ( Nakajima, 2023 ) , and file-centric state abstractions like InfiAgent materialize planning artifacts outside the prompt ( Yu et al., 2026 ) . The direction points toward plans as first-class harness objects: persistent, inspectable, revisable, and shareable across agents or between agents and humans. That would convert planning from a transient reasoning act into a managed state artifact—the same representational shift that memory already performs for historical context.

##### Evaluation and verification.

Most evaluation logic today lives either inside the model’s chain of thought or in external benchmark harnesses that run post hoc. Externalizing evaluation criteria, rubrics, and verification procedures as runtime harness components—rather than leaving them implicit in model judgment—would allow the agent to check its own outputs against explicit standards during execution. Early signs of this direction are visible in verifiability-first engineering frameworks ( Zhu and Lu, 2026 ) and in self-refine loops that separate generation from critique ( Madaan et al., 2023 ) . The broader opportunity is to treat evaluation as externalized quality infrastructure rather than as a post-hoc measurement.

##### Orchestration logic itself.

The most recursive form of externalization is making the harness’s own configuration, policies, and execution logic into objects that the agent can inspect, critique, and revise. Once orchestration logic is externalized, the agent system can adapt not only what it knows and does, but how it organizes knowing and doing. This direction connects directly to the next subsection.

##### Multi-modal externalization.

The externalization framework developed so far assumes text as the dominant representational medium: memory stores textual traces, skills encode natural-language procedures, and protocols exchange structured text messages. As foundation models become natively multi-modal—processing images, video, audio, and screen content alongside text—each externalization dimension faces new design demands. Multi-modal skills must encode not only textual procedures but also visual perception workflows and cross-modal decision logic; early examples include computer-use skills that package GUI interaction sequences as reusable units ( Chen et al., 2026b ) . Multi-modal memory must index and retrieve visual and auditory experience, not only text-based episodic traces; MemVerse, for instance, maintains a multimodal knowledge graph that periodically distills fragmented sensory experience into more abstract representations ( Liu et al., 2025a ) , and MuSEAgent accumulates stateful multimodal experiences to inform future reasoning ( Wang et al., 2026d ) . Multi-modal reasoning distillation extends the skill-acquisition loop to non-textual modalities: TED demonstrates that successful multimodal reasoning trajectories can be distilled into reusable experience without additional training ( Yuan et al., 2026 ) . The broader implication is that multi-modal externalization is not simply a matter of adding new data types to existing stores. It changes the design assumptions of skill specification, memory indexing, and protocol schemas, and it opens a substantially wider frontier for the externalization of cognitive burden Wang et al. (2026e) ; Xu et al. (2026a) .

### 8.2 From Digital Agents to Embodied Externalization

The externalization framework developed in this paper applies to digital agents that read, write, and call APIs. A natural question is whether the same architectural logic extends to embodied systems—robots that must also perceive, move, and physically interact with the world. Recent developments in robot learning suggest that it does, and that the embodied domain is undergoing a decomposition strikingly parallel to the one analyzed here.

##### The monolithic starting point.

Early approaches to embodied intelligence pursued an end-to-end strategy analogous to the pre-externalization LLM agent. Vision-Language-Action (VLA) models ( Brohan et al., 2023 ; Kim et al., 2024 ) were positioned as monolithic “brains”: given a natural-language instruction and a visual observation, the model directly outputs a continuous action sequence, handling perception, reasoning, planning, and motor control within a single forward pass. This design mirrors the pattern in which early LLM agents attempted to manage memory, skills, and orchestration entirely through in-context reasoning—and it encountered the same category of limitations. Complex multi-step tasks exceeded the model’s planning horizon; failures in intermediate steps could not be diagnosed or recovered from; and the tight coupling of high-level cognition with low-latency motor control created irreconcilable requirements on inference speed and model capacity.

##### Decomposition: the cerebrum–cerebellum split.

The emerging architectural response recapitulates the externalization logic at the level of the whole body. A high-level robot agent —typically an LLM or multimodal model—assumes the role of cerebrum: it interprets goals, decomposes tasks into subtask sequences, maintains state across steps, handles exceptions, and revises plans when execution feedback indicates failure ( Ahn et al., 2022 ; Singh et al., 2023 ; Liang et al., 2023 ) . VLA models, meanwhile, are repositioned as a cerebellum : each one becomes a callable skill module responsible for a single atomic manipulation primitive—grasping, placing, pouring, inserting—executed with real-time sensorimotor feedback and low-latency control. The VLA no longer decides what to do; it ensures that how it is done is precise, stable, and adaptive to local physical perturbations.

This decomposition maps directly onto the externalization dimensions of the present paper. Task planning and goal management migrate from the VLA’s implicit parametric reasoning into an explicit, inspectable agent loop—precisely the shift from in-context planning to externalized plan objects discussed in Section 8.1 . Each VLA skill module functions as an externalized skill artifact: a reusable, composable unit with a defined interface, analogous to the skill files and tool specifications analyzed in Section 4 . The communication between agent and skill—structured action requests, execution status reports, error codes—constitutes a protocol layer that enables the agent to orchestrate heterogeneous motor capabilities without embedding their implementation details.

##### Why the parallel matters.

The convergence is not coincidental. Both digital and embodied agents face the same fundamental tension: a single model cannot simultaneously optimize for slow, deliberative cognition and fast, reactive execution. Externalization resolves this tension by routing each class of cognitive work to the substrate best suited for it—persistent, inspectable structures for planning and memory; specialized, low-latency modules for execution. In the digital case the execution modules are tool calls and code interpreters; in the embodied case they are visuomotor policies. The harness pattern—a runtime that loads context, dispatches skills, enforces protocols, and manages state—is equally applicable to both, suggesting that embodied and digital agent architectures may ultimately share not only a design philosophy but a concrete engineering stack.

##### Open challenges.

Embodied externalization introduces constraints that the digital case does not face. Physical actions are irreversible in ways that API calls are not: a dropped object cannot be “rolled back.” Real-time control demands latency budgets orders of magnitude tighter than text generation. Perception is noisy, and the gap between simulated training environments and physical deployment remains substantial. These constraints will shape how memory, skills, and protocols are designed for embodied harnesses, but they do not change the fundamental argument: the logic of externalization—decomposing monolithic capability into specialized, composable, and governable external structures—extends naturally from digital cognition to physical action.

### 8.3 Toward Self-Evolving Harnesses

Most current agent systems still rely on humans to revise memory policies, rewrite skill artifacts, and tighten execution logic after failures. If orchestration logic is itself externalized—as the previous subsection suggests—then the harness becomes an object that can be adapted programmatically rather than only manually. The question is how to make that adaptation reliable.

From a systems perspective, self-evolution can occur at three levels. At the module level , the architecture stays fixed but internal policies—retrieval granularity, skill-ranking heuristics, protocol-routing rules—are adjusted in response to observed failures. At the system level , the execution pipeline itself is restructured: scheduling strategies, execution order, or resource allocation may change when logs reveal recurring bottlenecks that local tuning cannot resolve. At the boundary level , the scope of the harness expands or contracts as models and tasks change, adding new externalized components where needed and pruning redundant ones—precisely the frontier dynamics discussed in Section 8.1 .

Several technical pathways are emerging. Reinforcement learning can optimize discrete runtime policies—search depth, compression ratio, retry strategy—against rewards such as task success, latency, or resource cost. Program synthesis treats harness adaptation as code repair: the model proposes patches after a failed trajectory, and sandboxed tests validate them before deployment. Evolutionary methods search over the topology of the harness—how modules are connected and in what order they are invoked. Imitation learning provides a stronger prior when exploration is too costly, by distilling execution logs from human experts or strong models into better orchestration patterns. These pathways target different search spaces—policy, program, structure, and prior experience—and are likely to be combined rather than used in isolation.

Self-evolution is attractive because it targets infrastructural failure modes directly, but it also amplifies the costs and risks discussed next: an adaptive harness that drifts without adequate governance can introduce new failure modes faster than it resolves old ones.

### 8.4 Costs, Risks, and Governance

As more cognitive burden is moved outward, two classes of cost accumulate: cognitive overhead from the externalized infrastructure itself, and security risks from the expanded attack surface.

##### Cognitive overhead.

Externalization is not free ( Wang et al., 2026b ) . Every additional memory layer, API schema, or safety rule imposes latency and reasoning overhead, and past a certain point the model spends more effort discovering, parsing, and coordinating modules than solving the task itself. In memory, over-retrieval floods the context with marginally relevant traces. In skills, verbose or overlapping files compete for context budget and can cause the model to follow local procedure while losing sight of the global objective. In protocols, tool sprawl turns action selection into an unnecessary disambiguation problem.

These failure modes suggest that the design target should be efficient and utility-positive rather than maximal externalization Liu et al. (2025c) . Minimal sufficiency asks whether a given module actually reduces the model’s cognitive burden or merely adds one. Lazy loading defers detailed guidance until the task structure requires it. Budget-aware routing treats context allocation as an explicit optimization variable, dynamically adjusting how much space is devoted to memory, skills, and protocol metadata as the task phase changes ( Zhang et al., 2026b ; Patel et al., 2025 ; Sui et al., 2026 ) . A good harness simplifies the model’s decision problem; it does not create a second one.

##### Security and integrity risks.