Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses — part 1/4

Cleaned-text code points [0, 15307). Bibliography omitted here; source.md includes it.

# Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

###### Abstract

Harnesses are now central to agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce A gentic H arness E ngineering ( AHE ), a closed loop that addresses these challenges through three matched observability pillars: ❶ component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; ❷ experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and ❸ decision observability pairs every edit with a self-declared prediction, later verified against the next round’s task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex (71.9%) and the self-evolving baselines ACE and Training-Free GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it achieves the highest aggregate success while using 12 % 12\% fewer tokens than the seed, and on Terminal-Bench 2 it yields + 5.1 +5.1 to + 10.1 +10.1 pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations further localize the gain to tools, middleware, and long-term memory rather than the system prompt. These results position observability-driven evolution as a practical pathway to keep coding-agent harnesses continually improving alongside their base models.

## 1 Introduction

Coding agents are increasingly deployed on long-horizon software-engineering tasks, with measurable progress on issue resolution over real-world code repositories [ 14 , 46 , 7 ] and multi-step terminal workflows [ 21 ] . In practice, such progress relies not only on the underlying language model, but equally on the surrounding engineering components: the system prompt that shapes work style, the tools that expose the file system and shell, and the middleware that controls context, execution, and recovery. This collection of model-external, editable components is collectively referred to as the agent’s harness [ 30 , 18 , 42 , 45 , 33 , 31 ] .

Harness design materially shifts task completion on long-horizon coding benchmarks, even with the base model held fixed [ 40 , 42 ] , making harness engineering a first-class lever for improving coding agents. Moreover, the optimal harness is model-specific: a harness tuned for one base model often underperforms on another and must be re-adapted as the base model changes. In current practice, this adaptation is performed manually—developers inspect trajectories, identify recurring failure patterns, and hand-craft edits across prompts, tools, middleware, and skills. Yet as base models advance rapidly [ 39 , 38 , 44 , 6 , 36 , 35 ] , this manual loop struggles to keep pace, creating a widening gap between model capability and the harness needed to realize it [ 33 ] .

An intuitive direction is to automate this loop with an evolution agent that optimizes harness components based on experience [ 1 , 49 , 4 ] . However, few existing approaches jointly evolve the full set of editable components [ 16 ] ; most focus on a single component, typically the prompt [ 32 , 50 , 20 ] , skills [ 19 , 43 ] , or an in-context playbook [ 49 ] . Jointly evolving multiple components end-to-end faces two structural obstacles: long, unstructured trajectories yield little actionable signal, and tightly coupled harness frameworks make edits beyond the prompt error-prone. This leaves the central question of agent-driven harness evolution open: How can an evolution agent jointly and stably evolve all editable components of a coding agent’s harness?

Our central insight is that this question is bottlenecked by observability , not by agent capability: once the evolution agent receives structured context over a clear action space, it can reliably converge on better harness designs [ 34 , 53 ] . We implement this in A gentic H arness E ngineering ( AHE , Figure 2 ), a closed loop driven by three observability pillars: ❶ component observability via a decoupled harness that exposes seven editable component types as files, so each failure pattern maps cleanly to a single component class; ❷ experience observability via a layered, drill-down evidence corpus distilled from millions of raw trajectory tokens, so the evolver consumes structured root causes rather than raw logs; and ❸ decision observability via a change manifest that pairs every edit with a self-declared prediction, later verified against the next round’s task-level outcomes, so each edit becomes a falsifiable contract and ineffective ones are reverted at file granularity.

We empirically validate AHE on Terminal-Bench 2 [ 21 ] : ten iterations lift pass@1 from 69.7% to 77.0%, surpassing the human-designed Codex [ 25 ] and the self-evolving baselines ACE [ 49 ] and Training-Free GRPO [ 4 ] . Without further evolution, the frozen harness transfers to SWE-bench-verified [ 14 ] , and across three alternate base-model families it yields consistent pass@1 gains of + 5.1 +5.1 to + 10.1 +10.1 pp, with the largest on bases further from saturation, suggesting that AHE encodes coordination patterns that less-saturated models lean on more heavily. A component ablation pinpoints where this gain lives: tools, middleware, and long-term memory each carry the improvement on their own, while the system prompt alone regresses, indicating that factual harness structure transfers across tasks and models whereas prose-level strategy does not.

This paper makes three contributions: • We formulate agent-driven harness evolution for coding agents and propose AHE , which identifies observability across components, trajectories, and decisions as the design pivot and turns every harness edit into a falsifiable, file-level contract through three observability pillars: a decoupled component substrate, a layered trajectory-distillation pipeline, and a change manifest whose self-declared predictions are verified by next-round task deltas.

• We empirically show that AHE lifts pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpasses human-designed and automated baselines, and produces a frozen harness that transfers across benchmarks and base-model families.

• Our analysis reveals two limits of agent-driven evolution: harness components interact non-additively, so stacking effective edits caps the aggregate gain; and the loop’s self-attribution is reliable for fixes but blind to regressions, pinpointing regression foresight as the clearest direction for future self-evolution loops.

## 2 Related Work

### 2.1 Harness Engineering and Evaluation for Coding Agents

Harness engineering refers to the practice of designing the system surrounding the model, including its tools, interfaces, memory, execution constraints, and feedback loops, which together shape what an agent can do on long-horizon tasks [ 30 , 18 , 40 , 3 , 33 , 31 ] . Concretely, the harness mediates how the model perceives and acts on its environment: it exposes the action and observation interfaces over which tool-augmented reasoning unfolds [ 3 ] , custom agent-computer interfaces for repository navigation, file editing, and command execution [ 45 ] , as well as sandboxed execution and orchestration support that keep long-horizon runs reproducible [ 42 ] .

Verifying that such systems actually help has driven the parallel maturation of coding-agent evaluation along two axes: task horizon and environmental realism. Coverage extends from short-horizon function-level benchmarks focused on contamination and freshness control [ 52 , 12 ] , through repository-scale executable patch resolution [ 14 , 46 , 7 ] , to multi-hour, terminal-driven workflows that exercise long-horizon, realistic execution [ 22 , 5 , 21 ] . A parallel infrastructure track packages executable runtimes and verifiers around these benchmarks [ 28 , 13 , 47 ] , whose attention to reproducible, traceable, and verifiable execution directly motivates the observation system AHE builds on.

### 2.2 Automated Optimization of LLM Agents

Approaches to automated agent optimization differ in what evidence the optimizer observes and what it can edit. Some revise the agent’s own outputs through episodic critique and reflection [ 20 , 32 , 9 ] . Others target prompts and instructions [ 15 ] : structured playbooks [ 49 ] , semantic-advantage priors [ 4 ] , jointly optimized instruction-demonstration pipelines for multi-stage programs [ 27 ] , and reflective updates driven by Pareto-frontier traces [ 1 ] . A separate line edits program structure itself, in the form of skill libraries [ 41 ] , scored program and agent archives evolved through mutation [ 24 , 11 ] , and graph-structured workflows searched or learned from rollouts [ 48 , 51 ] .

AHE tunes the full harness as a combinatorial whole rather than a single editable surface, so cross-component trade-offs become legible to the optimizer. It also keeps the human prior minimal, leaving methodology for the optimizer to discover from rollouts rather than fixing it by hand. We describe the substrate, trajectory analysis, and iteration that realize these choices in Section 3 .

## 3 Method

AHE turns harness optimization into a closed loop driven by another agent, with the base model held fixed and only the explicit harness edited. Our design principle is that every phase of this loop must be observable : AHE faithfully records the artifacts each phase produces (the harness components an iteration writes, the rollout trajectories it generates, the edit decisions it commits) and represents them in structured, layered forms that another agent can read and act on.

Three observability layers implement this principle. Component observability (§ 3.1 ) is realized by a decoupled, file-level harness substrate that maps each failure pattern to a single component class. Experience observability (§ 3.2 ) is realized by a layered evidence corpus distilled from raw rollouts and indexed for drill-down access. Decision observability (§ 3.3 ) is realized by a change manifest that pairs every edit with a self-declared prediction the next round verifies. The three layers compose into the iteration of Algorithm 1 , which runs unattended round after round.

### 3.1 NexAU: an editable, decoupled harness substrate

We instantiate the harness H H on the NexAU framework [ 23 , 37 ] , which exposes seven orthogonal component types as explicit files at fixed mount points in a single workspace: system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory. The component types are loosely coupled, so adding a middleware does not require editing the system prompt, and adding a skill does not require touching any tool.

This decoupling is what realizes component observability : each failure pattern maps to a single component class, giving the evolve agent a clean action space and localizing every pass-rate change to one file rather than scattering it across hundreds of lines of unstructured prompt prose. Each logical edit becomes one commit on the workspace’s git history, which yields file-level diffs and rollback granularity for free.

Our seed harness H 0 H_{0} is deliberately minimal: a single shell-execution tool, no middleware, no skills, no sub-agents. A seed already fitted to the target benchmark would contaminate every subsequent edit’s attribution, since we could not tell whether a gain came from the loop or from the seed. The minimal seed forces every component AHE adds to earn its place against measured rollouts.

### 3.2 Agent Debugger: layered trajectory evidence

We generate k k traces for each task in a benchmark using a harness H H , which may contain errors resulting from the deficiencies of the harness that can be acted on, but scattered across millions of tokens of raw messages. To extract insights from agent trajectories and enable experience observability , we apply Agent Debugger [ 17 ] framework to use an agent to explore trajectories framed as a navigable, file-based environment where each trajectory message lives in its own file and is reached through generic shell and scripting tools. Traces with the same query are placed in one environment, and the debugger is required to analyze the root cause of the failure or the success pattern, which is stored in per-task analysis report for each task. The analysis also includes pass/fail status of the task to ground the Evolve Agent. Finally, a benchmark-level overview is aggregated from every report into a single document as an entry point for every iteration.

In addition to these reports, we also provide original traces in case the agents need to verify the claims in the reports. The traces are provided both in raw form and lightly processed to remove unnecessary content. All of these content is provided as files allowing progressive disclosure [ 29 ] which saves on tokens and enable better agent decisions.

### 3.3 Evolve Agent: evidence-driven, auditable edits

The Evolve Agent closes the AHE loop. In each round it reads the layered evidence corpus produced by the Agent Debugger, decides which harness components to add, modify, or remove, applies those edits to the workspace, and records the reasoning behind every edit. Two constraints govern these edits, and together they realize decision observability : every edit becomes a falsifiable, file-level claim recorded in a versioned manifest, and the next round’s verdict either confirms or reverts it.

The first constraint is controllability : the Evolve Agent writes only inside the harness workspace, while the runs directory, tracer, verifier, and LLM configuration are read-only, and the seed system prompt (Appendix B.1 ) is marked non-deletable. These restrictions block the shortcuts an unconstrained self-modifier would take, such as disabling the verifier, swapping the model, or raising the reasoning budget, and keep every recorded gain attributable to harness edits.

The second constraint is that every change is evidence-driven and ships with a recorded prediction. Each edit attaches a manifest entry that names the failure evidence, the inferred root cause, the targeted fix, and a predicted impact comprising both expected fixes and at-risk regressions; this manifest is the loop’s evidence ledger (see Appendix B.2 ). In the next round, the loop intersects the predicted-fix and predicted-regression sets with the observed task-level deltas to produce a per-edit verdict. Each edit thereby becomes falsifiable by the next evaluation, which replaces rationale-driven self-justification with a measurable contract between rounds.