Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses — part 2/4

Cleaned-text code points [15307, 31201). Bibliography omitted here; source.md includes it.

Algorithm 1 composes the three substrates into one iteration: rollout, clean, attribute the prior manifest and revert rejected edits, distill, edit, commit. We run k ≥ 2 k\geq 2 rollouts per task so each task carries a pass-rate signal, which stabilizes pass@1 and lets partial-pass tasks anchor comparative diagnosis. Attribution runs before distillation, so its verdict lands inside the evidence corpus and binds each prior manifest entry as a contract rather than a rationale. A one-shot explore agent (Appendix B.3 ) runs in parallel with iteration 1 1 to seed a small number of reusable skills from the NexAU source and public coding-agent references. These skills receive no special protection: from iteration 2 2 onward the Evolve Agent may keep, refine, or remove them based on observed rollouts.

## 4 Experiments

We organize our empirical study around three questions: where AHE sits on the map of existing approaches to harness design, whether what it produces is portable beyond its optimization target, and what inside the loop drives the gain.

### 4.1 Setup

##### Evaluation.

We drive evolution on the full 89 tasks of Terminal-Bench 2 [ 21 ] , split as 4 easy, 55 medium, and 30 hard, with per-task timeout extended to 1 hour. For cross-benchmark transfer we evaluate the AHE harness on SWE-bench-verified [ 14 ] , 500 tasks across seven repositories. We report two metrics per configuration: pass@1 , the mean binary success rate over k k rollouts per task; and tokens/trial , the mean per-trial total of prompt plus completion tokens across all LLM calls, in thousands. Infrastructure-aborted or timed-out trials count as failures under pass@1 (matching the official terminal-bench leaderboard) and are excluded from token means to avoid truncated figures. Runtime infrastructure (framework, dispatcher, sandbox, tracer, and concurrency) is detailed in Appendix A .

##### Models.

For both the evolution loop and the main experiment of § 4.2 , all three role agents (the Code Agent, the Agent Debugger, and the Evolve Agent) share one base model, GPT-5.4 [ 26 ] at the high reasoning setting. For cross-model transfer (§ 4.3 ), we re-evaluate the Code Agent on five alternate bases: GPT-5.4 [ 26 ] at medium and xhigh reasoning, qwen-3.6-plus [ 38 , 44 ] , gemini-3.1-flash-lite-preview [ 8 ] , and deepseek-v4-flash [ 6 ] .

### 4.2 RQ1: Main Results

We run a single AHE campaign of ten iterations from the bash-only NexAU 0 seed (§ 3.1 ) on Terminal-Bench 2, finishing in roughly 32 hours; the best resulting configuration is reported as AHE . The two self-evolve baselines ACE [ 49 ] and Training-Free GRPO (TF-GRPO) [ 4 ] start from the same NexAU 0 seed.

##### AHE outperforms both human-designed and self-evolve baselines.

AHE outperforms every baseline on our panel: three human-designed harnesses, OpenCode [ 2 ] , Terminus-2 [ 10 ] , and Codex [ 25 ] , and the two self-evolve baselines. Figure 1 shows the gain accumulates across iterations, with continued evolution pushing pass@1 further above the NexAU 0 seed. By difficulty, the only exception is the Hard tier, where AHE marginally trails Codex. We trace this gap to interference between AHE’s components on long-horizon tasks rather than to a missing capability: swapping AHE’s long-term memory alone into the NexAU 0 seed, without the other AHE components, already surpasses Codex on Hard (§ 4.4.1 ).

##### Prompt-only self-evolution misses the components that carry AHE’s gain.

The gaps to ACE and TF-GRPO trace to a layer mismatch. ACE distills natural-language playbooks the agent reads in-context, and TF-GRPO is a trajectory-feedback variant of GRPO that reinforces successful tool sequences; neither method opens the surrounding scaffolding to edits. AHE instead jointly evolves system prompt, tools, middleware, and long-term memory, and § 4.4.1 shows the gain concentrates in the latter three components, exactly the layers ACE and TF-GRPO leave untouched.

### 4.3 RQ2: Transfer to Unseen Tasks and Base Models

AHE’s harness is evolved on Terminal-Bench 2 with GPT-5.4 high. We probe whether it encodes general coding-agent experience or overfits to that target by transferring the evolved harness unchanged, without further evolution, to two off-target settings: a different task surface (SWE-bench-verified) and three alternate base models.

##### Cross-benchmark transfer.

We evaluate the AHE harness on SWE-bench-verified against the seed and the two self-evolve baselines (NexAU 0 , ACE, TF-GRPO) under identical infrastructure (Table 2 ).

ACE and TF-GRPO both regress below the NexAU 0 seed in aggregate success while spending 11 % 11\% to 29 % 29\% more tokens than the seed: the playbook ACE injects and the trajectory distribution TF-GRPO reinforces were distilled on Terminal-Bench 2 traces and ride the prompt at every model call, so on a different task surface that text adds cost without reshaping the underlying policy.

AHE instead achieves the highest aggregate, with the seed-relative gain concentrating on django and sphinx-doc, the two largest and most token-expensive repositories whose multi-step edit-and-verify loop matches the structure AHE’s tools, middleware, and long-term memory compress on Terminal-Bench 2. Marginal regressions appear only on the three smallest repositories, consistent with pass@1 variance on small repos exceeding the per-repo gain. AHE also cuts aggregate tokens by 32 % 32\% against ACE, 21 % 21\% against TF-GRPO, and 12 % 12\% against the seed: encoding behavior in tools, middleware, and memory avoids the per-call re-derivation cost prompt-only baselines incur.

##### Cross-model transfer.

We re-evaluate both the NexAU 0 seed and AHE on the five alternate bases listed in § 4.1 . Figure 3 reports five positive pass@1 gains from + 2.3 +2.3 to + 10.1 +10.1 pp.

Cross-family gains dominate within-family ones: deepseek-v4-flash moves + 10.1 +10.1 pp from 51.7 % 51.7\% to 61.8 % 61.8\% , qwen-3.6-plus + 6.3 +6.3 pp from 56.2 % 56.2\% to 62.5 % 62.5\% , and gemini-3.1-flash-lite-preview + 5.1 +5.1 pp from 36.5 % 36.5\% to 41.6 % 41.6\% , all above the + 2.3 +2.3 pp on GPT-5.4 medium and xhigh. We read this as bases further from saturation leaning more on the coordination patterns AHE has fixed inside tools, middleware, and long-term memory, while a stronger base re-derives the same coordination from its prompt at low marginal cost.

Within one family the profile is non-monotone: + 2.3 +2.3 pp on medium, + 7.3 +7.3 pp on high from § 4.2 , and + 2.3 +2.3 pp on xhigh. AHE’s step budget and per-task timeout were fitted to GPT-5.4 high during evolution; medium has more time-per-step slack but loses a reasoning tier of raw capability, while xhigh pushes more trials past the per-task timeout, which our pass@1 convention (§ 4.1 ) counts as failures. Either direction discounts the gain.

The headline finding is that all five gains land positive: the AHE workspace is not specific to one provider’s idioms or one reasoning depth. Their magnitude tracks the evolution operating point rather than raw base capability, so we treat the timeout-budget coupling as a generalization hazard discussed in our Limitations section.

### 4.4 RQ3: Analysis

We analyze the loop along two architectural choices that § 3 places weight on: decomposed components (§ 4.4.1 ) and self-declared attribution (§ 4.4.2 ).

#### 4.4.1 RQ3a: where value accumulates across components

Table 3 decomposes the AHE gain at the component level by isolating one of the four evolved layers, long-term memory, tools, middleware, or system prompt, while holding the other three at their seed defaults. Three of the four single-component variants outperform the seed; the system-prompt swap is the only regression.

##### Each component owns a different failure surface.

Memory adds 12 boundary-case lessons (performance margin, queued-over-limit cancellation, evaluator-style closure, source-packaging layout); on Hard the lessons lift it above full AHE, while on Easy they reduce to superfluous re-verification. Tools become a 1364-line shell that auto-surfaces contract hints from files near each command; on Medium it lands within 0.9 0.9 pp of full AHE, while on Hard a built-in publish guard closes the loop too early. Middleware adds a finish-hook that forces one evaluator-isomorphic closure check; on Easy it clears every task, while on Hard it inflates turn count. The system prompt encodes 79 lines of universal discipline whose executability depends on the other three; inserted alone it scores − 2.3 -2.3 pp aggregate.

##### Components interact non-additively, capping the aggregate gain.

The three positive single-component gains sum to + 11.1 +11.1 pp against full AHE’s + 7.3 +7.3 pp, and on Hard the memory-only variant exceeds full AHE: memory, middleware, and the system prompt all push toward the same closure-style verification, so stacking them spends turns on redundant re-checks within the long-horizon budget. Since the evolve agent optimises an aggregate dominated by 55 Medium tasks, it converges to a Medium-heavy trade-off that returns part of the Hard memory effect, and we leave interaction-aware evolution to future work.

#### 4.4.2 RQ3b: how reliably the loop’s self-attribution tracks reality

Each evolution round, our evolve model produces a change manifest naming which Terminal-Bench 2 tasks it expects to fix in the next round and which it flags at risk of regression. We compare the round- N − 1 N{-}1 prediction against the round- N N ground truth, computing standard precision and recall over the 89 tasks separately for fixes and regressions.

##### Evidence-driven targeting.

The fix panel of Figure 4 shows the evolve model’s targeting is evidence-driven rather than guesswork. Cross-iteration fix-precision of 33.7% and fix-recall of 51.4% sit roughly 5x above the random-prediction baselines of 6.5% and 10.6%, so each harness edit lands on a real, agent-anticipated target rather than on an arbitrary subset of the panel.

##### Regression blindness.

The regression panel tells the opposite story: cross-iteration regression-precision of 11.8% and regression-recall of 11.1% sit only about 2x above their random baselines of 5.6% and 5.4%, so most upcoming regressions go unforeseen. The agent can justify why an edit should help, but it cannot reliably name the tasks the same edit is about to break, which is what produces the non-monotone steps in the evolution curve of § 4.2 . Closing this gap is the clearest direction for future self-evolution loops. Appendix D gives the per-round breakdown.

## 5 Conclusion

We introduced Agentic Harness Engineering (AHE) , an observability-driven loop that turns a coding agent’s harness into a learnable adaptation surface while the base model remains fixed. AHE exposes components as files, distills rollouts into a layered evidence corpus, and binds each edit to a falsifiable next-round prediction; ten iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, and the frozen harness transfers to SWE-bench-verified and three alternate model families. We see harness-level evolution as a complementary axis to model-side training: an externalized, auditable surface where coding-agent experience can accumulate.

## Limitations

This work studies a promising but high-variance setting, and the scope of our claims should be interpreted accordingly.

##### Benchmark scope.

Our evaluation drives evolution on Terminal-Bench 2 and probes transfer on SWE-bench-verified. Even though the frozen harness transfers to a second task surface and to three alternate base-model families, broader programming languages, repository-scale deployments, and human-in-the-loop workflows remain untested.

##### Evolution operating point.

AHE’s step budget and per-task timeout were fitted to GPT-5.4 high during evolution, so cross-model transfer numbers conflate harness portability with operating-point coupling—within one family the gain is non-monotone across reasoning tiers (§ 4.3 ). Untangling these factors will require re-running the loop under multiple operating points.

##### Self-modification governance.

AHE bounds edits to a workspace, attributes every change in a versioned manifest, and rolls back ineffective edits at file granularity, but it does not provide a complete guardrail stack. Long-horizon harness cleanup and stronger misuse prevention remain incomplete, and AHE should be viewed as a controlled research prototype rather than a fully mature autonomous self-improvement system.

## Appendix A Experimental Setup: Full Details

This appendix expands the condensed Setup in § 4.1 with the formal metric definitions and the runtime infrastructure.

##### Seed agent.

The seed configuration, denoted NexAU 0 , is a simple code agent built on the NexAU framework of § 3.1 that exposes only the bash tool to the model, with no skills, no middleware, and no long-term memory. Every iteration of the AHE outer loop edits this workspace, so all reported gains are measured against NexAU 0 as the common starting point.

##### Runtime infrastructure.

All runs use the NexAU framework of § 3.1 to instantiate the coding agent. Harbor dispatches tasks, isolates each rollout, and verifies pass/fail, with a 3600-second per-task timeout. Every rollout runs inside a fresh E2B remote sandbox, so shell side-effects cannot leak between tasks. InMemoryTracer records trajectories and mirrors them to Langfuse. The Agent Debugger executes at concurrency 16 with a 600-second per-task timeout.

##### Hyperparameters.

Table 4 consolidates the operating points of the four agents and the outer loop, taken from the configuration snapshot of the reference run.

##### Terminal-bench difficulty labels.

The official terminal-bench-2 leaderboard 0 0 0 https://www.tbench.ai/benchmarks/terminal-bench-2 partitions the 89-task subset into 4 easy, 55 medium, and 30 hard tasks.

##### pass@1.

For a configuration on a task set D D with k k rollouts per task, let r i , j ∈ { 0 , 1 } r_{i,j}\in\{0,1\} denote the binary reward of rollout j j on task i i . The pass@1 score is the mean pass ​ @ ​ 1 = 1 k ​ | D | ​ ∑ i = 1 | D | ∑ j = 1 k r i , j . \mathrm{pass@1}=\frac{1}{k|D|}\sum_{i=1}^{|D|}\sum_{j=1}^{k}r_{i,j}. (1) Trials that terminate on an infrastructure exception, such as a sandbox crash or API timeout, contribute r = 0 r=0 rather than being dropped, a strictly harsher convention than discarding failures that keeps our numbers comparable to the official terminal-bench leaderboard. The rollout count k k varies across experiments; each table states it explicitly.

##### Token cost and Succ/Mtok.

For token cost we count every LLM call as prompt plus completion across the rollout and report the mean over completed trials in thousands, denoted Tokens k; infrastructure-aborted trials are excluded to avoid truncated figures. To compare configurations that trade accuracy for cost we combine the two via Succ / Mtok = pass ​ @ ​ 1 × 10 6 mean ​ tokens ​ per ​ trial , \mathrm{Succ/Mtok}=\frac{\mathrm{pass@1}\times 10^{6}}{\mathrm{mean\ tokens\ per\ trial}}, (2) the expected number of successes per million tokens. The main paper reports pass@1 and Tokens k separately so each axis stays legible; Table 5 folds them into Succ/Mtok per repository on SWE-bench-verified, derived from the pass@1 and Tokens k columns of Table 2 .

## Appendix B Prompts and Configurations

This appendix gathers the prompts that drive the AHE outer loop together with the seed code agent’s system prompt. The five blocks below reproduce the literal contents of the corresponding files in the public repository at https://github.com/china-qijizhifeng/agentic-harness-engineering as of the commit that produced the experiments in Section 4 . Jinja-style {{ var }} placeholders are filled in by the harness at runtime.

### B.1 Code Agent Seed System Prompt