HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 2/7

Cleaned-text code points [15310, 31254). Bibliography omitted here; source.md includes it.

The creator starts from its own frozen RQ1 code harness H 0 H_{0} . During development, it receives results from a fixed 100-task SWE-Pro feedback set and all 89 Terminal-Bench tasks. The 100 SWE-Pro tasks are a subset of the 731-instance public split used in Creation, and the 630-instance held-out split of Section 4.3 is drawn from the same split. In the reported protocol, the controller first evaluates H 0 H_{0} on both benchmarks. Each official post- H 0 H_{0} candidate is then frozen and submitted as a pair: one complete 100-task SWE-Pro evaluation and one complete 89-task Terminal-Bench evaluation of the same commit. A candidate enters the official trajectory only after both legs settle. Same-commit infrastructure repairs are merged; probes, partial legs, stopped runs, and invalid instances are excluded.

The controller provides a budget of ten post- H 0 H_{0} full-evaluation pairs. Between two charged pairs, the creator may use at most two fixed-subset probes, each covering the same first five tasks from both benchmarks. Probe results are diagnostic and never become official scores. The creator terminates by declaring a non- H 0 H_{0} commit that has a complete official pair.

Both benchmarks shown during Evolution are feedback-bearing development sets, so in-trajectory scores measure online adaptation and version selection. Generalization is measured separately after freezing: every official version is additionally evaluated on 630 SWE-Pro instances disjoint from the feedback set, and these scores are never shown to the creator. Throughout this paper, held-out means withheld from the creator’s development loop; Section 4.3 gives the full setting.

### 3.3 Domains and downstream benchmarks

Creation covers four domains and five downstream benchmarks (Table 2 ); Evolution currently focuses on code harnesses. Together, the suites contain 2,207 unique downstream instances. The Evolution feedback tasks come from the same benchmark suites and are therefore not counted again.

Mature open-source systems define the capability surface in each domain. Depending on availability, a system may serve as a human-engineered reference or provide a development environment. These roles are assigned separately; inclusion does not imply that a system serves both. Appendix A lists the candidate systems, while the benchmark release fixes their roles, versions, and licenses.

\FloatBarrier

### 3.4 Evaluation protocol

Every score is produced by a frozen harness in a standardized runtime. The executor LLM L E L_{E} and evaluator J J remain fixed within each comparison, so score changes reflect changes to the harness. Development and evaluation are also separated: hidden scores are not returned in Creation, and Evolution exposes only its designated feedback set during development.

#### Creation.

We compare a created harness with the common seed and, where available, a mature human-engineered harness. Self-Eval sets L E = L C L_{E}=L_{C} and measures the complete creator–harness system. Unified-Eval runs every generated harness with the same fixed L E L_{E} , making harnesses directly comparable.

#### Evolution.

Evolution candidates are evaluated on the designated feedback benchmarks during development. Only complete two-benchmark pairs enter the official trajectory, and the creator selects a final paired candidate. After all trajectories end, every official version is additionally evaluated on a disjoint held-out set that is never shown to the creator, so adaptation to observed feedback and held-out generalization are reported separately. Appendix D details the evaluation settings and model roles.

#### Constraint compliance.

The creator-visible specification states what a submitted harness may not do: hard-code instance-specific solutions, derive patches from task identifiers, file-name allowlists, or known answers, consult hidden tests, hidden answers, hidden patches, private scorer internals, or official evaluation feedback, or replace the provided provider-neutral runtime interface with its own LLM access path. Two properties make these constraints checkable rather than advisory. First, the score path is isolated from the harness: a harness’s self-reported status is never a scoring input, SWE-Pro credit comes only from the real repository diff left in the task workdir, and Terminal-Bench credit only from final environment state, so no harness can earn score by asserting success. Second, every run retains its trajectory, result, and metric artifacts alongside the frozen harness source, which supports a post-hoc audit of the delivered code and of what that code actually executed. We audited the delivered harness source and the recorded execution artifacts of every run reported in this paper, and report the outcome as a null result: no harness obtained score through a prohibited route, and no run is excluded on these grounds.

### 3.5 Metrics

For each frozen harness, we report two quantities: downstream task performance under the benchmark’s native metric and the executor-model tokens consumed during evaluation. Execution cost is reported as both the total and the mean per task; tokens used by the creator to build or modify the harness are excluded. Creation compares each harness with the weak seed and available human-engineered references. Evolution reports the performance change from its initial harness H 0 H_{0} under the same executor and scorer.

## 4 Experiments

We instantiate the benchmark defined in Section 3 and report results for Creation, Evolution, and cross-model behavior. The behavioral analysis compares resource use, response to feedback, and transfer across executors.

### 4.1 Experimental setup

We evaluate six creator LLMs L C L_{C} : Opus 4.8 ( Anthropic, 2026 ) , GPT-5.5 ( OpenAI, 2026a ) , Gemini 3.1 Pro ( Google, 2026 ) , DeepSeek V4 Pro ( DeepSeek-AI, 2026 ) , Qwen 3.7 Max ( Alibaba Group, 2026 ) , and Seed 2.0 Pro ( ByteDance Seed, 2026 ) . Models run through their official APIs or OpenRouter ( OpenRouter, 2026 ) . We use Claude Code 2.1.177 as the development environment D D , except that GPT-5.5 uses Codex 0.144.3. Appendix B gives the model, decoding, and development-environment configuration, the downstream execution resources and time limits, and the human-reference sources.

We follow the development and evaluation protocols in Sections 3.2 – 3.4 . For RQ1, we independently create and evaluate three harnesses for each creator–benchmark pair and report avg@3. Under Self-Eval , the creator also serves as executor; under Unified-Eval , every harness uses Gemini 3.1 Pro, which isolates executor compatibility. Human-reference results are verified public system results rather than paired controls under one executor; their sources are listed in Appendix B.2 .

### 4.2 Harness Creation

RQ1 asks whether a model can turn the runnable weak seed into an effective harness for a task family. We compare the generated harnesses with the zero-scoring seed and with verified human-engineered systems, reporting both downstream performance and execution tokens.

#### Overall findings.

Creation quality varies substantially across task families (Tables 3 – 4 and Figure 4 ). Under Self-Eval , Opus 4.8 has the highest overall score (67.8), but remains below the human-engineered reference (86.2). Writing harnesses approach the reference, whereas Search shows the largest gap and Code also remains behind. Opus 4.8 and Gemini 3.1 Pro lead MLE-bench with medal rates of 32.9 and 32.4. More broadly, 77.8 % 77.8\% of failed Data tasks are attributed to harness defects, showing that the bottleneck is not only executor capability.

#### Performance variation and executor dependence.

Independent creations from the same model can still differ sharply. The clearest example is an Opus Code harness that performs well under Self-Eval but nearly collapses under Gemini because it hard-codes a 120-step limit around the original executor. Similar failures arise from overly strict stopping rules in Data. A single generated harness is therefore not representative, which motivates reporting avg@3.

Fixing the executor changes the ranking substantially. Qwen, Seed, and DeepSeek improve in several Data and Search settings under Gemini, whereas Opus and GPT-5.5 are often stronger with their own executors. Thus, Self-Eval reflects harness design, executor capability, and the compatibility between them. Cost is similarly uneven: MLE-bench token use varies by about nineteen-fold, yet higher cost does not reliably produce a higher score (Figure 9 ).

#### Implementation behavior.

The six creators follow distinct implementation strategies. Opus often rewrites the execution stack; GPT-5.5 adds a large monolithic agent; DeepSeek, Qwen, and Seed extend the seed with agent, tool, context, and state modules; and Gemini mostly edits the runner in place. The 18 Code artifacts add 17,111 net lines in total (Table 5 ), but edit size does not predict performance. Gemini adds the fewest lines (1,006) yet obtains the best Terminal-Bench score (68.8), suggesting that focused changes and frequent verification matter more than code volume.

All 18 Code harnesses implement an explicit execution loop; tools, lifecycle control, and verification are complete in 13/18, 13/18, and 15/18 artifacts, respectively (Figure 5 ). State and memory are the clearest gap: 11/18 artifacts define a State class, but only one exposes a state-saving interface and only one implements periodic checkpointing. No checkpoint event appears in 26,679 recorded task trajectories. Another recurring weakness is executor-specific configuration: hard-coded step or output limits can make a functional harness fail when the executor changes. Validation is also mostly syntactic; 441 of 2,325 executed Data tasks produce degenerate submissions that no harness detects.

Some generated mechanisms never affect execution. Of 108 component instances in Code, 72 trigger in real runs, 18 have only partial evidence, and 18 are never observed; all unobserved instances concern state and memory. The same pattern appears beyond Code: 124 of 587 Writing features are confirmed dead code, and 36 Data mechanisms sit on dead paths. Small implementation errors can also disable an entire tool chain. Self-test count alone is a weak signal: its Spearman correlation with downstream score is only 0.13–0.26 and is not significant, whereas revision calls reach 0.57 ( p ≤ .0005 p\leq.0005 ). Testing helps when the creator reads the failure, makes a targeted change, and re-verifies it.

#### Executor transfer.

Portability depends on the individual harness (Figure 6 ). Several Qwen and DeepSeek harnesses improve under Gemini, indicating that their original executors were a bottleneck; Qwen gains 17.6 points on BrowseComp and 12.9 on MLE-bench. Opus shows the opposite pattern: its Self-Eval SWE-Pro score falls from 69.3 to 33.0 under Gemini, and its Writing score falls from 84.6 to 74.2. In the Opus Search harness, the duplicate-query rate rises from 10.1 % 10.1\% to 88.2 % 88.2\% after the executor changes, showing that its deduplication, review, and termination rules are adapted to the original model. A runnable harness can therefore be used by another model, but capability transfers only when its prompts, tool protocol, budgets, and stopping rules remain compatible.

\FloatBarrier

### 4.3 Harness Evolution

#### Overall findings.

All five self-runtime creators improve on the visible feedback pair, but the gains shrink on held-out tasks. Opus 4.8 has the largest held-out improvement at + 4.44 +4.44 points. Transfer is weaker under the fixed Gemini executor: only Opus improves on held-out tasks, while the other three lineages regress. Evolution can therefore produce useful local changes, but the gains remain small and often specialize to the current executor or feedback set.

#### Evaluation protocol.

RQ2 asks whether a creator can improve its RQ1 Code harness using downstream execution feedback. Tasks repeatedly evaluated during Evolution form the feedback set ; tasks evaluated only after Evolution, with results never returned to the creator, form the held-out set . We report the individual benchmark scores and an equally weighted pair score: P ¯ t = 1 2 ​ ( P t SWE100 + P t Term89 ) , \bar{P}_{t}=\tfrac{1}{2}\left(P^{\mathrm{SWE100}}_{t}+P^{\mathrm{Term89}}_{t}\right), (2) where the two terms are percentage scores and t t indexes the frozen versions that completed a formal evaluation.

Each lineage starts from its RQ1 harness H 0 H_{0} . We run five self-runtime trajectories and four fixed-Gemini ablations, using the same creator, development environment, and starting harness. As defined in Section 3.2 , an official version must complete both the 100-task SWE-Pro and 89-task Terminal-Bench evaluations; probes are diagnostic only. The nine lineages produce 73 official versions and 64 adjacent version switches; Figure 7 shows every feedback-set trajectory.

After all trajectories end, we evaluate every official version on the 630-instance SWE-Pro held-out split, drawn from the public-split instances disjoint from the 100-task feedback set (Section 3.2 ). These scores are never shown to the creator and cannot affect editing, stopping, or final version selection. Table 6 therefore separates visible feedback gains from held-out generalization, and Figure 8 overlays the two trajectories for every lineage.

#### Editing and feedback use.

The median declared version changes eight files, adding 476 lines and deleting 38 (Table 7 ). Across 64 official switches, 58 change execution or control flow, 37 change tools, 17 change lifecycle recovery, 16 change context, and only four change state; no switch modifies a standalone verifier. Edit size does not reliably predict improvement, and the same creator may adopt different strategies under different executors. DeepSeek, for example, expands its self-runtime harness but later rolls back much of a fixed-Gemini rewrite after context compression breaks tool-message pairing. Evolution therefore resembles local program search around runtime feedback, where deletion can be as useful as addition.

Eight of the nine lineages complete at least one full loop from reading results to editing, re-evaluation, and version selection. Failure diagnosis remains the weakest step: the dedicated trajectory interface is called only twice, and explicitly inspected cases cover just 0.5 % 0.5\% – 40.2 % 40.2\% of the 189 feedback tasks, depending on the lineage. Creators instead rely on custom scripts and small probes, both of which can disagree with the full evaluation; one GPT-5.5 candidate passes all five Terminal probes but scores only 0.584 on the full set. Opus gives the clearest positive example: it finds that 99 of 100 runs report success while only 48 pass, traces the gap to premature completion, and adds a completion check. Feedback is most useful when it exposes a concrete failure mode and the resulting change is verified end to end.

#### Stability and final-version selection.

Evolution is not monotonic. Of the 64 official switches, eight regress on both benchmarks, 16 show a single-benchmark regression, three show a cross-benchmark trade-off, seven produce no measurable change, 27 report gains that remain inside the repeated-run noise band, two have clear positive evidence beyond the noise band, and one contains no executable code change. The same commit can vary by about ± 4.75 \pm 4.75 pair-score points, so small gains cannot be attributed to code changes from score alone. Added code is not necessarily active either: of 169 new functions or classes, 113 are reachable from the entry point, 31 are reachable only through dead code, and 25 have no caller. Opus’s completion gate is a positive example with path and case-level evidence; Qwen’s message sanitizer is the opposite, breaking valid Gemini tool-result sequences.