HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 3/7

Cleaned-text code points [31254, 47061). Bibliography omitted here; source.md includes it.

Creators usually select a version near the best visible feedback score, but that choice rarely matches the best held-out version. All five self-runtime declarations improve over H 0 H_{0} on held-out tasks, with gains of + 1.43 +1.43 to + 4.44 +4.44 points and a mean gain of + 3.11 +3.11 . Under fixed Gemini, however, only Opus improves and the other three regress. Across 64 comparable switches, feedback and held-out scores move in the same direction only 34 times ( 53.1 % 53.1\% ), and only 2/9 declared versions are held-out optimal. Visible feedback is therefore useful for local search but unreliable for final selection: repeatedly optimizing a noisy score can favor a lucky run and amplify overfitting.

\FloatBarrier

## 5 Related Work

HarnessDev lies at the intersection of agent benchmarking, autonomous agent construction, and harness evolution.

#### Agent benchmarks and harness development.

Benchmarks such as SWE-bench ( Jimenez et al., 2023 ) , GAIA ( Mialon et al., 2023 ) , WebArena ( Zhou et al., 2023 ) , τ \tau -bench ( Yao et al., 2024 ) , and AgentBench ( Liu et al., 2023 ) standardize tasks, environments, and scoring, but generally evaluate task execution under a selected harness. Harness-Bench ( Yao et al., 2026 ) instead measures how harness choice changes model performance. The Meta-Agent Challenge ( Lu et al., 2026 ) directly evaluates development: a meta-agent iteratively programs an agent artifact in a sandbox and is scored on protected held-out tests across five domains. It is closely related to Creation , while HarnessDev also studies continued development, separates creator and executor models, and measures execution cost.

HarnessOpt-Bench ( Ursekar et al., 2026b ) is the closest concurrent benchmark to Evolution . An LLM optimizer receives a seed harness, graded feedback, and a fixed evaluation budget; a trusted environment then scores its nominated candidate by normalized gain on an inaccessible test partition. Its focus is optimizing a provided harness, although its near-empty GAIA seed also requires construction. In contrast, HarnessDev connects from-scratch Creation to Evolution and evaluates the same frozen artifact under self and fixed runtime models. It uses in-trajectory scores as feedback, then tests every frozen version on a disjoint 630-task SWE-Pro set, separating adaptation from held-out generalization throughout the trajectory.

Evo-Bench ( Huang et al., 2026 ) also evaluates Evolution by asking evolver models to improve a shared CodeAct seed while holding the runtime model fixed. It scores each lineage’s final revision on a disjoint, sensitivity-calibrated multi-domain suite. HarnessDev instead connects Creation and Evolution, includes both self- and fixed-runtime views, measures execution-token cost, evaluates transfer to another executor, and scores every frozen version on its held-out split. Thus, Evo-Bench emphasizes cross-domain final-revision quality, whereas HarnessDev also studies how held-out performance changes along a feedback-driven trajectory.

#### Automated agent design and evolution.

Human-engineered systems such as Claude Code ( Anthropic, 2024 ) , Codex ( OpenAI, 2025 ) , OpenHands ( Wang et al., 2024 ) , and SWE-agent ( Yang et al., 2024 ) combine execution loops, tools, context management, and recovery mechanisms ( Rombaut, 2026 ) . ADAS ( Hu et al., 2024 ) , AFlow ( Zhang et al., 2024 ) , MASS ( Zhou et al., 2025 ) , and EvoAgentX ( Wang et al., 2025 ) search over prompts, workflows, operators, or agent topologies; other systems let models construct more of the agent in natural language or code ( Pan et al., 2026 ; Ning et al., 2026 ; Li et al., 2026 ) .

Recent systems optimize executable harness code directly. VeRO ( Ursekar et al., 2026a ) provides versioned snapshots, budget-controlled evaluation, and structured traces. Meta-Harness ( Lee et al., 2026b ) uses a coding-agent proposer that can inspect prior candidates, scores, and traces before proposing the next harness. Other methods improve prompts, workflows, memory, skills, tools, or code between episodes ( Shinn et al., 2023 ; Wang et al., 2023 ; Liu et al., 2025 ; Tao et al., 2026 ; He et al., 2025 ; Luo et al., 2026 ; Karten et al., 2026 ; Chen et al., 2026b ; Lin et al., 2026a ; Lee et al., 2026a ) . Self-Harness ( Zhang et al., 2026a ) derives model-specific edits from failures and applies regression testing; HarnessFix ( Chen et al., 2026a ) uses a provenance- and control-flow-aware representation for localized repair; DemoEvolve ( Che et al., 2026 ) uses demonstrations when reward feedback is unreliable; and HarnessCompass ( Zhang et al., 2026b ) addresses overfitting and component interference. Harness-R1 ( Shao et al., 2026 ) instead trains a harness engineer to convert failure batches into validated patches. These works propose methods or infrastructure for harness improvement; HarnessDev evaluates how well general-purpose frontier models perform the broader developer role under one Creation-and-Evolution protocol.

#### Evaluating harness evolution.

Final task gain can conflate informed diagnosis with blind search. Priority-ranking evaluation ( iunn Ong et al., 2026 ) tests whether an optimizer identifies the components most worth changing. Harness Updating Is Not Harness Benefit ( Lin et al., 2026b ) separates producing a useful update from an executor’s ability to exploit it, motivating our creator–executor separation and Self-Eval / Unified-Eval views. SEAGym ( Zheng et al., 2026 ) records intermediate snapshots, cost, and in- and out-of-distribution results, showing that later updates need not preserve held-out gains. Matched-budget studies likewise find that harness evolution can overfit its search benchmark and may not beat simpler search baselines ( Wang et al., 2026 ) . HarnessDev therefore freezes runnable artifacts, records development trajectories and execution cost, and evaluates transfer across runtime models. We treat its Evolution trajectories as adaptation to feedback-bearing sets and assess held-out generalization afterward on disjoint SWE-Pro tasks; matched-search evaluation remains future work.

## 6 Discussion, Limitations, and Conclusion

The results show why harness development should be evaluated directly. Creation performance varies sharply by domain: under Self-Eval , current models match the human reference in writing and exceed it in machine-learning experimentation, but remain far behind in search and research and still trail it in code. Cross-executor comparisons show that some harnesses improve under a stronger executor, while others exhibit creator co-adaptation. Evolution is harder still: useful intermediate updates are often erased by later changes, and more updates do not guarantee a positive final gain. Together, these findings separate the quality of the persistent execution system from the capability of the model running inside it. The fixed-Gemini Evolution ablation sharpens this point: changing only the runtime binding can substantially move H 0 H_{0} and alter which harness changes are useful.

### 6.1 Limitations

The four categories cover many but not all real deployments. Human baselines are uneven and not guaranteed optimal. Unified-Eval reduces but cannot fully remove executor-model differences, since harness–model interaction is complex. The behavioral comparisons are descriptive and are limited by incomplete benchmark coverage. Evolution currently has one trajectory per creator–runtime cell and one unfinished main-runtime cell, and its post-freeze held-out evaluation covers SWE-Pro only, so the trajectories do not support uncertainty estimates or population-level comparisons. The development environment D D is held fixed across both stages; whether an evolved harness can itself serve as the development environment for further evolution is left to future work. Finally, HarnessDev measures model-external learning and does not claim heuristic learning can replace parameter training.

### 6.2 Conclusion

HarnessDev moves agent evaluation from whether a model can solve tasks inside a fixed system to whether it can create and maintain the systems that solve future tasks . Through a four-category human baseline corpus, from-scratch creation tasks, feedback-driven evolution, Self- and Unified-Eval, and comparisons with human-engineered references, it makes agent-built execution harnesses a measurable object. If model weights are one place intelligence accumulates, the harness is another: explicit, inspectable, testable, reusable, and continually improvable through failure, feedback, and real engineering pressure.

#### Ethics statement.

HarnessDev is built from publicly available benchmark suites and open-source harnesses; no human-generated content is collected. Automated harness construction risks amplified insecure tool use, so the benchmark states explicit constraints on what a submitted harness may do and audits compliance after every run (Section 3.4 ); no violation was observed in this study, and we release the audit artifacts so the check can be repeated. Creation runs and downstream benchmark tasks execute in containers, but that boundary is provisioned for reproducibility rather than for containment; anyone reusing the generated harnesses should treat them as untrusted code and isolate them more strictly than we did.

## 7 Contributions

Core Contributors Yuhao Wu, Jingyuan Zhang, Jiajun Shi

Contributors Xinping Lei, Qingshui Gu, Yuxuan Zhang, Zexuan Wang, Chen He, Chen Huang, Maojia Song, Zhiyuan Zeng, Shaowen Wang, Jinkai Liu, Yunfeng Shi, Jiaheng Liu

Corresponding Authors Yuhao Wu () Shen Yan () Wenhao Huang () Ge Zhang () Wenxuan Zhang ()

## Appendix A Candidate Harness Systems

The benchmark draws candidate systems from four categories. A system may define the scope of a category, serve as a human-engineered counterpart, supply an accessible update history, or act as a development environment; inclusion below does not imply that every system serves every role. The final pinned set records these role assignments along with commit hashes and licenses where applicable and is released with the benchmark.

• Code agent: Claude Code, OpenCode, OpenHands, SWE-agent, mini-SWE-agent.

• Notebook / data-analysis: DataAgent, DB-GPT.

• Writing agent: AutoResearchClaw, webnovel-writer.

• Research / retrieval: Alibaba-NLP/DeepResearch, dzhng/deep-research, modelscope/ms-agent, gpt-researcher.

## Appendix B Experimental Configuration

Table 8 records the creator LLM, development environment, and decoding configuration used by the reported experiments. Sampling parameters follow each provider’s official defaults; all creators run at high reasoning effort with streaming enabled, and output length is set to the endpoint maximum. Participation is stage-specific, so the presence of a creator in this table does not imply complete coverage of every benchmark.

### B.1 Downstream execution configuration

Data-analysis (MLE-bench) downstream runs execute each task in an isolated container with one NVIDIA A800-SXM4-80GB GPU (80 GB of GPU memory), 14 vCPUs, and 227 GiB of RAM. Each task has a wall-clock limit of 36,000 s, split into a 34,200 s budget for the generated agent harness and a 1,800 s reserve for the fixed grader, together with a 500-step cap. Dataset download and preparation complete before the task clock starts and do not consume this budget. The RQ2 code benchmarks run each task with the same 500-step cap and a 7,200 s limit, as stated in the evolution contract of Appendix E .

### B.2 Human-engineered reference systems

For each downstream benchmark, we use the highest publicly available system-level result that we could verify from the benchmark’s official leaderboard or the corresponding system report. These references pair a human-engineered harness with the executor model used by that system; they are not scores obtained with one common executor. Table 9 records the exact pairs used in the main paper.

The three starred values in Table 3 are external reports rather than local reruns: SWE-Pro 80.0 for Claude Fable 5, Terminal-Bench 2.1 88.8 for GPT-5.6 Sol, and BrowseComp 92.2 for GPT-5.6 Sol. Each value is taken from the corresponding benchmark row in OpenAI’s official GPT-5.6 release report [ OpenAI, 2026b ] .

### B.3 Creation performance and execution cost

### B.4 Evolution execution cost by frozen version

Figure 10 complements the RQ2 score trajectories with executor-side usage across every frozen harness version. We keep task-agent/runtime usage separate from the tokens spent by the creator, judge, and diagnostic probes.

\FloatBarrier

## Appendix C Harness Interface Specification

Each admissible harness implements six functional modules:

{specbox} execution.py — run(task) -> Result step(state, observation) -> Action tools.py — register(toolspec) -> None call(name, **params) -> Observation context.py — build(task, history, state) -> Prompt compress(messages) -> Messages state.py — save(checkpoint) -> None load(id) -> State resume() -> State lifecycle.py — beforeAction(action) -> Action | Abort afterAction(action, result) -> None onFailure(error) -> Recovery onTimeout() -> Graceful evaluation.py — evaluate(result, criteria) -> Score recordTrajectory(step) -> None

All methods return JSON-serializable objects. The reference seed implementation, the audit script, and the held-out task splits will be released with the benchmark.

### C.1 Seed Harness Skeleton

The weak seed shared across Creation domains (Section 3.2 ) supplies only the runnable floor beneath the contract above: a stable CLI, runtime model configuration, audit writers, and policy-free primitives. Its concrete packaging may vary with the execution environment, but no variant contains a task-solving policy. Figures 2 and 3 in the main text give the domain-general structure and separate the shared audit contract from domain-specific final artifacts.

{specbox} seed workspace/ dev runner # public development-feedback runner harness/ # the package: python -m harness entry module # stable task/config/output CLI seed runner # parse inputs; optional summary probe audit contract # result.json · trajectory.jsonl · # response.md · stdout/stderr logs llm gateway # configured connectivity, no policy primitives/ # passive helpers, no policy paths # resolve · contains · info files # read · write · replace · json io search # list tree · glob · grep process # run command artifact io # create · validate · record paths

The unmodified seed performs one non-acting pass: it parses the task and environment, optionally probes the configured LLM with a summary-only prompt, and terminates with status partial after writing the audit envelope. It provides no execution loop, tool policy, context management, state or memory, failure recovery, or verifier. The creator must therefore implement these modules using only feedback from public development tasks; hidden tasks, answers, and official scores remain withheld.

The interface and honest-status semantics are shared, but the authoritative final artifact is domain-specific. Code requires real repository changes and a patch; data analysis requires a scorer-readable submission; writing requires final user-facing prose; and research/search requires a concise answer grounded in retrieved evidence. Each frozen evaluator reads the corresponding authoritative artifact. Producing only the common JSON and log files is therefore an incomplete execution and, as shown in Table 3 , the unmodified seed scores zero across all five downstream benchmarks.

## Appendix D Evaluation Settings and Roles in Full

This appendix gives the evaluation-setting and model-role definitions summarized in Section 3.4 .

### D.1 Self and unified evaluation