HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 1/7

Cleaned-text code points [0, 15310). Bibliography omitted here; source.md includes it.

\newtcolorbox specbox enhanced, breakable, colback=specbg, colframe=medgray, boxrule=0.5pt, arc=1.5mm, borderline west=2pt0ptseedaccent, left=3mm, right=2.5mm, top=0.5mm, bottom=0.5mm, fontupper= , before skip=8pt, after skip=8pt

# HarnessDev : Can LLMs Create and Evolve Their Own Agent Harness?

###### Abstract

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness . Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model’s ability to develop the harness itself comparatively underexplored. We introduce HarnessDev , a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation , the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution , it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability —task success on held-out benchmarks, and efficiency —execution-token cost. The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

## 1 Introduction

As agents move from research prototypes to deployed tools such as coding assistants ( Anthropic, 2024 ; OpenAI, 2025 ) , data-analysis copilots, browser workers ( browser-use contributors, 2026 ) , and research pipelines, their capability increasingly depends on software outside the model’s weights. This surrounding execution infrastructure, commonly termed the agent harness ( Pan et al., 2026 ; Ning et al., 2026 ) , manages the execution loop, tool use, context, failure recovery, and result verification that turn model outputs into actions ( Anthropic, 2025 ) . Its impact is substantial: with identical weights, GPT-5 solves 35.2% of Terminal-Bench 2.1 inside Terminus 2 but 49.6% inside Codex CLI ( The Terminal-Bench Team, 2026 ) . As agents specialize to more domains, the demand for purpose-built harnesses will continue to grow. Because these systems require continuous development rather than one-time implementation, a practical question is whether LLMs can assist harness engineers—or even take over such a role—in building and continually improving the harness.

Despite this practical need, most agent evaluations select a harness for a given comparison and report model performance on downstream tasks ( Jimenez et al., 2023 ; Mialon et al., 2023 ; Zhou et al., 2023 ; Yao et al., 2024 ; Liu et al., 2023 ) . This setup supports controlled task-level comparison, but treats the harness as part of the experimental configuration rather than as an artifact to be developed. Recent work has begun to study harness representations, automated agent design, and agents that build or improve agent systems ( Pan et al., 2026 ; Ning et al., 2026 ; Hu et al., 2024 ; Zhang et al., 2024 ; Lu et al., 2026 ; Zhang et al., 2026a ) . However, it remains underexplored whether models can both create and continually improve runnable, persistent harnesses. Answering this question requires separating the model that develops the harness from the model that executes downstream tasks, recording the development environment, and measuring downstream performance, transfer across executors, distance from human-engineered systems, regression, and cost.

This evaluation gap is particularly consequential because harness engineering is fundamentally different from ordinary code editing. When a model modifies a standalone program, the target behavior is externally specified and success is locally verifiable. When a model modifies its own harness, it is editing the execution substrate through which it acts: the change alters how the model itself observes, plans, and recovers in all future tasks. Effective harness improvement therefore demands that the model recognize its own behavioral limitations from execution traces ( Shinn et al., 2023 ) , diagnose structural bottlenecks in the system it runs inside, and commit targeted changes that accumulate into lasting, reusable capability gains rather than one-off fixes ( Yang et al., 2024 ; Wang et al., 2024 ) . As frontier models grow capable enough to edit multi-file codebases and close real pull requests, this ability is already latent; what is missing is a benchmark that measures it.

We introduce HarnessDev (Figure 1 ), a benchmark that fills this gap by shifting the unit of evaluation from task outputs to runnable infrastructure: measuring a model’s ability to construct and maintain execution systems that are durable, inspectable, and reusable. The name reflects the software-engineering sense of develop : developing a harness includes both building it from scratch (i.e., Creation) and improving it through continued iteration and maintenance (i.e., Evolution). The benchmark covers two stages of harness development. In Creation , a creator LLM starts from a deliberately weak but runnable seed and builds a complete harness for a new task family. In Evolution , it starts from an existing harness and continues to develop it toward better downstream task performance. Together, the two stages evaluate whether models can complete harness development tasks and continuously improve the resulting system.

Evaluating a generated harness is harder than evaluating a generated answer. A harness can overfit to the model that wrote it, memorize development examples, improve one capability while silently regressing another, or improve the feedback-set score through benchmark-specific changes that do not transfer to new tasks. We therefore evaluate along two axes. Capability measures whether the harness works: we run it on held-out downstream tasks and report task-level success rates. Efficiency measures how many executor-model tokens the frozen harness consumes when deployed to solve downstream tasks.

Our findings follow the two stages of harness development:

#### Harness Creation.

Current models can construct runnable harnesses from a weak seed, but the gap from mature human-engineered systems varies substantially by harness type. When each harness runs with the model that built it, model-built harnesses match the reference on short-form writing and exceed it on machine-learning experimentation. The gap is largest for search and research harnesses, which require long-horizon information seeking, and remains substantial for code harnesses, which must coordinate repository inspection, editing, and verification over many turns. Harnesses built by different creator models also differ substantially not only in downstream task performance, but also in the number of executor tokens they consume. Higher execution cost does not reliably produce better results, so harness quality must be assessed through both capability and efficiency.

#### Harness Evolution.

Current models can use downstream execution feedback to improve their own harnesses, but reliable evolution remains difficult. Performance often rises and falls across successive revisions, and gains observed during development become smaller and less consistent on unseen tasks. The outcome also depends strongly on the model that runs the harness: changing this runtime model alters both the starting performance and whether subsequent revisions help. These results show that models can make useful local improvements, while robust evolution across unseen tasks and runtime models remains an open challenge.

## 2 Background

Most agent benchmarks begin after the problem has already been made executable: the task is specified, the reward or judge is defined, and the execution scaffold is fixed. This setting is necessary for controlled comparison, but it hides the work that dominates real deployment. In industry that work is spread across several roles—solutions architects, applied and platform engineers—but its most visible recent crystallization is the forward-deployed engineer (FDE), a title popularized by Palantir and since adopted by frontier-model companies ( Palantir Technologies, 2020 ; Orosz, 2025 ) . An FDE is embedded at the customer site after a system is adopted and turns a general-purpose model into something that runs against that customer’s data formats, workflows, and compliance constraints—for example, rewriting an ingestion path because logs may only be retained for a fixed period, or localizing a failure in a cross-jurisdiction contract pipeline. The role’s success criterion is not a demo or a benchmark score but whether the deployed system is genuinely used, keeps working, and improves; its failures are folded back into the product as fixes and feature requests ( Orosz, 2025 ) . The rapid growth of FDE hiring across frontier-model and data-platform companies ( The New Stack, 2026 ) reflects a simple fact: a capable model is not yet a working system ( MIT Project NANDA, 2025 ) , and today the gap is closed by human engineers.

Viewed from the model’s side, FDE work supplies three pieces of structure that benchmark designers normally presuppose. First, the target is vague: an informal business intent must be translated into concrete objectives, constraints, and success criteria. Second, the feedback signal is absent or unreliable: tests, judges, traces, or other self-evaluation must be constructed before anyone can tell whether the system is improving—“compliant” only becomes checkable once someone encodes what compliance means here. Third, the execution system does not exist in a usable form: the tools, context management, state, lifecycle logic, and verification interface through which future tasks will run must be built, adapted, and then maintained —an FDE stays with the system as requirements shift, rather than delivering once and leaving.

This paper focuses on the third layer. In a typical controlled agent evaluation, researchers select an agent configuration and report task completion under that configuration. The harness is therefore usually part of the evaluation setup rather than the object being developed. HarnessDev instead asks whether language models can create this execution scaffold from a weak starting point and then improve it using feedback while preserving constraint compliance and held-out performance. Whereas Aspire studies how broad deployment needs become capability growth and S 3 Gym studies whether interaction experience can be judged and reused, HarnessDev isolates how models build and maintain the systems that carry them. \FloatBarrier

## 3 Benchmark

### 3.1 Overview

HarnessDev evaluates the execution system that a model develops, rather than the answer it produces for a single task. The submitted artifact is a runnable harness that is frozen and then reused across downstream tasks. A creator LLM L C L_{C} works inside a development environment D D to produce a runnable harness H H . The development signal differs by setting and is defined in Table 1 . After development, H H is frozen. An executor LLM L E L_{E} then runs inside it on a downstream task x x , and evaluator J J scores the resulting output y y : ( L C , D ) → H , ( H , L E , x ) → y → 𝐽 score . (L_{C},D)\rightarrow H,\qquad(H,L_{E},x)\rightarrow y\xrightarrow{J}\mathrm{score}. (1) Thus, D D is used to build H H , whereas L E L_{E} is used only after H H is frozen. In implementation terms, H H contains the execution loop, tools, context management, persistent state, lifecycle control, and verification; we describe these components in words rather than assigning each another symbol.

The benchmark studies two stages of harness development.

#### RQ1—Creation.

Can a model build an effective harness from a weak but runnable seed? The creator must turn a task specification and a few development cases into infrastructure that generalizes to unseen tasks.

#### RQ2—Evolution.

Can a model improve an existing harness while preserving behavior that already works? The creator evolves its own Creation harness from downstream execution feedback. We additionally analyze the resulting artifacts and trajectories, including edit statistics, feedback response, held-out generalization, and transfer across executors.

### 3.2 Development settings

All settings provide a mutable development workspace, but they differ in the starting harness and the signal available to the creator. Table 1 gives the central distinction.

#### Weak seed H seed H_{\mathrm{seed}} .

Creation should measure whether a model can design an execution system, not whether it can reproduce benchmark boilerplate. Every creator therefore receives the same H seed H_{\mathrm{seed}} : a runnable compatibility layer, not a task-solving agent. It parses task and model configuration, exposes permitted low-level tools, and writes the required results, trajectories, logs, and task artifacts. Its tools are passive and act only when the harness calls them.

The seed has no agent loop, task decomposition, tool policy, context management, persistent task state, verifier, retry or recovery logic, or stopping rule. It may issue one connectivity probe, but it does not attempt the task. Unmodified, it produces an empty or partial artifact and scores zero on every downstream benchmark. Any nonzero Creation score must therefore come from execution logic added by the creator. This boundary matters because real scaffolds combine many control primitives, and their composition affects task performance ( Pan et al., 2026 ; Ning et al., 2026 ; Rombaut, 2026 ) .

This design avoids two extremes. An empty repository would mix harness design with command-line and file-format setup; a mature agent would give away the planning and verification structure being tested. H seed H_{\mathrm{seed}} removes the setup burden without providing a solution policy. Figure 2 shows the seed and its development environment; Figure 3 shows the control layer the creator must implement and the scorer-readable artifacts a finished harness delivers. Appendix C.1 gives its implementation skeleton.

#### Creation (RQ1).

Along with H seed H_{\mathrm{seed}} , the creator receives a task-family specification, tool and permission constraints, a short design tutorial, and one to three development cases. It may revise the harness using feedback from those cases, but it never sees the human implementation or the hidden evaluation set. The resulting harness H H is frozen before evaluation.

#### Evolution (RQ2).