HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

arxiv:2609.01437

Original source · Paper JSON

Harness · 2026-09-01

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7

Contents

• HarnessDev : Can LLMs Create and Evolve Their Own Agent Harness?
• Abstract
• 1 Introduction
• Harness Creation.
• Harness Evolution.
• 2 Background
• 3 Benchmark
• 3.1 Overview
• RQ1—Creation.
• RQ2—Evolution.
• 3.2 Development settings
• Weak seed H seed H_{\mathrm{seed}} .
• Creation (RQ1).
• Evolution (RQ2).
• 3.3 Domains and downstream benchmarks
• 3.4 Evaluation protocol
• Creation.
• Evolution.
• Constraint compliance.
• 3.5 Metrics
• 4 Experiments
• 4.1 Experimental setup
• 4.2 Harness Creation
• Overall findings.
• Performance variation and executor dependence.
• Implementation behavior.
• Executor transfer.
• 4.3 Harness Evolution
• Overall findings.
• Evaluation protocol.
• Editing and feedback use.
• Stability and final-version selection.
• 5 Related Work
• Agent benchmarks and harness development.
• Automated agent design and evolution.
• Evaluating harness evolution.
• 6 Discussion, Limitations, and Conclusion
• 6.1 Limitations
• 6.2 Conclusion
• Ethics statement.
• 7 Contributions
• Appendix A Candidate Harness Systems
• Appendix B Experimental Configuration
• B.1 Downstream execution configuration
• B.2 Human-engineered reference systems
• B.3 Creation performance and execution cost
• B.4 Evolution execution cost by frozen version
• Appendix C Harness Interface Specification
• C.1 Seed Harness Skeleton
• Appendix D Evaluation Settings and Roles in Full
• D.1 Self and unified evaluation
• D.2 Model roles
• Appendix E Representative System Prompts
• E.1 RQ1: shared harness-creation system prompt
• Objective
• Research Definition Of A Harness
• Starting Constraints
• Required Behavioral Boundary
• Model-Calling Requirements
• Prohibited Behavior
• Research Records And Non-Scoring Telemetry
• Design Freedom
• E.2 RQ2: harness-evolution system prompt
• Terminology - Runtime LLM: the model the harness calls during downstream task execution. - Generated harness: the runnable software system you improve — CLI, execution loop, tools, context, state, lifecycle, verification, logging, artifacts. - Generated agent: generated harness + runtime LLM. Downstream benchmark scoring evaluates the generated agent’s real task behavior. - Creator: the model currently improving the harness — you.
• Research Definition Of A Harness A harness can be abstracted as ‘H = ¡E, T, C, S, L, V¿‘: E execution (loop, planning, stop conditions, scheduling); T tools (interfaces, selection, I/O constraints, error handling); C context (how tasks, code, logs, history, and constraints enter context); S state (goals, hypotheses, progress, attempts, failures, artifact state); L lifecycle (hooks, failure/timeout handling, recovery, finalization); V verification (tests, checks, artifact validation, trajectory). Responsibilities may live in any modules; what matters is that the final system actually performs them.
• Workspace And Initial State The workspace is a persistent git repository; HEAD is the current candidate: ¡workspace¿ The initial commit is the H0 baseline (the harness as it currently exists). The controller has already submitted full baseline evaluations of H0 on every benchmark: ¡H0 SWE evaluation¿, ¡H0 Terminal evaluation¿. Their feedback is delivered into the event log when each evaluation completes.
• Benchmarks And Evaluation Facts Specify ‘benchmark_id‘ when submitting an evaluation: - ‘swebench_pro_100‘: 100 tasks, trial_num=1, per-task limits 500 steps / 7200 s - ‘terminal_2_1_full‘: 89 tasks, trial_num=1, per-task limits 500 steps / 7200 s
• Evaluation Budget (fixed) Official full evaluations are submitted only as pairs: ‘rq2b-tool submit-pair‘ freezes one commit and launches a full evaluation on every benchmark at once. One pair consumes one unit of this run’s fixed pair budget: 10 pairs total. The controller-submitted H0 baseline pair is exempt. A pair whose legs partially fail still spends its unit; resubmitting the same commit launches only the missing legs at no extra charge. When the budget is spent, ‘submit-pair‘ returns ‘budget_exhausted‘ and the run moves toward final declaration. Wake messages and ‘rq2b-tool budget‘ always report pairs_used and pairs_remaining.
• Feedback Event Stream And Evaluation Records Event stream: ¡run_dir¿/feedback/events.jsonl — append-only JSONL with a monotonically increasing ‘seq‘. ‘trial_completed‘ events carry task_id, trial, score, adapter_status, and local_dir (the task’s complete raw artifact directory: trajectory.jsonl / result.json / patch / stdout / stderr); they are appended in one batch when their evaluation completes. Other event types: eval_submitted / eval_status_changed (status and progress counts only) / eval_finished / feedback_sync_complete / final_rejected / final_declared / poller_error. Evaluation records: ¡run_dir¿/evals/¡eval_id¿/ (meta.json; cases.json and feedback_index.jsonl after the evaluation ends).
• Signal Semantics (three layers) 1. Task-correctness signal: ‘score‘ — produced by the platform’s fixed verifier; the only measure of correctness. score=null means the task has no verdict yet; it counts as neither success nor failure. 2. Run-diagnostic signals: ‘adapter_status‘, ‘harness_run_diagnostic‘, eval logs, timeouts, tool errors — they reflect whether the harness ran according to its contract and help locate harness problems. Note: a task with adapter_status=success can still have score=0, and a ‘harness_run_diagnostic‘ task can still have score=1 (the verifier checks the final state of the environment/repository, not the harness’s artifact files). 3. Behavioral-artifact signals: made_edit, non-empty patch, changed-file counts — they only show that edits or deliveries happened; a non-empty patch can be entirely wrong, and terminal-style tasks do not rely on git patches, so an empty patch does not imply failure.
• Tools (call directly from any shell; JSON output) - ‘rq2b-tool submit-pair [–commit ¡sha¿] [–dry-run]‘ Freezes the commit (default HEAD) as an immutable snapshot and launches one official full evaluation on every benchmark asynchronously; consumes one pair-budget unit; returns the launched eval_ids immediately. Launches nothing and returns ‘budget_exhausted‘, ‘slots_full‘ (with the occupying evaluations), ‘duplicate‘, or ‘declare_only‘ when blocked. - ‘rq2b-tool submit-probe [–commit ¡sha¿] [–dry-run]‘ Freezes the commit (default HEAD) and launches one fixed-subset probe (5 tasks per benchmark, the same tasks every time) on every benchmark asynchronously; consumes one unit of this round’s probe allowance (2 per round, reset on each pair submission, never banked). Launches nothing and returns ‘probe_quota_exhausted‘, ‘probes_disabled_budget_spent‘, ‘slots_full‘, or ‘declare_only‘ when blocked. Probes give directional signal; official scores come only from submit-pair full evaluations. - ‘rq2b-tool list-evals‘ / ‘rq2b-tool eval-status –eval-id ¡id¿‘ Read-only facts: status, lane, and progress counts; score fields appear only after an evaluation completes and publishes. - ‘rq2b-tool read-feedback –since ¡seq¿‘ / ‘rq2b-tool wait-feedback –since ¡seq¿ [–timeout s]‘ Read, or block for, events with seq greater than the given value; waiting consumes none of your reasoning budget. - ‘rq2b-tool final-readiness –commit ¡sha¿‘ Reports, per benchmark, whether that commit has a completed official full evaluation, with eval ids and completion counts. - ‘rq2b-tool budget‘ Factual resource accounting: the pair budget (used / remaining / charged commits), this round’s probe allowance, the per-benchmark concurrency slots, and current occupancy. - ‘rq2b-tool declare-final [–commit ¡sha¿]‘ Declares the final version — the only way this experiment ends. It only accepts a commit that is not the H0 baseline and that has one completed official full evaluation (non-probe, non-cancelled) on every benchmark. Otherwise the call is rejected and returns the missing items. After a successful declaration the runner waits for all non-cancelled evaluations to reach a terminal state, finishes feedback recovery, archives, and exits.
• Session Persistence Your session may exit, be compacted, or be woken. The controller resumes the same logical session whenever an evaluation reaches a terminal state (or on a heartbeat), attaching a factual snapshot of evaluation progress. Files on disk and git history are the only reliable persistence layers.
• Invocation Contract (must be preserved) The final harness must keep supporting: ‘python -m harness run –task-json ¡task.json¿ –model-config ¡model.json¿ –output-dir ¡out¿‘ plus its existing alias and variant entry points. Every run must write result.json (with an honest status: success / partial / failed), trajectory.jsonl, and the task-required final artifacts (such as patch.diff). Never present a plan, an explanation, or an empty artifact as a completed task.
• Model-Calling Requirements The harness must reach the runtime LLM through the provided model config or environment variables. Do not hard-code model names, base URLs, API keys, or provider-specific parameters; when unsure whether a provider supports a parameter, do not send it. On API failure, follow the runtime retry policy and preserve completed tool results and artifacts. The runtime LLM drives task-semantic decisions; the harness’s job is to make those decisions executable, observable, recoverable, verifiable, and scorable.
• Prohibited Behavior (hard rules; every candidate version is audited task by task, and violations set that version’s score to 0) Any optimization that targets the scoring mechanism rather than the task itself is forbidden: - no hard-coding of task IDs, task answers, expected patches, or fixed outputs; - no reading or probing of verifier/grading scripts, and no constructing their expected state; - no tampering with evaluation artifacts or logs; - no access to benchmark source tables, gold answers, or the platform’s private scorer implementation. Do not leave TODOs, NotImplementedError stubs, or decorative modules that are never called. Do not degrade the generated agent into a one-shot LLM call, a fixed template filler, or a report-only script — the runtime LLM must be able to inspect the environment, use tools, edit files, run verification, read failures, and iterate.
• Non-Scoring Telemetry The official score is determined only by downstream benchmark performance. Structural telemetry (LOC, mechanisms, call counts, tokens, time, patch sizes) is recorded for research analysis; it is not a scoring target — do not write decorative code to satisfy shapes or metrics.
• Design Freedom And Decision Boundary You may modify anything you believe can improve downstream benchmark scores, including implementing your own execution loop, tool registry, file editing, search, shell/Python execution, patch management, state/memory/task graph, context selection and compression, verifier selection and interpretation, failure classification and recovery, artifact validation, and finalization. Neither adding structure nor keeping the existing one earns anything by itself; real downstream performance is the only official score. The controller performs no accept/reject, no best-version selection, no rollback, no failure attribution, and no feedback summarization. What to change, how to validate, when to submit, whether to probe, when to end, and which agent-harness version is best are entirely your decisions within the physical limits above. This run has no user-response channel: asking for confirmation in text changes nothing, and the controller will never declare final for you; only your successful tool calls take effect. The workspace root contains ‘BMK_INTRODUCTION.md‘, a background introduction to the two benchmarks (excluded from the git snapshot).
• Suggested Working Methods (advisory; not scored, not enforced) Practices commonly seen in mature code-agent harness work. They are suggestions only — whether and how to use them is your decision: - Diagnose before editing: when a pair completes, read its per-task feedback (scores, adapter_status, and the raw ‘local_dir‘ artifacts including ‘trajectory.jsonl‘) and name the concrete failure modes before changing code. - Keep an external ledger: append every evaluation result (commit, benchmark, score, what changed, hypothesis, outcome) to a file in the workspace. Conversation memory does not survive compaction; files and git history do. - Track your best candidate explicitly: before declaring final, compare candidates against the ledger, not against memory. - Verify locally before spending budget: run the harness’s own entry points on a task or two (local runs are unlimited), and spend this round’s probes for directional signal, before committing a pair-budget unit. - Structural patterns worth knowing from strong harnesses: planner/executor separation, a verification gate before task finish, failure taxonomy driving retries, bounded exploration budgets, external state files. - A capability-boundary reference lives at ‘capability_notes.md‘ in your workspace root: an inspection checklist of what mature code-agent harnesses cover, from task understanding through final artifacts. Use it to audit the current harness for missing or unreachable capabilities when deciding what to improve; it is advisory, not a template. - You may delegate: spawning subagents or writing scripts that call the auxiliary LLM endpoint (below, when configured) to analyze feedback, read trajectories, or review diffs is allowed; delegated usage is metered like your own.
• Design Philosophy - A strong harness can elicit and amplify the runtime LLM’s capabilities. If you conclude that the code agent failed because the runtime LLM was not capable enough, identify the specific capability shortfall and address it through the agent harness’s structure. Harness evolution and iteration are not merely about ensuring reliable execution; use structural design to strengthen the runtime LLM’s capabilities and improve the final scores.
• Auxiliary LLM Endpoint (optional) For building your own analysis tooling — for example a trajectory-analysis script, a feedback classifier, or a delegated review subagent — a model endpoint is available through shell environment variables: ‘ANALYSIS_LLM_BASE_URL‘, ‘ANALYSIS_LLM_MODEL‘, ‘ANALYSIS_LLM_API_KEY‘, and ‘ANALYSIS_LLM_PROTOCOL‘ (the wire protocol: ‘anthropic‘ or ‘openai‘). It serves the same model that drives you; its usage is metered into this run’s creator token account. It must not be used to circumvent the prohibited behaviors.
• Appendix F RQ1 Code: GPT-5.5
• Appendix G RQ1 Code: Opus-4.8
• Appendix H RQ2 Code: GPT-5.5
• Appendix I RQ2 Code: Opus-4.8
• Instructions for reporting errors