HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 6/7

Cleaned-text code points [78857, 94822). Bibliography omitted here; source.md includes it.

{trajectorybox} 02 Replace the one-shot runner with a JSON-action tool loop

Architecture and first implementation

Plan. Runtime LLM selects actions → \rightarrow the harness parses JSON → \rightarrow executes read, write, search, shell, or patch tools → \rightarrow returns observations → \rightarrow verifies → \rightarrow emits benchmark-readable artifacts.

Modification. A new harness/agent.py centralized parsing, dispatch, state, verification, and finalization while retaining seed primitives. GPT deliberately concentrated reliability logic in one control surface so deviations in runtime-model output could be repaired directly from trajectories. {diffbox} - from .seed_runner import run as run_seed

+ from .agent import run as run_agent

+ harness/agent.py # 1,245 lines in the final version

Final seed-to-final: 12 paths, +1251 / -4 lines.

{trajectorybox} 03 Local smoke A executed the model but misreported the artifact

Local harness execution 1/5

Observation. The runtime model made one real LLM call, invoked write_file, and created smoke_out.txt. However, the result was partial : patch.diff and changed_files were empty even though the non-git target file existed. The initial scan of the broad /tmp directory also consumed noisy context.

Modification. Narrow the initial environment summary; add artifact_paths; and register files created by write, replacement, and patch tools independently of git diff. {tcolorbox} [enhanced,breakable,colback=trajamberbg,colframe=trajamber!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] status=partial LLM calls=1 tool calls=1 file writes=1 artifact accounting failed

{trajectorybox} 04 Local smoke B confirmed the accounting repair

Local harness execution 2/5

Evaluation. A fresh run created result.txt containing exactly OK . result.json reported status=success , two LLM calls, two tool calls, one file write, and artifact_paths=[result.txt] even though the directory was not a git repository.

Decision. The original defect was artifact bookkeeping rather than an inability to execute the task; the repaired local path was ready for the public contract smoke.

{trajectorybox} 05 Validate the execution contract before spending a real benchmark budget

Official dev run 1/6 - settled

Observation. The official smoke confirmed the CLI, runtime LLM, batched actions, required files, and trajectory capture. It established execution health, not difficult-task capability. {tcolorbox} [enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260804-182040 code_contract_smoke 1.0 harness started trajectory found

{trajectorybox} 06 SWE diagnostic A exposed multi-object JSON loss

Official dev run 2/6 - interrupted diagnostic

Observation. The created agent was investigating the repository, but some assistant turns contained several complete JSON actions. The first parser accepted only one object and recorded the whole turn as invalid, wasting steps and context.

Modification. Add _extract_json_objects() with JSONDecoder.raw_decode and convert all valid objects into a batch action. {diffbox} + def _extract_json_objects(text, max_objects=8)

+ return {’action’:’batch’,’args’:{’actions’:actions}}

Run dev-20260804-182112 produced actionable trajectory evidence but no complete summary. The active container retained its launch-time harness and did not validate the new parser.

{trajectorybox} 07 SWE diagnostic B exposed the shell mismatch

Official dev run 3/6 - interrupted diagnostic

Observation. Multi-object recovery worked in the new run, which then reached commands using source , environment activation, and other Bash syntax. subprocess.run(shell=True) invoked /bin/sh and rejected otherwise executable fallbacks.

Modification. Select /bin/bash when shell execution is requested and Bash exists. {diffbox} + executable=’/bin/bash’ if shell and Path(’/bin/bash’).exists() else None

Run dev-20260804-182709 remained incomplete. Its old container did not hot-load the shell fix.

{trajectorybox} 08 The first Terminal attempt reached real compilation but did not settle

Official dev run 4/6 - interrupted diagnostic

Observation. The partial created-agent trajectory showed the harness creating and compiling /app/gpt2.c. This demonstrated genuine Terminal execution beyond a toy smoke, but the run ended without a complete summary or verifier result.

Decision. Preserve the run as pending diagnostic feedback, not as a score, and continue from the same workspace rather than claim completion. {tcolorbox} [enhanced,breakable,colback=trajamberbg,colframe=trajamber!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260804-183216 real harness activity no settled summary no score

{trajectorybox} 09 End attempt 1 with a clean local contract check

Local harness execution 3/5

Evaluation. The final_harness_check run used the alternate --workspace/--output/--max-turns CLI form, created check.txt containing OK , returned status=success , and recorded artifact_paths=[check.txt] .

Decision. The package remained locally runnable despite the three incomplete benchmark attempts; the unresolved benchmark state was carried into attempt 2 rather than erased.

{trajectorybox} 10 Resume the same workspace and verify retained behavior

Local harness execution 4/5 and same-workspace continuation

Observation. Attempt 2 reread DESIGN_NOTES.md, the file listing, prior dev state, compile/help output, and agent.py. It explicitly treated the previous SWE and Terminal runs as incomplete feedback.

Evaluation. A resume check created out.txt containing OK , returned status=success , recorded the artifact, and passed both a content assertion and an artifact-existence check.

{trajectorybox} 11 Completed SWE revealed duplicate actions and a Codex-style patch envelope

Official dev run 5/6 - settled

Observation. Multi-object recovery executed duplicate reads, searches, and status calls. The runtime model also produced a *** Begin Patch envelope, while the old tool accepted only unified diffs.

Modification. Deduplicate only identical safe actions within a batch; never deduplicate write, patch, or finish. Route Codex envelopes through a dedicated parser while retaining git apply and patch -p1/-p0 fallbacks. {diffbox} + safe dedupe: list_tree, find_files, search_text, read_file, git_status, git_diff

+ writes, patches, and finish are never deduplicated

+ Codex envelope -> dedicated parser -> git apply -> patch fallbacks

{tcolorbox} [enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260804-184509 SWE-Pro 0.0 pipeline complete non-empty patch

The settled score validates the launch-time snapshot. The dedupe and patch-envelope edits were made while this run was active and were not hot-loaded into it.

{trajectorybox} 12 Terminal passed, while live feedback prompted one final dedupe extension

Official dev run 6/6 - settled

Observation. The live trajectory again showed an identical run_command repeated within one batch. GPT extended safe deduplication to commands with identical arguments while preserving every write operation. The run created and compiled /app/gpt2.c.

Decision. Wait for the outer grader and read both verifier reward and official summary before claiming success. {tcolorbox} [enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260804-190052 Terminal 1.0 verifier passed trajectory found

The command-dedupe extension was written after launch, so Terminal 1.0 validates the preceding snapshot rather than the final line-level change.

{trajectorybox} 13 Validate the final snapshot locally, then stop

Local harness execution 5/5 and closeout

Evaluation. The final smoke made one LLM call, created hello.txt, returned status=success , recorded the artifact, and passed the artifact check. GPT also compiled the package, scanned placeholders, repaired two remaining pass statements, rescanned, verified output files, and removed __pycache__.

Final decision. Stop with the complete record: contract smoke=1, SWE=0, Terminal=1. The official scores and the local final-snapshot verification remain causally distinct.

## Appendix G RQ1 Code: Opus-4.8

{tcbraster} [raster columns=4,raster equal height=rows,raster column skip=1.5mm,raster row skip=2mm] {metricbox} 20

seed reads before first edit {metricbox} 10

harness modules written upfront {metricbox} 3

valid official self-tests {metricbox} +1252 / -596

seed-to-final harness diff

{trajectorybox} Complete self-test ledger

Five local invocations and four official dev-run directories are shown below. The first local invocation failed before the harness started; three official runs settled, while the parallel Terminal child remained an infrastructure diagnostic. {diffbox} Local: import-path failure → \rightarrow terminal task success → \rightarrow repository task success → \rightarrow post-prune terminal success → \rightarrow task-json CLI success.

Official: contract smoke (settled) → \rightarrow parallel SWE (settled) + Terminal child (infra-incomplete) → \rightarrow serial Terminal (settled).

{trajectorybox} 01 Read the seed, runtime client, and platform contract before rewriting

H0 audit

Observation. The creator read the task, interface, and capability documents; seed CLI, runner, I/O contract, LLM client, five primitive classes; and the dev runner’s task fields, artifact directories, and self-test commands.

Diagnosis. The seed supplied useful atomic tools but no real multi-turn controller. The runtime client required one persistent message list to reuse the provider signature correctly, so the core should be a single-threaded native tool-calling ReAct loop.

{trajectorybox} 02 Define module boundaries, then rewrite the execution system nearly all at once

Overall design

Modification. Opus mapped real responsibilities to modules: agent.py for execution and lifecycle; tools.py and schemas.py for structured tools; taskspec.py and prompts.py for classification and context; runner.py for diffs, untracked files, verification, and artifacts. {diffbox} Execution/lifecycle: agent.py +246

Tools: tools.py +271; schemas.py +103

Context: taskspec.py +104; prompts.py +91

Result/verification: runner.py +242; model.py +80; util.py rewritten

CLI/package: __main__.py +74/-43; __init__.py +9/-1

Final seed-to-final: +1252 / -596 lines.

{trajectorybox} 03 Encode budgets, context control, and honest termination upfront

Key implementation

Modification. Set max_steps=120, a 6,600-second time budget, and a 150,000-token soft context limit. Retain the latest three tool observations when compacting context; stop on budget; and downgrade repeated no-tool turns or an empty repository diff to partial rather than falsely report success. {diffbox} + DEFAULT_MAX_STEPS = 120

+ TIME_BUDGET_SECONDS = 6600

+ CONTEXT_TOKEN_SOFT_LIMIT = 150000

+ repeated no-tool turns or empty repo diff -> partial

{trajectorybox} 04 The first local invocation failed before the harness started

Local invocation 1/5 - environment diagnostic

Observation. Calling python -m harness from /tmp returned No module named harness . No created-agent loop or tool trajectory had started.

Diagnosis and decision. The local working directory lacked the online package path. Opus changed only the test command to PYTHONPATH=/workspace ; it did not modify harness logic or count the failure as a harness result.

{trajectorybox} 05 The corrected terminal-path test ran end to end

Local invocation 2/5

Evaluation. With the correct package path, the runtime LLM and tool loop created smoke_result.txt containing OK . Opus then inspected the output files and trajectory phases to confirm a real model call and real tool execution.

Decision. The terminal execution path worked; proceed to a repository task rather than treating one file-write toy as sufficient coverage.

{trajectorybox} 06 A local repository task exercised edit, verification, and patch generation

Local invocation 3/5

Evaluation. In a temporary git repository, the created agent diagnosed return a - b , changed it to addition, adapted when pytest was unavailable, and produced a clean non-empty patch and successful result.

Decision. Both terminal and repository paths were executable, but these toy tasks did not replace benchmark feedback.

{trajectorybox} 07 The official contract smoke passed without prompting a code change

Official dev run 1/4 - settled

Observation. The smoke validated the CLI, runtime model invocation, output files, and trajectory. The feedback matched the intended contract check, so Opus retained the implementation. {tcolorbox} [enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260803-183520 code_contract_smoke 1.0 harness started trajectory found

{trajectorybox} 08 Parallel submission created a separate infrastructure-invalid Terminal child

Official dev run 2/4 - infra-incomplete Terminal child

Observation. The parallel command launched SWE and Terminal directories. While the SWE child continued, the platform rename from current.json.tmp to current.json raised FileNotFoundError; the Terminal child had an empty log and no summary.

Decision. Do not modify the harness around a control-plane live-file race. Preserve dev-20260803-183608-terminal_2_bench as infra-incomplete and rerun Terminal serially.

{trajectorybox} 09 SWE completed real work but failed hidden tests

Official dev run 3/4 - settled

Observation. The created agent located Ansible shebang logic, edited source and changelog files, and ran pytest/import checks. The loop completed investigate → \rightarrow edit → \rightarrow verify, but the concrete patch failed the grader.

Decision. Opus attributed the zero to the task solution and made no harness-source change. This retained the original harness hypothesis rather than forming a feedback-driven edit-and-retest loop. {tcolorbox} [enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260803-183608-swebench_pro SWE-Pro 0.0 38 steps 19 Bash calls about 5.9K patch characters

{trajectorybox} 10 Serial Terminal produced the target artifact but still scored zero

Official dev run 4/4 - settled

Observation. The created agent reverse-engineered the GPT-2 checkpoint layout, compiled and linked repeatedly, and wrote /app/gpt2.c. The long execution chain was real, but the verifier rejected the solution.

Decision. Opus again retained the core harness and made no feedback-driven logic change after the zero. The result shows execution coverage, not improvement from failed benchmark feedback. {tcolorbox} [enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] dev-20260803-184604 Terminal 0.0 94 steps 81 Bash calls wrote /app/gpt2.c

{trajectorybox} 11 Remove unused seed remnants and immediately rerun the terminal path

Source closeout and local invocation 4/5

Observation. Reference searches showed that the new control flow no longer used primitives/ or io_contract.py. llm_client.py remained a real fallback when runtime_llm import failed.

Modification. Delete the unused primitive modules and io_contract.py, then rerun imports and a terminal task. {diffbox} - harness/primitives/{files,git_ops,paths,process,search}.py

- harness/io_contract.py

+ retain harness/llm_client.py as model.py fallback

Evaluation. The post-prune smoke created out.txt containing 42 ; result.json returned status=success .