HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 7/7
Cleaned-text code points [94822, 109925). Bibliography omitted here; source.md includes it.
{trajectorybox} 12 Verify the task-json CLI form and every required output artifact
Local invocation 5/5
Evaluation. Using --task-json , --workdir , and --output-dir , the harness created ok.txt containing DONE . The output directory contained changed_files.json, patch.diff, response.md, result.json, stderr.log, stdout.log, and trajectory.jsonl.
Decision. The pruned final package still satisfied the alternate invocation contract and complete artifact contract.
{trajectorybox} 13 Close with verified interfaces but no benchmark-driven harness revision
Final decision
Observation. Opus checked all three CLI forms, target-file contents, seven required artifact classes, module imports, and the package listing. The final harness/*.py contained 1,876 lines.
Final decision. Record contract smoke=1, SWE=0, and Terminal=0. The run demonstrates broad up-front system construction and execution coverage, but not a core harness edit caused by either settled zero.
## Appendix H RQ2 Code: GPT-5.5
{tcbraster} [raster columns=4,raster equal height=rows,raster column skip=1.5mm,raster row skip=2mm] {metricbox} 51.0 / 67.416
H0 · SWE / Terminal {metricbox} T2 · 56.0 / 74.157
final selection {metricbox} +5.87 pp
pair gain {metricbox} 7
complete evaluation loops
{trajectorybox} 01 Rewrite the architecture before complete H0 was available
H0 → \rightarrow T1 | | 0b33583 · core 5038dd0 | | edit first | | 4 files | | +426 / -48
Observation and analysis. Complete settled H0 was not available. From the source and a local compileall failure, the creator inferred a Python 3 entry point and predicted Terminal failures from implicit artifact paths, permissions, and repeated actions.
Modification and core diff. Add stat_path, mkdir, chmod, and plan tools; artifact-path discovery; JSON-action recovery; repeated-action unlock; and a manifest entry point that prefers python3. {diffbox} - basic read/write/command tools and weak artifact state
+ stat_path / mkdir / chmod / plan
+ artifact-path discovery and JSON-action recovery
+ manifest entry point prefers python3
• H0 was later confirmed as SWE 51/100 and Terminal 60/89.
• This round was architecture-first rather than feedback-driven.
[enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 48.0 Terminal 65.169 vs. H0: SWE -3, Terminal -2.25 {trajectorybox} 02 Replace weak completion checks with a final-review gate
T1 → \rightarrow T2 | | 55c501c | | selected final | | 1 file | | +70 / -1
Observation and analysis. Both T1 benchmarks regressed. Terminal failures showed that the adapter had not crashed; the agent declared success after weak self-written checks. SWE inspection covered only a few cases.
Modification and core diff. Block immediate success under low evidence or failed verification; add _final_review_if_needed(); and check exact paths, permissions, output format, and real verification before finish. {diffbox} - low-evidence tasks could finish immediately
+ _final_review_if_needed()
+ path / permission / format / verification review
• Hypothesis: low evidence or failed verification must block finish(success).
• This was the only round with a helper-level local assertion.
[enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 56.0 Terminal 74.157 vs. T1: SWE +8, Terminal +8.99 {trajectorybox} 03 Extend final review to every successful Terminal task
T2 → \rightarrow T3 | | d1f9384 · includes ccec8ee | | 1 file | | +129 / -12
Observation and analysis. After T2 became the current best, the creator judged the gate’s trigger surface too narrow and extended final review from low-evidence tasks to all Terminal successes.
Modification and core diff. Review every Terminal success and add _is_test_path, _verification_gaps, and path-candidate filtering. {diffbox} - review only low-evidence Terminal tasks
+ review every Terminal success
+ path and test filtering
• Task-path extraction and test-path filtering were tightened together.
• The creator extrapolated an existing mechanism rather than identifying a new failure bucket.
[enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 52.0 Terminal 75.281 vs. T2: SWE -4, Terminal +1.12 {trajectorybox} 04 Remove verification-gap enforcement after probe regression
T3 → \rightarrow T4 | | 4d2c74d | | probe-driven | | 1 file | | +5 / -12
Observation and analysis. The creator injected verification gaps into the decision prompt. When the fixed Terminal probe fell from 4/5 to 3/5, it retained path filtering and removed gap enforcement.
Modification and core diff. Remove forced verification-gap injection while retaining path-candidate filtering and avoiding second intervention in already corrected tasks. {diffbox} - force verification gaps into successful completion
+ retain path-candidate filtering
+ prevent review from degrading corrected tasks
• The probe explicitly controlled whether to roll back.
• No relationship was estimated between the n=5 probe and full evaluation.
[enhanced,breakable,colback=trajamberbg,colframe=trajamber!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 57.0 Terminal 73.034 vs. T3: SWE +5, Terminal -2.25 {trajectorybox} 05 Track files created by shell commands
T4 → \rightarrow T5 | | 3eaac25 · includes b67d86e | | 1 file | | +48 / -68
Observation and analysis. Terminal trajectories showed that shell commands modified files without write_file, so the artifact tracker never recorded them. The final gate therefore lacked awareness of real artifact changes.
Modification and core diff. Add _file_snapshot(); compare files before and after run_command; record new and changed files. {diffbox} - track only explicit write tools
+ _file_snapshot()
+ before/after run_command artifact comparison
• No new SWE failure case was opened.
• The iteration remained focused on Terminal artifact handling.
[enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 52.0 Terminal 71.910 vs. T4: SWE -5, Terminal -1.12 {trajectorybox} 06 Preserve output head and tail, then review asynchronous cancellation
T5 → \rightarrow T6 | | 615a735 · includes a81df31 | | probe-driven | | 2 files | | +6 / -34
Observation and analysis. The creator tried retaining the head and tail of long output. After probes returned SWE 0.6 and Terminal 0.8, it narrowed the remaining failure to cancel-async-tasks.
Modification and core diff. Keep output head and tail, and require final review to check asynchronous cancellation and residual tasks. {diffbox} - keep only the beginning of long output
+ keep output head and tail
+ review async cancellation and residual tasks
• The probe served as a crash/regression gate.
• 56c8380 lacked a settled SWE leg and remained transport diagnostic.
[enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 56.0 Terminal 73.034 vs. T5: SWE +4, Terminal +1.12 {trajectorybox} 07 Try and revert large-file offsets, then constrain finish actions
T6 → \rightarrow T7 | | 0a183a0 · 88aa80b / revert 137a41c | | rollback | | 1 file | | +8 / -3
Observation and analysis. The creator suspected large files were read from the wrong position. A probe returned Terminal 1.0 but SWE 0.6, so it reverted offset reading and required finish to be a standalone explicit action. Probes then reached 4/5 and 5/5.
Modification and core diff. Fully revert requested-offset reads for large files; retain only the standalone finish requirement. {diffbox} - requested-offset large-file read (reverted)
+ finish must be a standalone action
• No compileall or smoke test ran in this round.
• High probe scores did not transfer to the full evaluation.
[enhanced,breakable,colback=trajredbg,colframe=trajred!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 50.0 Terminal 68.539 vs. T6: SWE -6, Terminal -4.49 {trajectorybox} 08 Select T2 instead of the latest version
Final decision | | declare-final 55c501c | | 3 remaining pairs unused
Observation and analysis. The creator maintained a combined-score ledger. Thirteen seconds after T7 settled, it confirmed that T2 remained the true argmax and did not submit another candidate merely to sample a higher score.
Modification and core diff. Do not use T7; select T2 / 55c501c; end evolution. {diffbox} - latest T7
+ T2 / 55c501c
+ end evolution
• Final version: SWE 56.0, Terminal 74.157.
• The creator never repaired its own 54% tool-call rejection.
## Appendix I RQ2 Code: Opus-4.8
{tcbraster} [raster columns=4,raster equal height=rows,raster column skip=1.5mm,raster row skip=2mm] {metricbox} 68.0 / 74.157
H0 · SWE / Terminal {metricbox} T3 · 74.0 / 74.157
final selection {metricbox} +3.00 pp
pair gain {metricbox} 3
complete evaluation loops
{trajectorybox} 01 Build failure categories from 189 cases before a structural repair
H0 → \rightarrow T1 | | 9d6f78e | | diagnosis-first | | 8 files | | +304 / -18
Observation and analysis. The creator found 56 SWE events with score=null and refused to score from the surface event stream. It read cases.json, the feedback index, and raw results to confirm H0=68/100, then classified about 10 of 32 SWE failures as hidden-grader build, compile, or import errors.
Modification and core diff. Add a pre-finish verification gate, SIGTERM/SIGALRM always-finalize and emergency finalize, exact interface/output constraints, and stronger reliability guidance. {diffbox} - success could finish without execution after an edit
+ pre-finish verification gate
+ always-finalize / emergency finalize
+ exact interface and output constraints
• Thinking-block replay and prompt caching were checked first; latency was rejected as the main lever.
• Hypothesis: agent self-tests do not imply hidden-grader interface correctness; finish must follow real execution.
[enhanced,breakable,colback=trajgreenbg,colframe=trajgreen!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 73.0 Terminal 75.281 vs. H0: SWE +5, Terminal +1.12 {trajectorybox} 02 Give the model the real git diff for one self-review
T1 → \rightarrow T2 | | 881b01b | | 1 file | | +68 / -0
Observation and analysis. After T1 improved, the creator inspected 728 edits and found only 14 str_replace_no_match events (1.8%), rejecting the stronger-editor route. It attributed remaining failures to subtle logic errors in multi-file changes.
Modification and core diff. Add _current_diff() and one mandatory diff-grounded self-review before a repository task ends. {diffbox} - finish review relied on model memory
+ _current_diff()
+ mandatory diff-grounded self-review
• The probe was explicitly treated as a noisy n=5 crash check.
• The hypothesis was written to the ledger before the change.
[enhanced,breakable,colback=trajamberbg,colframe=trajamber!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 75.0 Terminal 73.034 vs. T1: SWE +2, Terminal -2.25 {trajectorybox} 03 Extend reflection to non-git and Terminal tasks
T2 → \rightarrow T3 | | 76de0e6 | | 1 file | | +88 / -10 | | selected final
Observation and analysis. Many Terminal artifacts were created through shell commands, so H2’s git-diff reflection never fired. The creator recorded write_file/str_replace paths and run_command state so non-git tasks could inspect recent artifacts before finishing.
Modification and core diff. Add _written_paths, _ran_command, and _recent_workdir_files(); run artifact self-review for Terminal and non-git tasks. {diffbox} - reflection only for repo tasks with git diff
+ written paths and command state
+ recent workdir files
+ non-git artifact self-review
• All 25 T2 SWE failures had already triggered reflection.
• R3 inspected concrete interface mismatches but still focused on general reflection.
[enhanced,breakable,colback=trajamberbg,colframe=trajamber!45,boxrule=0.35pt,arc=1mm,left=2mm,right=2mm,top=1mm,bottom=1mm,before skip=2mm,after skip=0mm] SWE 74.0 Terminal 74.157 vs. T2: SWE -1, Terminal +1.12 {trajectorybox} 04 Attempt process-group termination, then revert it
No formal T4 | | 5faaa15 · LEDGER.md only | | no full evaluation | | session fork
Observation and analysis. The creator computed cross-version unions and intersections: SWE 81/65/19 and Terminal 71/59/18 for union/intersection/never-pass. Reflection caused further edits only about 4% of the time. It then attempted setsid + killpg for Bash timeouts.
Modification and core diff. One session added setsid/killpg while another checked out harness/tools.py and reverted it. Final commit 5faaa15 changed only LEDGER.md; no H4 code commit existed. {diffbox} + session A: setsid / killpg
- session B: revert tools.py
+ final commit changes LEDGER.md only
• The creator judged it too risky to change Bash behavior for 189 tasks because of one edge case.
• Two concurrent sessions made opposite edits to the same file.
{trajectorybox} 05 Stop sampling and select T3
Final decision | | declare-final 76de0e6 | | 3 of 10 pairs used
Observation and analysis. The creator estimated noise of roughly ± \pm 3–4 tasks in one full evaluation and judged another trivial commit to be additional sampling rather than meaningful evolution. It declined to spend seven remaining pairs chasing a higher random peak.
Modification and core diff. Do not add candidates or repeat samples; select T3 as the balanced, validated, mechanism-complete tie-break. {diffbox} - more candidates or repeated samples
+ select T3 / 76de0e6
• Selected T3: SWE 74.0, Terminal 74.157.
• T1 pair was higher by about 0.06 pp, while T1–T3 passed the same total number of tasks.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .