HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness? — part 5/7

Cleaned-text code points [62928, 78857). Bibliography omitted here; source.md includes it.

Stuck submissions release themselves. A submission whose launch fails part way (a transient platform or CLI error) leaves a row that never reaches the platform; it stops holding its lane automatically 30 minutes after submission, and any lane held by an evaluation that never terminates is released after 12 hours. Both cases free the lane without any action from you: wait and resubmit. The evaluation ledger under ‘evals/‘ is the controller’s record and is read-only to you – editing it is not a way to free a lane, and any edit is detected and reverted.

Dedup: when a (commit, benchmark) already has a running or completed official full evaluation, submitting it again launches nothing for that benchmark and returns ‘duplicate‘ with the existing eval_id.

Official version: a commit is an official version only when it has a completed official full evaluation on every benchmark. Non-official versions cannot pass ‘declare-final‘ and do not enter any result statistics.

## Evaluation Budget (fixed) Official full evaluations are submitted only as pairs: ‘rq2b-tool submit-pair‘ freezes one commit and launches a full evaluation on every benchmark at once. One pair consumes one unit of this run’s fixed pair budget: 10 pairs total. The controller-submitted H0 baseline pair is exempt. A pair whose legs partially fail still spends its unit; resubmitting the same commit launches only the missing legs at no extra charge. When the budget is spent, ‘submit-pair‘ returns ‘budget_exhausted‘ and the run moves toward final declaration. Wake messages and ‘rq2b-tool budget‘ always report pairs_used and pairs_remaining.

Probes are rationed per round. A round is the interval between two consecutive charged pair submissions (the interval before your first pair counts as a round); within one round at most 2 probe actions are available, the allowance resets to 2 when a pair is submitted, and unused probes are never banked. One probe action — ‘rq2b-tool submit-probe‘ — freezes a commit and launches, on every benchmark at once, a probe over that benchmark’s first 5 tasks. The subset is fixed: every probe runs the same 5 tasks per benchmark, so probe results are comparable across your versions but are not a sample of the full distribution. A probe leg scores over n=5, so a single task moves that leg’s score by 20 points — read the per-task feedback rather than the aggregate number. A completed probe delivers the same evidence package as a full evaluation: per-task scores, adapter status, and raw artifact directories (trajectory.jsonl / result.json / patch / stdout / stderr). A probe whose legs all fail before launch is not charged. When the pair budget is spent, probes are disabled (‘probes_disabled_budget_spent‘). Running the harness locally inside the workspace is unlimited and free; platform probes are the scarce remote signal.

There is no time budget: waiting for running evaluations costs you nothing. There is a liveness guard: when all feedback has been delivered, budget remains, and you make no new commit or submission for 12 hours, the controller sends a factual idle notice; after 2 unanswered notices the run enters declare-only mode (submissions rejected, only declare-final available), and after 24 more hours without a final declaration the run archives without one. Unused budget is forfeited, never banked.

## Feedback Event Stream And Evaluation Records Event stream: ¡run_dir¿/feedback/events.jsonl — append-only JSONL with a monotonically increasing ‘seq‘. ‘trial_completed‘ events carry task_id, trial, score, adapter_status, and local_dir (the task’s complete raw artifact directory: trajectory.jsonl / result.json / patch / stdout / stderr); they are appended in one batch when their evaluation completes. Other event types: eval_submitted / eval_status_changed (status and progress counts only) / eval_finished / feedback_sync_complete / final_rejected / final_declared / poller_error. Evaluation records: ¡run_dir¿/evals/¡eval_id¿/ (meta.json; cases.json and feedback_index.jsonl after the evaluation ends).

## Signal Semantics (three layers) 1. Task-correctness signal: ‘score‘ — produced by the platform’s fixed verifier; the only measure of correctness. score=null means the task has no verdict yet; it counts as neither success nor failure. 2. Run-diagnostic signals: ‘adapter_status‘, ‘harness_run_diagnostic‘, eval logs, timeouts, tool errors — they reflect whether the harness ran according to its contract and help locate harness problems. Note: a task with adapter_status=success can still have score=0, and a ‘harness_run_diagnostic‘ task can still have score=1 (the verifier checks the final state of the environment/repository, not the harness’s artifact files). 3. Behavioral-artifact signals: made_edit, non-empty patch, changed-file counts — they only show that edits or deliveries happened; a non-empty patch can be entirely wrong, and terminal-style tasks do not rely on git patches, so an empty patch does not imply failure.

## Tools (call directly from any shell; JSON output) - ‘rq2b-tool submit-pair [–commit ¡sha¿] [–dry-run]‘ Freezes the commit (default HEAD) as an immutable snapshot and launches one official full evaluation on every benchmark asynchronously; consumes one pair-budget unit; returns the launched eval_ids immediately. Launches nothing and returns ‘budget_exhausted‘, ‘slots_full‘ (with the occupying evaluations), ‘duplicate‘, or ‘declare_only‘ when blocked. - ‘rq2b-tool submit-probe [–commit ¡sha¿] [–dry-run]‘ Freezes the commit (default HEAD) and launches one fixed-subset probe (5 tasks per benchmark, the same tasks every time) on every benchmark asynchronously; consumes one unit of this round’s probe allowance (2 per round, reset on each pair submission, never banked). Launches nothing and returns ‘probe_quota_exhausted‘, ‘probes_disabled_budget_spent‘, ‘slots_full‘, or ‘declare_only‘ when blocked. Probes give directional signal; official scores come only from submit-pair full evaluations. - ‘rq2b-tool list-evals‘ / ‘rq2b-tool eval-status –eval-id ¡id¿‘ Read-only facts: status, lane, and progress counts; score fields appear only after an evaluation completes and publishes. - ‘rq2b-tool read-feedback –since ¡seq¿‘ / ‘rq2b-tool wait-feedback –since ¡seq¿ [–timeout s]‘ Read, or block for, events with seq greater than the given value; waiting consumes none of your reasoning budget. - ‘rq2b-tool final-readiness –commit ¡sha¿‘ Reports, per benchmark, whether that commit has a completed official full evaluation, with eval ids and completion counts. - ‘rq2b-tool budget‘ Factual resource accounting: the pair budget (used / remaining / charged commits), this round’s probe allowance, the per-benchmark concurrency slots, and current occupancy. - ‘rq2b-tool declare-final [–commit ¡sha¿]‘ Declares the final version — the only way this experiment ends. It only accepts a commit that is not the H0 baseline and that has one completed official full evaluation (non-probe, non-cancelled) on every benchmark. Otherwise the call is rejected and returns the missing items. After a successful declaration the runner waits for all non-cancelled evaluations to reach a terminal state, finishes feedback recovery, archives, and exits.

## Session Persistence Your session may exit, be compacted, or be woken. The controller resumes the same logical session whenever an evaluation reaches a terminal state (or on a heartbeat), attaching a factual snapshot of evaluation progress. Files on disk and git history are the only reliable persistence layers.

## Invocation Contract (must be preserved) The final harness must keep supporting: ‘python -m harness run –task-json ¡task.json¿ –model-config ¡model.json¿ –output-dir ¡out¿‘ plus its existing alias and variant entry points. Every run must write result.json (with an honest status: success / partial / failed), trajectory.jsonl, and the task-required final artifacts (such as patch.diff). Never present a plan, an explanation, or an empty artifact as a completed task.

## Model-Calling Requirements The harness must reach the runtime LLM through the provided model config or environment variables. Do not hard-code model names, base URLs, API keys, or provider-specific parameters; when unsure whether a provider supports a parameter, do not send it. On API failure, follow the runtime retry policy and preserve completed tool results and artifacts. The runtime LLM drives task-semantic decisions; the harness’s job is to make those decisions executable, observable, recoverable, verifiable, and scorable.

## Prohibited Behavior (hard rules; every candidate version is audited task by task, and violations set that version’s score to 0) Any optimization that targets the scoring mechanism rather than the task itself is forbidden: - no hard-coding of task IDs, task answers, expected patches, or fixed outputs; - no reading or probing of verifier/grading scripts, and no constructing their expected state; - no tampering with evaluation artifacts or logs; - no access to benchmark source tables, gold answers, or the platform’s private scorer implementation. Do not leave TODOs, NotImplementedError stubs, or decorative modules that are never called. Do not degrade the generated agent into a one-shot LLM call, a fixed template filler, or a report-only script — the runtime LLM must be able to inspect the environment, use tools, edit files, run verification, read failures, and iterate.

## Non-Scoring Telemetry The official score is determined only by downstream benchmark performance. Structural telemetry (LOC, mechanisms, call counts, tokens, time, patch sizes) is recorded for research analysis; it is not a scoring target — do not write decorative code to satisfy shapes or metrics.

## Design Freedom And Decision Boundary You may modify anything you believe can improve downstream benchmark scores, including implementing your own execution loop, tool registry, file editing, search, shell/Python execution, patch management, state/memory/task graph, context selection and compression, verifier selection and interpretation, failure classification and recovery, artifact validation, and finalization. Neither adding structure nor keeping the existing one earns anything by itself; real downstream performance is the only official score. The controller performs no accept/reject, no best-version selection, no rollback, no failure attribution, and no feedback summarization. What to change, how to validate, when to submit, whether to probe, when to end, and which agent-harness version is best are entirely your decisions within the physical limits above. This run has no user-response channel: asking for confirmation in text changes nothing, and the controller will never declare final for you; only your successful tool calls take effect. The workspace root contains ‘BMK_INTRODUCTION.md‘, a background introduction to the two benchmarks (excluded from the git snapshot).

## Suggested Working Methods (advisory; not scored, not enforced) Practices commonly seen in mature code-agent harness work. They are suggestions only — whether and how to use them is your decision: - Diagnose before editing: when a pair completes, read its per-task feedback (scores, adapter_status, and the raw ‘local_dir‘ artifacts including ‘trajectory.jsonl‘) and name the concrete failure modes before changing code. - Keep an external ledger: append every evaluation result (commit, benchmark, score, what changed, hypothesis, outcome) to a file in the workspace. Conversation memory does not survive compaction; files and git history do. - Track your best candidate explicitly: before declaring final, compare candidates against the ledger, not against memory. - Verify locally before spending budget: run the harness’s own entry points on a task or two (local runs are unlimited), and spend this round’s probes for directional signal, before committing a pair-budget unit. - Structural patterns worth knowing from strong harnesses: planner/executor separation, a verification gate before task finish, failure taxonomy driving retries, bounded exploration budgets, external state files. - A capability-boundary reference lives at ‘capability_notes.md‘ in your workspace root: an inspection checklist of what mature code-agent harnesses cover, from task understanding through final artifacts. Use it to audit the current harness for missing or unreachable capabilities when deciding what to improve; it is advisory, not a template. - You may delegate: spawning subagents or writing scripts that call the auxiliary LLM endpoint (below, when configured) to analyze feedback, read trajectories, or review diffs is allowed; delegated usage is metered like your own.

## Design Philosophy - A strong harness can elicit and amplify the runtime LLM’s capabilities. If you conclude that the code agent failed because the runtime LLM was not capable enough, identify the specific capability shortfall and address it through the agent harness’s structure. Harness evolution and iteration are not merely about ensuring reliable execution; use structural design to strengthen the runtime LLM’s capabilities and improve the final scores.

## Auxiliary LLM Endpoint (optional) For building your own analysis tooling — for example a trajectory-analysis script, a feedback classifier, or a delegated review subagent — a model endpoint is available through shell environment variables: ‘ANALYSIS_LLM_BASE_URL‘, ‘ANALYSIS_LLM_MODEL‘, ‘ANALYSIS_LLM_API_KEY‘, and ‘ANALYSIS_LLM_PROTOCOL‘ (the wire protocol: ‘anthropic‘ or ‘openai‘). It serves the same model that drives you; its usage is metered into this run’s creator token account. It must not be used to circumvent the prohibited behaviors.

\newtcolorbox metricbox enhanced, colback=trajpanel, colframe=trajline, boxrule=0.45pt, arc=1.2mm, left=1.5mm, right=1.5mm, top=2.1mm, bottom=2.1mm, before skip=0pt, after skip=0pt

\newtcolorbox trajectorybox enhanced, breakable, colback=white, colframe=trajline, boxrule=0.45pt, arc=1.2mm, borderline west=1.8pt0ptseedaccent!72, left=3mm, right=3mm, top=2.4mm, bottom=2.4mm, before skip=2.4mm, after skip=0mm, fontupper= , coltext=trajink

\newtcolorbox diffbox enhanced, breakable, colback=trajpanel, colframe=trajline, boxrule=0.35pt, arc=0.8mm, left=2mm, right=2mm, top=1.2mm, bottom=1.2mm, before skip=1.5mm, after skip=0.4mm, fontupper=

## Appendix F RQ1 Code: GPT-5.5

{tcbraster} [raster columns=4,raster equal height=rows,raster column skip=1.5mm,raster row skip=2mm] {metricbox} 14

seed reads before first edit {metricbox} 20

editing operations observed {metricbox} 109

feedback artifacts read {metricbox} +1251 / -4

seed-to-final harness diff

{trajectorybox} Complete self-test ledger

Five local harness executions and six official dev runs are shown below. Only the three complete official runs carry benchmark scores; three interrupted runs are retained as diagnostic feedback, not promoted to settled results. {diffbox} Local: smoke A (partial) → \rightarrow smoke B (success) → \rightarrow attempt-1 final check (success) → \rightarrow resume check (success) → \rightarrow final smoke (success).

Official: contract smoke (settled) → \rightarrow SWE diagnostic A (interrupted) → \rightarrow SWE diagnostic B (interrupted) → \rightarrow Terminal diagnostic (interrupted) → \rightarrow SWE (settled) → \rightarrow Terminal (settled).

{trajectorybox} 01 Confirm the delivery boundary before judging the seed

H0 audit

Observation. The creator read the complete build contract, all three CLI forms, required outputs, public development commands, code-benchmark documentation, and the weak seed: __main__.py, seed_runner.py, the LLM client, and primitives.

Diagnosis. The seed could call tools but could not genuinely solve tasks. Because only harness/ and runtime_llm/ would be packaged, the agent loop had to be self-contained rather than rely on workspace notes or external scripts.