Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams — part 3/4

Cleaned-text code points [31284, 46948). Bibliography omitted here; source.md includes it.

Metrics. Let N N be the total number of markets in the stream and let 𝒯 \mathcal{T} be the set of executed trades, excluding gated tasks, empty decisions, and Skip decisions. Let s i ∈ { 0 , 1 } s_{i}\in\{0,1\} indicate whether trade i i is correct, b i b_{i} be its confidence-weighted investment, and g i g_{i} be its realized profit. The metrics reported for PolyBench are: Coverage \displaystyle\mathrm{Coverage} = | 𝒯 | N , \displaystyle=\tfrac{|\mathcal{T}|}{N}, Acc \displaystyle\mathrm{Acc} = 100 ⋅ 1 N ∑ i ∈ 𝒯 s i , \displaystyle=100\cdot\tfrac{1}{N}\textstyle\sum_{i\in\mathcal{T}}s_{i}, CWR \displaystyle\mathrm{CWR} = 100 ⋅ ∑ i ∈ 𝒯 g i ∑ i ∈ 𝒯 b i , \displaystyle=100\cdot\tfrac{\sum_{i\in\mathcal{T}}g_{i}}{\sum_{i\in\mathcal{T}}b_{i}}, Return \displaystyle\mathrm{Return} = Coverage ⋅ CWR . \displaystyle=\mathrm{Coverage}\cdot\mathrm{CWR}. Accuracy therefore rewards both broad coverage and correct decisions. Return is a portfolio-style profitability metric: CWR captures the dollar-weighted profit per unit invested over the markets the agent actually traded, and the Coverage scaling discounts a high CWR earned on only a thin slice of the stream. We report Return alongside Accuracy because Accuracy treats every trade equally and is blind to stake sizing, whereas under confidence-weighted investments a confident wrong trade can offset several confident correct ones; Return therefore reflects realized P&L rather than mere directional correctness.

### A.2 CTF-Dojo

Source and composition. CTF-Dojo is a 261-challenge security-competition stream drawn from pwncollege/ctf-archive , chronologically ordered from 2011 to 2024. Challenges cover binary exploitation, web security, cryptography, reverse engineering, and forensics, exposing changes in challenge style and tooling over time.

Sandbox and verification. Each challenge runs inside a per-task Docker sandbox with constrained network policy. Flags are submitted as text and verified by SHA-256 hash comparison against the official flag.

Metrics. We report Pass@1, the percentage of challenges solved within the benchmark budget. All CTF-Dojo aggregate results use the same chronological ordering as the released stream.

### A.3 FutureX

Source and composition. FutureX is a 503-question event-forecasting stream over 82 days from Jan–Apr 2026, drawn from FutureX-Past. Questions cover finance, technology, geopolitics, and entertainment, with both English and Chinese-language variants. The zh-finance slice requires source discovery beyond default English-only retrieval.

Temporal retrieval and filtering. FutureX tasks are historical, but web pages and search indices continue to change after the event. To avoid label leakage, each task is solved with a per-task cutoff date derived from its temporal metadata. In strict built-in retrieval, Wikipedia content is fetched through the revision API using the latest revision before the cutoff, DuckDuckGo results are fetched and filtered by extracted publication dates from htmldate and URL patterns, and structured economic series are queried with observation and realtime endpoints capped at the cutoff. For evolved tools executed through the sandbox, live command output is passed through an LLM temporal filter before the solver sees it: the filter receives the task, cutoff date, and retrieved content, then either returns Clean or replaces post-cutoff values, rows, and snippets with [REDACTED] . This preserves pre-cutoff evidence while blocking dynamic web content that would reveal the resolved answer.

Metrics. We report Pass@1 under the official FutureX criterion. Slice-level human-steering analyses use the same pass criterion and aggregate tasks by the groups shown in Figure 9 .

## Appendix B Implementation and Reproducibility Details

Execution protocol. All main runs use provider-hosted LLM APIs with native tool calling and the same chronological task order used by the benchmarks. Reported numbers are generated from the corresponding results.jsonl files. Closed provider-hosted models do not expose parameter counts; we therefore report the evaluated task counts and evolution cycles rather than GPU-hours. The full-system runs contain 5,075/261/503 solve trajectories and 51/14/26 evolution cycles for PolyBench/CTF-Dojo/FutureX, respectively.

Compute and resources. All model inference is performed through provider-hosted APIs; the primary Adaptive Auto-Harness runs use Claude Sonnet 4.6 for solving and Claude Opus 4.6 for evolution, with other provider-hosted models used only for the corresponding baseline rows. We do not train or fine-tune model weights, and no local GPU compute is used for model optimization. Local compute is used for orchestration, result aggregation, figure generation, and Docker-based benchmark execution, including the per-task CTF-Dojo and FutureX sandboxes. Because the evaluated closed models do not disclose parameter counts, task counts and evolution cycles are the main compute descriptors.

Hyperparameters. Table 3 lists the per-benchmark hyperparameters used by all main runs. Because temperature is zero throughout, the reported metrics are point estimates rather than averages over re-samples.

Seed harness. Table 4 reports what the evolver inherits before any cycle runs. The seed prompt is intentionally compact: PolyBench and CTF-Dojo seed prompts are 27 and 24 lines respectively, FutureX is longer because it documents the temporal-retrieval contract. All three benchmarks start with zero seed skills, tools, and memory entries, so any harness component beyond the seed prompt and the FutureX infrastructure scaffold is constructed by the evolver itself; this isolates the gains reported in the main results from prior hand-engineering of the seed.

Token cost and wall-clock. Table 5 summarises per-system token usage and wall-clock for the five most relevant systems. Solver tokens are summed from the per-task fields in results.jsonl ; evolver-side tokens are omitted because the orchestrator did not persist them in the released artifacts. Wall-clock is the sum of per-task elapsed seconds and excludes orchestration overhead.

Temporal reveal. Each task stores a release timestamp and, when available, a resolution timestamp. Solver calls are filtered against the release time. Evolution cycles receive the trajectory immediately but receive outcome feedback only after the task has resolved, so unresolved tasks remain unlabeled history rather than leaked supervision.

Workspace artifacts. The evolver workspace persists a task board, research logs, verifier notes, tests, and architecture notes across cycles. These artifacts are separate from the solver workspace: the evolver may update harness files during evolution, while the solve-time router only inspects branch metadata and selects a branch for the incoming task.

Branch replay protocol. For the routing analysis in Appendix G , every branch in the evolved harness tree is replayed on a curated task subset to construct the Oracle and Worst controls. Oracle and Worst are post-hoc diagnostic bounds; the deployed router sees only task context and branch metadata, not labels or branch outcomes.

Human-steering records. Human-steering events are author-provided system interventions rather than recruited human-subject annotations. Each event is logged with the triggering phase, requested external signal, and workspace location where the response is recorded; Table 13 reproduces the full event log. This makes steering auditable and keeps human input as source or access guidance rather than answer labels.

## Appendix C Benchmark Non-Stationarity Diagnostics

Figures 12 – 14 provide descriptive diagnostics for the temporal structure of the three streams. These plots are not used as evaluation metrics; instead, they show why the benchmarks are not static IID pools and why a harness fitted to earlier observations can become mismatched to later tasks. Most panels are computed from task metadata, task text, or benchmark-side properties; panels that use outcomes or a baseline solver are included only as descriptive solvability proxies.

PolyBench. Prediction markets shift in both difficulty and tradability over the evaluated period (Figure 12 ). Early markets are more often liquid and already decisive: the fraction of tradeable markets drops from 97% early to 31% late, and the fraction whose maximum price exceeds 0.95 drops from 44% to 29%. At the same time, near-even markets increase from 18% to 35%, meaning later tasks more often require evidence beyond simply following a strong market consensus. The market-price correctness proxy also changes over time, from 84% early to 77% late. Together, these shifts make a fixed prediction-market strategy brittle: calibration, abstention, and evidence gathering must adapt as the stream moves from liquid and decisive markets toward thinner and more ambiguous ones.

CTF-Dojo. CTF-Dojo exposes a different form of non-stationarity: the stream expands into competitions and challenge conventions not present in the early history (Figure 13 ). The cumulative number of source competitions keeps increasing across the chronological order, and by the late stream the fraction of tasks from competitions unseen in the first third reaches 100%. The number of competitions represented in a 50-task window also varies substantially, so neighboring tasks can require different assumptions about file layout, scoring conventions, and intended exploitation style. The cross-competition score coefficient of variation further indicates that competitions are not interchangeable pools; each event can calibrate difficulty and challenge design differently. This motivates persistent construction of reusable security skills, but also cautions against treating early CTF experience as uniformly transferable.

FutureX. FutureX shifts along source, language, and difficulty dimensions (Figure 14 ). Batch-level baseline accuracy ranges from 20% to 80%, showing that chronological batches differ substantially in solvability. Later batches contain more Chinese-titled questions and more questions tied to platforms that are difficult to search directly, while the share of harder Level 3–4 questions increases sharply in the same region. Chinese-language answer requirements also appear mainly in later batches. These changes explain why FutureX stresses both construction and adaptation: the harness must acquire better source-finding and temporal retrieval behavior, while solve-time routing must select branches suited to the task’s language, source, and difficulty profile.

## Appendix D Further Experiments

We include two supplemental diagnostics that clarify where the main gains come from without duplicating the RQ analyses in § 4 .

Evolver capability and construction budget. Figure 10 varies the evolver model and construction budget on CTF-Dojo. Stronger evolvers achieve higher pass rates, while additional budget mainly helps weaker evolvers and saturates for the strongest model.

Cross-domain workspace dilution. Figure 11 compares PolyBench CWR when using workspaces evolved on different domains. The PolyBench-specific workspace performs best, while the all-evolved workspace loses 57 points of CWR, showing that mixing heterogeneous experience can dilute domain-relevant harness structure.

## Appendix E Per-Domain and Per-Category Breakdowns

CTF-Dojo. Table 6 reports Pass@1 by category. The Full System gains the most on web ( + 27 +27 over Sonnet) and crypto ( + 19 +19 ); binary/pwn remains the hardest category at 14.8 % 14.8\% even after evolution and routing, consistent with the sandbox payload-handling bottleneck identified in § 4.3 .

FutureX. Table 7 reports Pass@1 by language and inferred domain. English-language slices benefit most from evolution; the small zh-finance slice illustrates the source-discovery bottleneck where neither evolution nor routing alone can recover when the platform is behind a search wall. Domains are inferred by keyword match on the question text, since FutureX results do not store the official domain tags; the other bucket therefore aggregates unmatched questions.

PolyBench. Table 8 reports Accuracy/Return per inferred category. Sports dominates the portfolio ratio because liquid sports markets carry most of the dollar-weighted profit; politics carries near-zero return despite high accuracy. Categories are inferred by keyword match on the trajectory prompt’s Event description.

## Appendix F Multi-Agent Evolution Dynamics

Table 9 reports the full-stream system contrast (No-evo → \to Single-agent → \to Multi-agent) plus the timing of the multi-agent run, complementing rather than reproducing Figure 6 . The figure runs the four-phase evolver against No-memory and No-feedback ablations on a curated subset of samples; the table here instead reports the headline metric of each system on the full benchmark, matching the Multi-agent column of Table 2 . The unique signal added is timing: the Peak column gives the cycle index at which the multi-agent run’s cumulative mean was highest, and a peak well before the final cycle is consistent with the overfitting trend in Figure 1 . On PolyBench the multi-agent peak occurs at cycle 22 of 51 and on FutureX at cycle 10 of 26, while CTF-Dojo continues to accumulate utility across all 14 cycles.

## Appendix G Routing Behaviour and Branch Performance

This appendix expands the routing analysis behind Figure 7 . We use two related but distinct subsets per benchmark: (i) the nav-run subset (CTF-Dojo: 60 tasks, PolyBench: 100, FutureX: 80) on which the LLM router was deployed live and we observe its branch assignments; and (ii) the replay subset (CTF-Dojo: 40, PolyBench: 80, FutureX: 58), a subset of the same tasks for which every branch has been replayed end-to-end so each task carries a complete cross-venue score vector. The replay subset is necessarily smaller because some early-cycle branches did not exist for batch 1 tasks. Four per-task series are derived from the replay subset: Oracle = best venue, Adapt = the LLM router’s actual choice on the nav run, Naive = the fixed main venue (no branching), and Worst = worst venue.

Where the router sends each task. Table 10 reports the per-branch routing volume and pass rate over the nav-run subset. The router prompt (Appendix J ) does allow a main fallback when no branch matches strongly, but on these subsets the router always identifies a regime-specific branch and never invokes that fallback. Branches with low realised pass rates (e.g. branch/pwn on CTF-Dojo, branch/lvl3 on FutureX) are not failing branches per se — the router sends genuinely hard tasks to them, and the corresponding tasks have low Oracle pass rates on the replay subset as well (Table 11 ).

How much routing recovers of the adaptation gap. Table 11 gives the headline Oracle/Adapt/Naive/Worst comparison on the replay subset, with 95% bootstrap CIs and Holm–Bonferroni-corrected paired Wilcoxon p -values. The Oracle − - Naive gap, our empirical estimate of the adaptation loss L adapt L_{\mathrm{adapt}} , is large and significant on CTF-Dojo ( + 37.5 +37.5 pp, p adj = 4.8 × 10 − 4 p_{\mathrm{adj}}\!=\!4.8\times 10^{-4} ) and PolyBench ( + 8.8 +8.8 pp CWR, p adj = 1.8 × 10 − 5 p_{\mathrm{adj}}\!=\!1.8\times 10^{-5} ); on FutureX the gap is smaller and not significant after correction, consistent with the § 4.3 finding that source acquisition rather than branch choice is the binding capability there. Adapt closes a substantial fraction of the gap on CTF-Dojo and PolyBench but trails Naive slightly on FutureX, again reflecting the source-acquisition bottleneck.