Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams — part 2/4

Cleaned-text code points [15328, 31284). Bibliography omitted here; source.md includes it.

The analytical decomposition identifies L adapt L_{\text{adapt}} as the loss from committing to one dense harness before observing the incoming task’s context. We reduce this loss by shifting heavy adaptation into evolution time: the evolver constructs a structured harness store, and solve time only requires a lightweight adaptation operator. The reduction is general: it factors into (i) how the evolver organizes the harness space and (ii) how the solver selects from that space per task. The harness space can be organized in many forms, including a linear chain that always uses the most recently evolved workspace, a tree of regime-specific branches, or a graph of skills with dependency edges. The adaptation operator can likewise take many forms, ranging from skill-level retrieval over a flat catalog ByteDance (2025) to branch-level routing over a structured space. Different combinations trade off construction cost, operator latency, and the granularity at which adaptation occurs.

We adopt a harness tree as the storage form and agentic routing as the adaptation operator. The tree is the natural fit for our setting because heterogeneous task streams cluster into a small number of recurring regimes (e.g., binary exploitation versus cryptography in CTF-Dojo, sports versus politics in PolyBench), branches isolate regime-specific prompts/skills/tools without cross-contaminating the others, and the branching gate gives the evolver an explicit lever to commit specialization only when warranted by failure evidence. We instantiate this with two designs (Figure 3 ). (1) Branching harness tree (built at evolution time). The solver’s workspace is a git repository. The evolver constructs regime-specific branches (e.g., branch/crypto-classical , branch/binary-reversing ) during evolution, each carrying its own prompt, skills, and tool registry; git provides versioning, isolation, and lineage tracking across branches. (2) Agentic routing (executed at solve time). A router agent reads each branch’s workspace via git show and selects the branch given x t x_{t} ’s context; the solver then checks out that branch and executes.

### 3.5 Human-in-the-Loop Channel

A third failure lies outside L evo + L adapt L_{\text{evo}}+L_{\text{adapt}} : some tasks require harness absent from ℋ t \mathcal{H}_{t} signal, such as API credentials, novel web sources, or proprietary endpoints. In this experience-insufficient setting, neither a stronger evolver nor solve-time routing can recover the missing signal. We address it with a human-in-the-loop channel that augments ℋ t \mathcal{H}_{t} through structurally triggered steering hooks (Figure 3 ). This design targets open-ended streams where new access requirements appear before autonomous evolution has relevant evidence. (1) Task-board steering. After the Analyst updates the task board, a human may review it to add entries, adjust priorities, or supply domain guidance and source access. This proactively steers the subsequent research cycle with direction the evolver cannot derive from trajectories alone. (2) Interactive assistance during research. When a Researcher agent hits a barrier mid-execution that requires human intervention (e.g., an authentication wall), the hook prompts the human in real time. This reactively unblocks the research agent at the point of failure.

## 4 Experiments

### 4.1 Experimental Setup

Benchmarks. We evaluate on three open-ended task streams (Table 1 ): PolyBench for prediction markets Cheng et al. (2026) , CTF-Dojo for security challenges Zhuo et al. (2025) , and FutureX for event forecasting Zeng et al. (2025) . All three enforce strict temporal order and covers the three dimensions of challenge. Details and non-stationarity diagnostics are in Appendix A and C .

Baselines. We compare with no-evolution runs with different solver agents using Sonnet-4.6 Anthropic (2025b) , DeepSeek-V3.2 Liu et al. (2025) , Claude Haiku-4.5 Anthropic (2025a) , GLM-4.7 Z.ai (2025) , and Kimi-K2.5 Team et al. (2026) . We compare with five auto-harness baselines: A-Evolve Lin et al. (2026) , GEPA Agrawal et al. (2025) , Meta-Harness Lee et al. (2026) , Continual Harness Karten et al. (2026) , and SkillOS Ouyang et al. (2026) . We also compare with one human-designed system OctoTools Lu et al. (2025) .

Solver and evolver. We use Claude Sonnet 4.6 as the solver for all experiments unless specified, and use Claude Opus 4.6 as the evolver, both at temperature 0 to attribute gains to the evolution algorithm rather than sampling noise; the no-evolution controls additionally report base-agent results for Haiku 4.5, DeepSeek-V3.2, Kimi-K2.5, and GLM-4.7. All algorithms share the same batch size (100/20/20 for PolyBench/CTF-Dojo/FutureX), batch loop, and temporal-reveal gate; only the evolution algorithm varies.

Metrics. For PolyBench, we report two complementary metrics: Accuracy , the fraction of all markets traded correctly; and Return , defined as Coverage × CWR \mathrm{Coverage}\times\mathrm{CWR} , where CWR is the confidence-weighted portfolio profit-to-investment ratio over traded markets and coverage is the fraction of traded. For CTF-Dojo and FutureX, we report the official pass rate (Pass@1) defined by the original benchmarks. We also report lift in the figures as the difference with baselines. Full definitions are in Appendix A .

Research questions. RQ1 (§ 4.2 ): How does Adaptive Auto-Harness compare against existing auto-harness systems on open-ended task streams? RQ2 (§ 4.3 ): How does Adaptive Auto-Harness address benchmark-specific bottlenecks? RQ3 (§ 4.4 ): Does stateful multi-agent evolution with evaluation feedback provide additional gains? RQ4 (§ 4.5 ): Can solve-time routing effectively leverage specialized harness branches? RQ5 (§ 4.6 ): Can human steering help under insufficient history signal?

### 4.2 Comparison with Baselines

Prior systems specialize on one metric cluster. A-Evolve leads the two pass-rate streams ( 45.2 % 45.2\% CTF-Dojo, 47.5 % 47.5\% FutureX) but covers only 21.1 % 21.1\% of PolyBench markets. Meta-Harness leads all three PolyBench metrics ( 55.3 % 55.3\% Coverage, 50.8 % 50.8\% Accuracy, + 320 % +320\% Return) but falls below the no-evolution Sonnet baseline on FutureX ( 29.4 % 29.4\% vs. 31.0 % 31.0\% ). Base solvers stay below 32.6 % 32.6\% PolyBench Coverage on every model, and OctoTools, the frozen human-designed system, places third on PolyBench Return but does not lead any row. The pass-rate cluster and the portfolio cluster therefore sit in different baselines.

Our three variants jointly lead all metrics. Without HITL intervention, the Full System combines multi-agent evolution with solve-time routing and reaches 97.9 % 97.9\% PolyBench Coverage, 80.9 % 80.9\% Accuracy, and 50.2 % 50.2\% CTF-Dojo Pass. The Multi-agent variant leads FutureX at 49.5 % 49.5\% , where evolving the right source and tooling matters more than per-task routing. The Adaptive variant leads PolyBench Return at + 352 % +352\% , where matching each market to a specialized strategy matters.

### 4.3 Benchmark Bottlenecks

RQ2 asks whether Adaptive Auto-Harness targets the bottlenecks that limit each benchmark. We organize the analysis around the two losses introduced in § 3.2 : the evolution loss L evo L_{\text{evo}} , the gap from capabilities the evolver class cannot construct; and the adaptation loss L adapt L_{\text{adapt}} , the gap from committing to a single harness across heterogeneous tasks.

Evolution bottlenecks ( L evo L_{\text{evo}} ) differ across benchmarks. Figure 4 plots a benchmark-specific stress axis chosen to expose the most predictive capability. The PolyBench panel plots mean stated confidence against market consensus, defined as the implied probability of the favored outcome from Polymarket prices; a well-calibrated harness from our multi-agent variant tracks the diagonal, while single-agent variants stay flat and over-state confidence on low-consensus markets, suggesting that the binding capability is consensus-aware confidence calibration rather than raw prediction skill. The FutureX panel plots pass rate against three retrieval tiers, ranging from offline to date-filtered Wikipedia plus DuckDuckGo to unrestricted DuckDuckGo web search. Pass rate increases monotonically from 34.0 % 34.0\% to 47.6 % 47.6\% to 57.1 % 57.1\% , identifying source acquisition rather than reasoning as the binding capability. The CTF-Dojo panel plots pass rate against the largest challenge file size, binned into five tiers from no-payload to > > 1MB. The best single-agent variant declines from 81.8 % 81.8\% to 30.4 % 30.4\% , and the multi-agent variant declines from 90.9 % 90.9\% to 39.1 % 39.1\% while retaining roughly a 9-point margin throughout, indicating that payload-handling infrastructure becomes the binding capability as inputs grow and that the multi-agent evolver mitigates but does not eliminate this bottleneck. In summary, the binding capability therefore differs by benchmark.

Adaptation bottlenecks ( L adapt L_{\text{adapt}} ) remain after evolution. Figure 5 reports per-task adaptation lift over evolver cycles, where lift on each task is the adaptative harness score minus the median score across single-committed-harness baselines. The solid curve is the within-cycle mean lift and the shaded band is performance variation across task categories; if a single committed dense harness were sufficient, the mean lift would approach zero as evolution progresses. We observe instead that the mean lift and variation remain positive across all cycles on all three benchmarks.

Takeaway. Although streams expose different bottlenecks, they can be addressed under the same principle: reduce evolution loss for missing capabilities and adaptation loss for task-specific fit.

### 4.4 Stateful Multi-Agent Evolution

(a) Example of the four-phase evolution cycle.

(b) Example of the solve-time routing trace.

RQ3 asks whether the four-role evolver in section § 3.3 (Analyst → \to parallel Researchers → \to Builder → \to Verifier) improves over a single-agent evolver, and whether its state channels matter. Using 100/60/80 tasks on PolyBench/CTF-Dojo/FutureX, Figure 6 compares the full system with variants that remove temporal-reveal feedback or cross-cycle memory. The full system improves over the single-agent evolver on all three benchmarks: 20.3 → 44.3 20.3\to 44.3 CWR on PolyBench, 38 % → 43 % 38\%\to 43\% on CTF-Dojo, and 38 % → 44 % 38\%\to 44\% on FutureX. Removing memory causes the broadest degradation, while removing feedback mainly hurts PolyBench, where outcomes resolve after trading.

Takeaway. The four-role evolver is strongest when paired with persistent state: memory preserves cross-cycle search, and feedback turns resolved outcomes into later evolution signal.

### 4.5 Solve-Time Routing on the Harness Tree

RQ4 asks how much adaptation headroom a specialized harness tree exposes, and how much solve-time routing captures. We measure this with a designed analysis rather than end-to-end deployment, isolating the headroom from router quality. On 80/40/58 tasks for PolyBench/CTF-Dojo/FutureX, we seed one branch per task category, evolve the tree over the stream, and replay every task through every branch. We then compare Oracle (best branch per task), Adapt (category-based routing), Naive ( main only), and Worst (worst per task). As shown in Figure 7 , Adapt improves on both CTF-Dojo and PolyBench, while leaving headroom toward Oracle branch selection. On FutureX, main outperforms Adapt, as web source retrieval ability dominates.

Takeaway. Harness specialization opens real adaptation headroom, but realized routing captures only part of it, so turning headroom into gain is a separate challenge from constructing the branches.

### 4.6 Human Steering for Auto-Harnessing

RQ5 asks whether human steering helps when history lacks the source or access signal needed for evolution. Since all previous experiment does not apply HITL to ensure fairness, we evaluate this on 100 FutureX tasks and restrict human input to two hooks: research-phase steering supplies credentials when research is blocked, and task-board steering adds source directions the evolver cannot infer (Figure 8 c). Figure 9 shows the slice-level effect: lift is 0 0 on broad polymarket questions, rises to + 5 +5 on broad search-dependent questions, peaks at + 20 +20 on the directly targeted finance&tech slice, and remains + 15 +15 on adjacent Western-specialty questions. The pattern indicates that HITL helps when it injects the missing external signal, not generic human advice.

Takeaway. Human steering is most useful when the missing ingredient is external source knowledge rather than additional autonomous evolution.

## 5 Conclusions

Open-ended task streams expose three challenges for auto-harness deployment: unbounded task arrival, heterogeneous tasks, and non-stationarity. Adaptive Auto-Harness addresses these challenges by pairing sustained harness construction with solve-time task adaptation. The decomposition into evolution loss and adaptation loss clarifies why a single repeatedly updated harness is insufficient: the system must build missing capabilities from stream evidence while selecting the right specialized branch for each task. The experiments further show that the three mechanisms are complementary rather than interchangeable: multi-agent evolution constructs benchmark-specific capabilities, routing exploits harness specialization when branch signals are reliable, and human steering supplies external signals that history cannot contain.

## 6 Limitations

Benchmark coverage. We evaluate on three open-ended task streams: prediction markets, cybersecurity challenges, and event forecasting. These domains cover unbounded streams, task heterogeneity, and distributional non-stationarity, but the same framing should be tested further on additional deployment streams where the stream is further expanded spatially and temporally to mimic the real-world deployment.

Diagnostic losses. The evolution loss L evo L_{\text{evo}} and adaptation loss L adapt L_{\text{adapt}} are analytical quantities, not directly estimated oracle losses. Our experiments diagnose them through bottleneck analyses, ablations, and routing controls rather than through a formal estimator of the oracle harness.

## 7 Ethics Statement

We use public research benchmark tasks and do not introduce private user data. CTF-Dojo runs only inside isolated benchmark containers and does not target real systems. Human steering is limited to source guidance, task-board edits, and credential decisions; humans do not label/expose answers or choose solver branches.

## 8 AI Usage Statement

We used AI assistants to refine the writing of this paper and to accelerate debugging and analysis during implementation and evaluation.

## Appendix A Benchmark and Evaluation Details

Across all three benchmarks, tasks are evaluated in chronological order. The solver receives task x i x_{i} with only the information available before its release time, while the evolver receives outcome labels only after the corresponding resolution time. Our analysis pipeline loads the first record for each instance_id , so retries or duplicate logs do not change the reported metrics. The three benchmarks together exercise the three open-ended-stream deployment dimensions identified in § 1 .

### A.1 PolyBench

Source and composition. PolyBench is a Polymarket-derived prediction-market stream with 5,075 tasks from Feb 6–22, 2026. The stream spans politics, sports, finance, crypto, and entertainment markets, and each task is resolved against the official market outcome.

Temporal ordering. For each market, we store the task release timestamp and the outcome resolution timestamp. Resolved labels are hidden from evolution until the corresponding market has resolved.