Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

arxiv:2604.25850

Original source · Paper JSON

Harness · 2026-04-28

Cites (5) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 Harness Engineering and Evaluation for Coding Agents
• 2.2 Automated Optimization of LLM Agents
• 3 Method
• 3.1 NexAU: an editable, decoupled harness substrate
• 3.2 Agent Debugger: layered trajectory evidence
• 3.3 Evolve Agent: evidence-driven, auditable edits
• 4 Experiments
• 4.1 Setup
• Evaluation.
• Models.
• 4.2 RQ1: Main Results
• AHE outperforms both human-designed and self-evolve baselines.
• Prompt-only self-evolution misses the components that carry AHE’s gain.
• 4.3 RQ2: Transfer to Unseen Tasks and Base Models
• Cross-benchmark transfer.
• Cross-model transfer.
• 4.4 RQ3: Analysis
• 4.4.1 RQ3a: where value accumulates across components
• Each component owns a different failure surface.
• Components interact non-additively, capping the aggregate gain.
• 4.4.2 RQ3b: how reliably the loop’s self-attribution tracks reality
• Evidence-driven targeting.
• Regression blindness.
• 5 Conclusion
• Limitations
• Benchmark scope.
• Evolution operating point.
• Self-modification governance.
• Appendix A Experimental Setup: Full Details
• Seed agent.
• Runtime infrastructure.
• Hyperparameters.
• Terminal-bench difficulty labels.
• pass@1.
• Token cost and Succ/Mtok.
• Appendix B Prompts and Configurations
• B.1 Code Agent Seed System Prompt
• B.2 Evolve Agent Prompt
• B.3 Explore Agent Prompts
• B.3.1 Source-code Exploration Agent
• B.3.2 Web-research Agent
• Appendix C Qualitative Case Study
• C.1 Trajectories: failing versus passing rollouts
• C.1.1 Trajectory 1: db-wal-recovery
• The task.
• Trajectory before and after the iteration-2 changes.
• C.1.2 Trajectory 2: path-tracing
• The task.
• Trajectory before and after the iteration-5 changes.
• C.1.3 Trajectory 3: mcmc-sampling-stan
• The task.
• Trajectory before and after the iteration-6 changes.
• C.1.4 Trajectory 4: configure-git-webserver
• The task.
• Trajectory before and after the iteration-8 changes.
• C.2 Changes shipped on the four winning rounds
• C.2.1 Iteration 2: prompt rules and shell-timeout argument
• C.2.2 Iteration 5: publish-state mechanism (prompt rules + shell-tool guard)
• C.2.3 Iteration 6: protected entrypoints and execution-risk middleware
• C.2.4 Iteration 8: hard blocks and FRAMEWORK reminders
• C.3 Reading the change-manifest figures
• Appendix D Per-round Self-attribution Breakdown
• Appendix E Broader Impact
• Instructions for reporting errors