Harnessing Agentic Evolution — part 3/3
Cleaned-text code points [31637, 41657). Bibliography omitted here; source.md includes it.
In the performance-engineering take-home, the goal is to optimize a kernel for a simulated VLIW SIMD machine while preserving correctness on the hidden tests. We report raw cycle count in the main table, so lower is better, although the evaluator also exposes a normalized score derived from the cycle count.
##### Reporting Protocol.
For Terminal-Bench and ARC-AGI-2, we report Avg@3 across three independent runs, together with the first optimization round that reaches the best score. For open-ended optimization, we report Best@3 across three runs under a fixed evaluation budget, together with the first round that reaches the best result and the average dollar cost per optimization round. Open-ended runs are capped at 100 rounds and are early-stopped if they reach the known target or plateau for 25 consecutive rounds.
##### Cost Computation.
For a run with R R optimization rounds, the average dollar cost per round is $ / R = 1 R ∑ r = 1 R ( p in n in ( r ) + p cache n cache ( r ) + p out n out ( r ) ) , \$/R=\frac{1}{R}\sum_{r=1}^{R}\left(p_{\mathrm{in}}n^{(r)}_{\mathrm{in}}+p_{\mathrm{cache}}n^{(r)}_{\mathrm{cache}}+p_{\mathrm{out}}n^{(r)}_{\mathrm{out}}\right), where n in ( r ) n^{(r)}_{\mathrm{in}} , n cache ( r ) n^{(r)}_{\mathrm{cache}} , and n out ( r ) n^{(r)}_{\mathrm{out}} denote the input, cached-input, and output token counts in round r r , and p in p_{\mathrm{in}} , p cache p_{\mathrm{cache}} , and p out p_{\mathrm{out}} are the corresponding provider prices.
### C.2 Initialization Details
#### C.2.1 Initial Procedure in Procedure-Based AEvo
The initial procedure used by procedure-based AEvo is intentionally minimal. It selects the current best valid candidate as the parent, applies a single LLM rewrite step, and then invokes the fixed evaluator. The excerpt below shows the corresponding editable surface.
⬇
This initialization exposes a limited search surface: it uses score-based selection, a single-parent rewrite step, and no richer failure analysis. This makes later meta-edits easier to interpret because they modify these exposed handles rather than the evaluator.
#### C.2.2 Initial Agent in Agentic and Reasoning Tasks
For Terminal-Bench and ARC-AGI-2, the initial agent is a minimal ReAct-style agent. The prompt surface and the ‘step()‘ loop are both editable, but the seed version contains only basic reasoning, short-horizon memory, and a strict JSON+‘bash‘ action protocol.
⬇
This weakly structured initialization makes subsequent improvements easier to attribute to changes in prompts, memory management, recovery logic, or context organization rather than to a highly engineered seed agent.
#### C.2.3 Meta-Agent Skill Excerpt
The meta-agent is governed by a compact operational skill that constrains how it reads the workspace, attributes failure, and chooses one causal intervention at a time. A representative excerpt is shown below.
⬇
The meta-agent therefore acts as a controller over future search rather than as an additional task-facing worker. The skill keeps that role boundary explicit.
### C.3 Representative Evolved Artifacts and Outcomes
#### C.3.1 Representative Evolved Procedures
The excerpt below is taken from a high-performing ARC-AGI-2 procedure. In later rounds, the evolved procedure forwards not only the current best artifact but also per-task slices from alternative references, enabling the optimizer to formulate a single causal hypothesis from those diagnostics.
⬇
⬇
This excerpt illustrates the main leverage of procedure-mode AEvo : evolution changes the evidence presented to the optimizer and the way that evidence is structured, not only the candidate artifact being scored.
#### C.3.2 Representative Evolved Agent Harness
In the agent-based setting, the evolved object is the harness seen by future inner-agent sessions rather than a single submitted artifact. In the performance-engineering run, this harness contained at least five durable layers: a task skill, a session-specific goal, a persistent family map, support utilities for evaluation accounting, and structured session notes written back into the workspace.
##### Task skill.
⬇
##### Session goal.
⬇
##### Persistent family map.
⬇
##### Replay utility.
⬇
##### Session memory written back by the inner agent.
⬇
Together, these excerpts show that the evolved agent harness is not a single prompt edit. It is a layered control structure consisting of persistent instructions, hypothesis-carrying goals, accumulated family-level memory, support code for evaluator interaction, and structured records that are promoted into future sessions.
#### C.3.3 Representative Optimization Outcomes
##### ARC-AGI-2 best artifact.
The best ARC-AGI-2 artifact in our run reaches accuracy 0.35 0.35 ( 7 / 20 7/20 ). Structurally, the final agent has three persistent components: a prompt surface that distinguishes normal feedback refinement from “fresh exploration,” a local Pass@K scorer over cached training pairs, and a validate-versus-submit controller that uses the best local score and plateau state.
⬇
⬇
##### Analysis.
The improvement over the seed agent comes from a tighter coupling between search control and task-local verification. First, caching the training pairs removes a brittle dependency on the observation format at later steps, so local verification remains available throughout the interaction. Second, Pass@K sampling with prompt diversification converts a single-sample ReAct loop into a small search procedure over candidate solvers, with selection driven by observed agreement on the training pairs rather than by the raw LLM output alone. Third, the separation between FEEDBACK_SUFFIX and FRESH_EXPLORE_SUFFIX makes the agent alternate explicitly between exploitation and hypothesis reset: partial but improving candidates are refined through concrete failure-conditioned feedback, whereas persistent plateaus trigger a prompt regime that suppresses anchoring to the current local optimum. Finally, the validate-versus-submit controller ties action choice to the best verified score rather than to the latest response, which reduces premature submission of partially correct programs.
##### Performance-engineering best artifact.
The best performance-engineering artifact is a two-file submission. The top-level program is only a wrapper; the schedule-level optimization resides in a benchmark-specialized base that exposes a small set of round-family control points. The final validated artifact reaches 1138 1138 cycles by combining evaluator-compatible packaging, explicit specialization of the benchmark rounds, and a non-uniform assignment of selector logic across engines.
##### Submitted wrapper.
⬇
##### Parameterized benchmark family.
⬇
##### Core benchmark schedule.
⬇
⬇
⬇
##### Analysis.
The low cycle count comes from three coupled changes. First, the file-local importlib wrapper is not cosmetic: without it, the evaluator rejects the whole family because sibling modules are not imported through the workspace package path. The wrapper therefore preserves a modular implementation while keeping the submission evaluator-compatible. Second, the benchmark-specialized base restructures the kernel around the exact round pattern of the benchmark rather than a generic loop. Its control flow is emitted as four specialized top-of-tree rounds, followed by seven gather rounds, followed by the same four specialized rounds and a final gather; this removes generic control overhead where the traversal repeatedly revisits the upper levels of the tree and makes the period- 11 11 reuse pattern directly schedulable. Within each specialized round, emit_hash is emitted phase-major across tiles, which exposes many independent chains to the scheduler and improves overlap between alu , valu , and memory operations. Third, the remaining gains come from engine balancing rather than from further structural refactoring. On this VLIW benchmark, flow has only one slot per cycle, so replacing a binary selector by multiply_add is profitable only when the induced valu pressure stays below the new bottleneck. The progression recorded in the run data is consistent with this view: the explicit-family port yields the dominant improvement ( − 597 -597 cycles), additional madd -based selectors at depth 1 and depth 3 save another 29 29 cycles, an alu -based parity/XOR rebalance saves 8 8 , and the final artifact gains the last 2 2 cycles by reverting only the second specialized depth-1 selector back to vselect . The best artifact is therefore not simply a shorter program; it is a schedule in which specialization, phase ordering, and per-round engine placement are co-tuned to the simulator’s slot limits.
Taken together, these artifacts illustrate two modes of durable improvement in AEvo : benchmark tasks improve through changes in how the optimizer reasons over failures, whereas open-ended optimization improves through the preservation and recombination of low-level implementation knowledge across many sessions.
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .