AlphaGo Moment for Model Architecture Discovery — part 2/3

Cleaned-text code points [15783, 31561). Bibliography omitted here; source.md includes it.

ASI-Arch maintains a candidate pool containing the top-50 highest-scoring architectures from all previous experiments. For each evolution step, we use a two-level sampling approach: one parent architecture is randomly selected from the top-10 performers to serve as the base for modifications, while 4 reference architectures are sampled from positions 11-50 to provide diverse design examples. This two-tier selection ensures that evolution builds on proven success while maintaining enough randomness to explore new directions. The parent architecture gets modified directly, while the reference architectures serve as examples of successful design patterns without being changed themselves.

#### Model Design

Before providing historical data to the Researcher for the next evolution cycle, we perform a crucial data summarization step. Each architecture in our database contains extensive information, including its motivation, implementation code, experimental results, and detailed analysis. To manage context size, we use a low-temperature LLM to generate brief, factual summaries of the natural language portions for each historical architecture. Critically, these summaries are generated on-the-fly for each evolutionary cycle and are not stored in the database. This dynamic summarization process inherently introduces subtle variations in how the same historical data is presented over time. This not only prevents the Researcher from receiving a static, repetitive context, which could limit the diversity of its outputs, but also actively encourages more varied design explorations. The final combined summaries provide the evolution model with both a manageable context and a constantly refreshed set of diverse insights to inform the next design attempt.

#### Program Implementation

Traditional approaches often separate architectural design from code implementation, using specialized models for each task. However, this separation creates a critical information gap—the implementation model, seeing only the high-level motivation, lacks the rich context that informed the original design. This often leads to implementation drift where the final code differs from the intended architecture. ASI-Arch addresses this by using a single agent for both tasks: first proposing the architectural motivation with full access to all context, then immediately implementing the corresponding code while maintaining complete awareness of the design reasoning.

#### Novelty and Sanity Check

To ensure that each proposed architecture is both novel and will be correctly implemented, we implement a two-stage validation process before it is accepted for training. The first stage is a similarity check to prevent redundancy. When a new architecture is proposed, we first extract its motivation and use embedding-based search to find the top-5 most similar historical motivations. A specialized LLM then evaluates whether the new proposal represents a genuine innovation or merely a variation of existing work. The second stage consists of code-level sanity checks to prevent fundamental implementation flaws, including verifying that the code does not exceed O ⁡ ( n 2 ) O(n^{2}) complexity and ensuring that masking is implemented correctly to prevent information leakage. If a proposal fails either the novelty or the correctness check, it is rejected, and the relevant feedback is returned to the agent to prompt a rewrite.

### 3.3 Engineer: Train and Evaluate Models

The evaluation process, which provides the data for the final fitness score, is composed of two parts: quantitative evaluation in a real code environment and qualitative scoring by an LLM-as-judge.

#### Real Code Environment

The quantitative evaluation takes place within an interactive coding environment where the agent must utilize a defined set of tools to initiate training, modify code, and inspect error logs. A key differentiator of ASI-Arch is its robust self-revision mechanism. In stark contrast to previous work Cheng et al. (2025) that often uses static analysis like Abstract Syntax Tree (AST) parsing and simply discards any architecture that fails these checks, ASI-Arch requires the agent to fix its own mistakes. When a training run fails due to an implementation error, the system automatically captures the full error log and delivers it back to the agent, which is then tasked with analyzing this feedback and revising its previously generated code. This iterative debugging loop continues until training is successful, ensuring promising ideas are not prematurely discarded due to simple coding mistakes. Furthermore, to maintain high efficiency, an automated quality assurance system monitors training logs in real-time. This is critical because some functional designs can be prohibitively inefficient, such as a model consuming two to three times the training duration of its peers. ASI-Arch detects such anomalies, as well as fundamental bugs indicated by abnormally low loss, and immediately terminates the run, reporting the issue back to the agent for revision. This proactive termination prevents wasting resources on flawed architectures and significantly accelerates the overall search process.

#### LLM-as-Judge Scoring

Following the quantitative evaluation, we initiate an LLM-based scoring module to provide a qualitative assessment. This scoring process considers not only the objective performance metrics but also the architectural complexity, computational efficiency, and the novelty of the proposed ideas, all benchmarked against baseline models. To ensure consistency and reproducibility, we provide a detailed syllabus in the prompt and slightly increase the model’s temperature, encouraging it to generate more detailed and nuanced justifications for its scores.

### 3.4 Analyzer: Mine Experimental Insights

To drive the evolutionary process, ASI-Arch provides the agent with two distinct sources of knowledge for each subsequent design step: cognition, derived from accumulated human expertise, and analysis, generated dynamically from the system’s own experimental history.

#### Cognition Base

To ensure ASI-Arch can leverage existing domain knowledge, we construct a cognition-centered knowledge base. We selected nearly 100 seminal papers from the field of linear attention and used a dedicated LLM to extract 1-3 distinct cognitions from each. Each cognition is a structured entry composed of three key elements: the applicable scenario, which describes the specific problem the original paper aimed to solve; the proposed algorithm, which summarizes the core technical solution; and the historical context, which situates the paper within the research trends of its time.

To guarantee the utility of this knowledge base, we carefully engineered the prompt for the extraction LLM. The prompt’s structure is specifically designed to ensure that the extracted “experiment trigger” align semantically with the “problem analyses” generated by our Analyst module. This alignment is crucial for effective retrieval. In the final stage of analysis, the Analyst summarizes the specific shortcomings observed in the current experiment, and this summary is used as a query for embedding-based retrieval against the scenarios in our knowledge base. The retrieved cognition content is then stored in our database for future reference, providing a highly relevant, information-dense, and targeted way for the Researcher module to find solutions.

#### Contextual Analysis

ASI-Arch generates its own insights through a dedicated Analysis Module driven by a large language model. This agent is provided with the complete set of information from the current experiment, including all performance metrics, training logs, and the performance of baseline models. Furthermore, to achieve an effect analogous to an ablation study, we also supply the data from the parent and sibling nodes of the current architecture in the phylogenetic tree. Based on the assumption that these nodes share significant structural similarities, we expect the agent to infer the specific contributions of individual modules by comparing the performance differences among these closely related architectures. The resulting analysis is then archived to inform subsequent design cycles.

### 3.5 Exploration-then-Verification Strategy

Given the resource-intensive nature of architecture evaluation, we adopt a two-stage exploration-then-verification strategy to maintain feasibility and efficiency. The underlying principle is that a truly superior architecture should demonstrate its advantages across different settings. Therefore, in the initial exploration stage, we use smaller models and resource-efficient protocols to rapidly identify a large pool of promising candidates that outperform a baseline. In the subsequent verification stage, only these promising candidates are scaled up with increased parameter sizes for extensive training and rigorous validation against established, state-of-the-art baselines. This two-stage approach allows us to balance the need for broad exploration with the necessity of thorough, high-fidelity validation.

## 4 Experiments

### 4.1 Experimental Setup

#### Fitness Function

Throughout our experiments, we use DeltaNet Yang et al. (2024b) as our baseline. As described in Section 3.1, our scoring system consists of three components. For quantitative scoring, since architectural differences in linear attention typically produce small performance variations, we focus only on differences within 10% of baseline and map these through a sigmoid function to obtain scores. Additionally, architectures with losses more than 10% below baseline are considered to have information leakage and are immediately discarded. For qualitative scoring, we establish a calibrated scale with DeltaNet at 5 points as the midpoint and Gated DeltaNet Yang et al. (2024a) at 10 points as the upper reference. The LLM evaluator assesses each architecture on this 1-10 scale.

#### Parallel Search Architecture and Policy

To accelerate discovery, we run numerous search processes in parallel. This is supported by a cloud-based database that stores all historical experiment results, allowing different agents to add or request data through API services and share all accumulated knowledge. To manage this parallel exploration effectively while encouraging diversity, we implement a strategic candidate pool update policy:

• Cold Start : At the beginning of our experiments, ASI-Arch conducts 200 explorations without database updates. This initial phase encourages the model to explore diverse architectural frameworks broadly rather than immediately converging on variations of early discoveries.

• Batched Updates : After the cold start period, we update the candidate pool only after every 50 new entries, rather than dynamically selecting the top-50. This batched approach ensures all agents work with stable reference sets during each exploration phase, promoting consistency while the initial delay fosters creativity.

#### Efficiency-Oriented Framework Settings

As outlined in our methodology, our search process is divided into two distinct phases with different configurations:

• Exploration Stage : To enable large-scale exploration, we constrain model size to approximately 20M parameters trained on 1B tokens. For evaluation, we sample only 500 examples per benchmark to balance assessment quality with computational efficiency.

• Verification Stage : The most promising candidates from the exploration stage are advanced to a rigorous verification phase. Here, we scale the architectures to 400M parameters and train them on 1B tokens for an initial performance validation. The top-performing models from this group are then selected for a final, more extensive training run on a 15B token dataset to comprehensively evaluate their capabilities against strong baselines.

### 4.2 Main results

To balance exploration efficiency with validation accuracy, we adopt a two-stage strategy: initial exploration on small models followed by verification on larger models. In the first stage, we conducted 1,773 explorations using 20M parameter models, consuming approximately 10,000 GPU hours. A partial phylogenetic tree of this first stage is shown in Figure 5 . From these experiments, we filtered architectures that outperformed DeltaNet at equivalent parameter scales in both loss and benchmark metrics, yielding 1,350 promising candidates. In the second stage, we scaled these candidates to 340M parameters matching DeltaNet’s configuration and filtered out architectures with excessive complexity or parameter counts. We then trained approximately 400 model architectures on 1B tokens, using 10,000 GPU hours. Among these, 106 architectures achieved state-of-the-art results, all of which are publicly available on our Model Gallery website for community reference. For final validation, we selected 5 top-performing architectures and trained them at 340M parameters setting on 15B tokens. These models were compared against DeltaNet, Gated DeltaNet, and Mamba2 under identical experimental settings. As presented in Table 1 , our models outperform almost all baselines on various benchmarks. The five architectures selected for this final validation are detailed below, each representing a distinct strategy for improving upon the DeltaNet baseline:

• Hierarchical Path-Aware Gating (PathGateFusionNet): This architecture introduces a hierarchical, two-stage router to resolve the trade-off between local and global reasoning. The first stage allocates budget between a direct copy path and a contextual pool, while the second stage distributes that contextual budget across short-range, long-range, and Delta-rule paths. It ensures stable gradient flow with a small, always-on residual connection and adds head-specific output gates for fine-grained local control.

• Content-Aware Sharpness Gating (ContentSharpRouter): This model addresses the challenge of creating a gate that is both content-aware and capable of making decisive (sharp) routing decisions. It fuses two key ideas: a content-aware gate that uses token embeddings and path statistics to inform its decision, and a learnable, per-head temperature parameter that allows the model to dynamically control the sharpness of the routing softmax, preventing premature gate collapse.

• Parallel Sigmoid Fusion with Retention (FusionGatedFIRNet): This architecture fundamentally changes the gating mechanism to break the “zero-sum” trade-off imposed by softmax. It replaces the single softmax router with parallel, independent sigmoid gates for each path. This allows the model to activate local and global paths simultaneously. It also enhances the Delta-rule with a learnable, per-head retention parameter, giving it a controllable memory horizon.

• Hierarchical Gating with Dynamic Floors (HierGateNet): This model employs a two-stage hierarchical gate to separate macro (local vs. global) and fine-grained routing decisions. Its key innovation is the use of dynamic, learnable floors for each path and head. This mechanism guarantees that no critical pathway (especially the Delta-path for long-range reasoning) is ever fully collapsed, adapting its minimum allocation based on the context.

• Adaptive Multi-Path Gating (AdaMultiPathGateNet): This design focuses on providing maximum control at the finest granularity. It implements a unified BalancedSparseGate that combines global, per-head, and per-token logits, allowing every path to be controlled at the token level. To prevent gate collapse, it uses a combination of a small epsilon-floor and a persistent, always-on entropy penalty, ensuring path diversity without complex training schedules.

## 5 Analysis