AlphaGo Moment for Model Architecture Discovery — part 3/3

Cleaned-text code points [31561, 47440). Bibliography omitted here; source.md includes it.

The evolution of architectures in ASI-Arch is driven by a candidate pool that is updated after every 50 new architectures are generated. Since each architecture mutation step exclusively references data from this pool, we analyze the search process sequentially according to the generation order, using a batch of 50 architectures as our fundamental unit of analysis. To facilitate our investigation into what distinguishes high-performing models, we collectively refer to the top 106 architectures as the “model gallery” .

### 5.1 Effectiveness of LLM-Driven Architecture Search

To demonstrate the effectiveness of our LLM-driven neural architecture search system, we examine how the search process evolves over time. Since our system exclusively selects parent architectures from the top-50 candidate pool for modification, the characteristics of this pool directly determine the search trajectory and ultimate performance. Therefore, we analyze two key sets of metrics related to this candidate pool: (1) both the overall trend of the average fitness score for the top-50 candidates and the individual trends of its three components: loss improvement, benchmark improvement, and the LLM judge score; and (2) the average raw performance, specifically the benchmark scores and loss values, of these same candidates. These metrics collectively provide a comprehensive view of our system’s search dynamics and continuous optimization process.

Analysis of the search dynamics reveals several complementary patterns. First, the average fitness score of the top-50 candidates follows a characteristic learning curve, with rapid initial gains that gradually stabilize Figure 6 b. The early-stage score increase is primarily driven by the optimization of the loss component. The subsequent stabilization is a direct result of our fitness function’s design; due to the sigmoid transformation, even significant performance gains in later stages are mapped to smaller score increases. This prevents reward hacking by capping the score contribution from any single metric and thus discouraging over-optimization. Importantly, while the fitness score growth flattens by design, the system does not encounter a performance bottleneck, as evidenced by the continued, steady improvement in the raw benchmark and loss metrics. This convergent evidence confirms that our LLM-driven search effectively learns to generate superior architectures throughout the search process.

### 5.2 Architectural Design Patterns

To understand the architectural preferences of LLMs during the search process which can provide insights into how these models approach the design space, we analyze both the complexity trends and component preferences.

#### Model Complexity Stability

A fundamental concern in neural architecture search is whether performance improvements come from simply increasing model size. We use parameter count as a proxy for model complexity to examine this issue. Figure 8 shows the distribution of parameter counts across iterations. The data reveals that while early iterations predominantly generate models in the 400-600M parameter range, the system quickly diversifies to explore models between 600-800M parameters. Importantly, after this initial exploration phase, the parameter distribution remains stable without systematic growth. The majority of architectures consistently fall within the 400-600M range throughout the search process, with no trend toward increasingly complex models. This stability demonstrates that ASI-Arch does not exploit complex component stacking as a simple strategy for performance improvement, maintaining architectural discipline even without explicit parameter constraints.

#### Architectural Component Preferences

To understand the LLM’s underlying design strategy, we performed a fine-grained analysis of the architectural components it chose to modify. We employed a separate Large Language Model to parse every motivation generated by the system, identifying which specific model components were targeted for modification in each step. This process yielded approximately 5,000 component instances, which we then manually curated and grouped into 40 high-level categories. We then statistically compared the proportional usage of these categories within our high-performing model gallery against that of the remaining models. This comparative analysis, visualized in Figure 7 , reveals two key insights into our LLM-driven design process. First, ASI-Arch shows a clear preference for established architectural components like gating mechanisms and convolutions, while less common ones like physics-inspired mechanisms appear infrequently, likely reflecting biases in the training literature. Second, and more revealingly, the model gallery exhibits a significantly less pronounced long-tail distribution in its component usage. This indicates that while the system explores many novel components, the top-performing models converge on a core set of validated and effective techniques. This mirrors the typical methodology of human scientists: achieving state-of-the-art results by primarily iterating and innovating upon a foundation of proven technologies, rather than pursuing novelty for its own sake.

### 5.3 Where Do Good Designs Come From?

To guide the future development of more efficient and adaptive frameworks, it is crucial to understand which components of ASI-Arch exert the most significant influence on model architecture design. Our system’s design process is constrained by its inputs: for each new architecture, the model’s context is strictly limited to the motivation, program, experiment result, analysis, and cognition sections of five historical experiment records drawn from the candidate pool. Given this bounded context, we can posit that any new design inspiration must originate from one of only three channels: knowledge distilled from human expert literature (which we term cognition), patterns identified through the analysis and summary of its related historical experiments (analysis), or novel ideas generated by the model itself (original). To quantify the contributions of these three channels, we designed an experiment to trace the provenance of each design idea. We prompted a LLM, acting as an impartial evaluator, to classify each architectural component (as identified in our prior motivation analysis) by its most likely origin, classifying it as derived from cognition, analysis, or as an original idea.

The results, presented in Table 5.3 , reveal a compelling two-fold pattern. Across the entire population of generated architectures, a majority of design ideas originate from the cognition phase, indicating a baseline reliance on direct, prior examples. However, a significant shift is observed when we focus exclusively on the model gallery. For these top-performing architectures, the proportion of design components attributed to the analysis phase increases markedly. This finding suggests a crucial parallel to human scientific progress: while competency can be built upon direct experience, achieving true excellence requires a deeper, more abstract level of understanding. It proves that for an AI to produce breakthrough results, it cannot merely reuse past successes (a reliance on cognition). Instead, it must engage in a process of exploration, summary, and discovery (a reliance on analysis) to synthesize novel and superior solutions.

## Discussion and Future Work

Our work successfully demonstrates a framework for AI self-optimization, where an autonomous agent can iteratively discover and refine novel neural architectures. The primary focus of this study was to establish the viability of this methodology—proving that an AI can navigate a complex design space to achieve state-of-the-art performance. Our findings open up several promising directions for future research.

#### Multi-Architecture Initialization

Our current approach initializes the search from a single, strong baseline (DeltaNet). This was a deliberate methodological choice, providing a clear objective and a stable foundation to drive continuous improvement, which is crucial in the early stages of exploring such a framework. A natural and exciting extension would be to initialize the process with a diverse portfolio of architectures simultaneously. This would not only test the framework’s ability to manage a more complex, multi-modal search but could also lead to the discovery of entirely new families of architectures. Such an endeavor would, however, demand a significant increase in computational resources and time.

#### Component-wise Analysis

Our experiments validate the effectiveness of our pipeline as a cohesive whole. Due to the substantial resources required for each design iteration, we did not perform a fine-grained ablation study to isolate the contribution of each component within the framework. A crucial avenue for future work is to dissect the pipeline from multiple angles to better understand the interplay and individual importance of its parts, such as the “cognition” and “analysis” modules. This would enable a more targeted optimization of the framework, potentially leading to even greater efficiency and creativity.

#### Engineering Optimization

The core contribution of this paper lies in the design of the AI-for-AI framework itself, with an emphasis on architectural innovation and performance. Consequently, we did not extend our work to include the labor-intensive task of writing custom accelerated kernels (e.g., using Triton) for the newly discovered architectures. As a result, a direct comparison of their computational efficiency is not provided. A critical next step, particularly for transitioning these designs from research to practice, would be to focus on this engineering aspect. Benchmarking the efficiency and latency of these models would be an invaluable follow-up study and would complete the cycle from automated discovery to practical deployment.

## Appendix A Experimental Setup

### A.1 Pipeline Configuration

#### Framework Overview

Our experimental framework implements an automated AI self-iterative system for exploring novel neural network architectures through three core phases: Evolve, Training, and Analysis. The system operates cyclically, extracting nodes from a MongoDB database, generating new motivations and implementations, conducting training and evaluation, and performing comprehensive analysis with knowledge integration.

#### Multi-Model Integration

We employ a hybrid multi-model approach to optimize both quality and efficiency. In the Evolve phase, we combine O3 and GPT-4.1 models for the planner component to balance motivation quality and generation speed while enhancing architectural diversity. The checker component utilizes O3 to ensure code validity and prevent resource waste, while motivation deduplication employs GPT-4.1 for rapid processing. During the Training phase, GPT-4.1 handles training initiation, testing, and debugging operations, focusing on detail-level modifications without structural changes for rapid iteration capabilities. The Analysis phase utilizes O3 to conduct comprehensive experimental analysis, providing high-quality insights to enhance subsequent exploration efficiency.

#### Data Management and Retrieval

For data management and retrieval, MongoDB serves as our primary storage solution, supporting name-based and sequential storage along with deletion functionality for experimental nodes. FAISS enables efficient similarity matching during motivation deduplication, identifying similar concepts in the database before agent-based verification to improve exploration efficiency. We extract cognitive insights from relevant literature and employ OpenSearch for RAG-based retrieval. For each experimental result, we extract the three most similar cognitive entries, integrating them into the experimental node for enhanced future exploration.

### A.2 Experimental Configuration

#### Progressive Evaluation Strategy

To balance exploration efficiency with computational constraints, we implement a three-tiered progressive evaluation approach with rapid architecture exploration using 20M parameter models, followed by validation phases at larger scales.

#### Model Architecture

The base 20M configuration employs 8 attention heads across 8 hidden layers with a hidden dimension of 256, maintaining computational tractability while preserving architectural expressiveness. The model uses SiLU activation for query-key transformations with L2 normalization and incorporates short convolutions with a kernel size of 4. For comparison, 340M parameter models employ 1024 hidden size, 8 attention heads, and 24 hidden layers with tied word embeddings disabled.

#### Training Protocol

We utilize the FLAME framework with AdamW optimization, employing a peak learning rate of 3 ​ × ​ 10 − 4 3×10^{-4} , epsilon value of 1 ​ × ​ 10 − 8 1×10^{-8} , and a warmup-stabilize-decay (WSD) learning rate schedule. Training proceeds for 2000 steps with 1000 warmup steps for 20M models, using mixed precision training with bfloat16 parameters and float32 gradient reduction. All models maintain a consistent batch size of 256 and employ GPT-2 tokenizer throughout training and evaluation phases.

#### Data Configuration

Training utilizes FinewWeb-edu sample-10BT and sample-100BT datasets with a context length of 2048 tokens. For comprehensive evaluation of discovered architectures, we scale to 340M parameter models that provide more reliable performance assessment. The 340M models are trained using 15 billion tokens, employing the same cosine learning rate schedule with a warm-up phase of 0.5 billion tokens, maintaining identical training hyperparameters to ensure consistent evaluation.

#### Evaluation Protocol

Model evaluation employs the LM-Evaluation-Harness framework, a standardized open-source tool developed by EleutherAI that provides unified benchmarking protocols for language models. The evaluation suite encompasses diverse cognitive capabilities including reading comprehension (LAMBADA, SQuAD), commonsense reasoning (HellaSwag, PIQA), knowledge-intensive tasks (ARC-Challenge, ARC-Easy, OpenBookQA), boolean question answering (BoolQ), and additional benchmarks (FDA, Social-IQA, SWDE, WinoGrande). During rapid exploration phase, we limit samples to 500 per dataset for 20M parameter models to accelerate architectural search, while validation phases utilize full datasets. All evaluations are conducted using consistent hyperparameters and random seeds to ensure reproducible comparisons across architectural variants. Final ranking incorporates LLM subjective evaluation, training loss metrics, and benchmark performance for comprehensive model assessment.

## Appendix B Prompts

### B.1 Planner

### B.2 Checker

### B.3 Debugger

### B.4 Analyser

### B.5 Cognition

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .