AlphaGo Moment for Model Architecture Discovery — part 1/3
Cleaned-text code points [0, 15783). Bibliography omitted here; source.md includes it.
# AlphaGo Moment for Model Architecture Discovery
###### Abstract
While AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity, creating an increasingly severe development bottleneck. We present ASI-Arch , the first demonstration of Artificial Superintelligence for AI research (ASI4AI) in the critical domain of neural architecture discovery—a fully autonomous system that shatters this fundamental constraint by enabling AI to conduct its own architectural innovation. Moving beyond traditional Neural Architecture Search (NAS), which is fundamentally limited to exploring human-defined spaces, we introduce a paradigm shift from automated optimization to automated innovation . ASI-Arch can conduct end-to-end scientific research in the challenging domain of architecture discovery, autonomously hypothesizing novel architectural concepts, implementing them as executable code, training and empirically validating their performance through rigorous experimentation and past human and AI experience. ASI-Arch conducted 1,773 autonomous experiments over 20,000 GPU hours, culminating in the discovery of 106 innovative, state-of-the-art (SOTA) linear attention architectures. Like AlphaGo’s Move 37 that revealed unexpected strategic insights invisible to human players, our AI-discovered architectures demonstrate emergent design principles that systematically surpass human-designed baselines and illuminate previously unknown pathways for architectural innovation (Fig. 2 ). Crucially, we establish the first empirical scaling law for scientific discovery itself—demonstrating that architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process . We provide comprehensive analysis of the emergent design patterns and autonomous research capabilities that enabled these breakthroughs, establishing a blueprint for self-accelerating AI systems. To democratize AI-driven research, we open-source the complete framework, discovered architectures, and cognitive traces.
## 1 Introduction
Artificial Intelligence (AI) is impacting human society with unprecedented depth and breadth, and is widely regarded as a key driver of civilization’s progress Russell and Norvig (2010) ; Agrawal et al. (2018) ; Brynjolfsson and Mitchell (2017) . However, a fundamental paradox emerges: while AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity The White House (2023) ; Ahmed et al. (2022) ; Sevilla et al. (2022) . This human-centric development model creates an increasingly severe bottleneck for AI advancement, where the velocity of innovation is constrained not by computational power, but by human research bandwidth . This motivates a transformative vision: Artificial Superintelligence for AI research (ASI4AI) —AI systems capable of autonomously conducting their own scientific research and designing more powerful next-generation models.
Neural architecture discovery stands as the most challenging and impactful frontier for realizing ASI4AI. Model architecture serves as the cornerstone of the AI technology stack, with each major leap in AI capabilities—from image recognition to natural language understanding—accompanied by corresponding architectural breakthroughs. The evolution from CNNs LeCun et al. (1995) to Transformers Vaswani et al. (2017) exemplifies how architectural innovation drives fundamental progress in AI. At the forefront of current research, a pivotal challenge involves enhancing computational efficiency while maintaining expressive power DeepSeek-AI et al. (2024) ; MiniMax et al. (2025) ; Yuan et al. (2025) . To ground our exploration in a domain of both fundamental importance and active research, we focus on attention-based architectures as our testbed, leveraging their extensive knowledge base to explore AI’s true architectural design potential Katharopoulos et al. (2020) ; Choromanski et al. (2020) ; Tay et al. (2022) ; Wang et al. (2020) .
Moving beyond traditional Neural Architecture Search (NAS), which is fundamentally limited to exploring human-defined spaces, our work represents a paradigm shift from automated optimization to automated innovation. While previous NAS methods Zoph and Le (2016) ; Real et al. (2017) ; Elsken et al. (2019) ; Cheng et al. (2025) could only optimize over predetermined building blocks at prohibitive computational costs, acting as sophisticated selection algorithms rather than creative agents, we present ASI-Arch —the first demonstration of ASI4AI in neural architecture discovery. Leveraging the advanced reasoning and coding capabilities of modern LLMs Brown et al. (2020) ; OpenAI (2023) ; Li et al. (2022) , ASI-Arch transcends human-designed search spaces by autonomously hypothesizing novel architectural concepts, implementing them as executable code, and empirically validating their performance through rigorous experimentation Chen et al. (2023) ; Zhang et al. (2024) .
This represents AI’s first demonstration of genuine scientific superintelligence in neural architecture design. Like AlphaGo’s Move 37 that revealed strategic insights invisible to human players, ASI-Arch discovers architectural principles that systematically surpass human intuition. After conducting 1,773 autonomous experiments over 20,000 GPU hours, ASI-Arch successfully discovered 106 novel, state-of-the-art linear attention architectures. Crucially, we establish the first empirical scaling law for scientific discovery itself—demonstrating that architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process and providing a concrete pathway toward ASI4AI.
Our primary contributions establish a blueprint for self-accelerating AI systems and advance this paradigm: • ASI4AI Framework: We design and build the first demonstration of Artificial Superintelligence for AI research through a highly autonomous, tool-centric multi-agent system that enables AI to independently conduct the entire scientific research process—from hypothesis generation to empirical validation—in neural architecture discovery.
• Emergent Design Intelligence: Through comprehensive analysis, we identify novel design patterns that emerge from AI-driven discovery, demonstrating qualitatively different architectural intelligence that expands beyond human design paradigms and establishes new principles for attention mechanism innovation.
• Computational Scaling of Discovery: We discover 106 novel, state-of-the-art linear attention architectures and establish the first scaling law for automated scientific breakthroughs, proving that research progress can be scaled with computational resources rather than human expertise. We open-source the complete framework, discovered architectures, and cognitive traces to democratize AI-driven research.
## 2 Related Work
#### AI For AI Research
The application of artificial intelligence to advance AI research itself represents a compelling frontier Kokotajlo et al. (2025) , best understood as a spectrum of increasing AI autonomy within the scientific process. Initially, AI’s role resembled that of a sophisticated assistant, handling specific tasks like code generation in a “copilot” model where human researchers retained full control of the research direction. The collaboration has since evolved toward AI as an “AI scientist” capable of independently generating novel hypotheses and proposing promising research ideas for human consideration Tshitoyan et al. (2019) ; Boiko et al. (2023) . More recently, several examples have demonstrated AI’s ability to navigate the entire research cycle with minimal human intervention. Frameworks such as AlphaEvolve Novikov et al. (2025) ; Cheng et al. (2025) , for instance, employ LLMs to iteratively mutate and select improved program variants, completing a full loop of discovery and refinement. Similarly, AlphaGeometry’s success in autonomously discovering mathematical proofs showcases a high degree of research autonomy from problem statement to solution Trinh et al. (2024) ; Chervonyi et al. (2025) . As the proportion of human involvement in this collaborative loop decreases, the potential for AI’s self-optimization becomes increasingly central. This concept is epitomized by self-referential systems like Darwin-Gödel machines Zhang et al. (2025) , which are designed to iteratively modify their own code and empirically validate these changes, marking a clear trajectory toward fully self-improving systems Schmidhuber (1997) ; Baum (2004) .
Building upon this path, ASI-Arch applies the principles of AI self-evolution to the highly complex domain of neural architecture design. This presents a greater challenge than prior self-improving systems, as architectural exploration involves a significantly more complex experimental environment and a vast search space where success is not guaranteed. Our work therefore represents a significant attempt to advance AI self-evolution in this more demanding and impactful frontier.
#### Efficient Architecture
The Transformer architecture has dominated sequence modeling since its introduction, but its quadratic attention complexity has catalyzed extensive research into sub-quadratic alternatives, creating an increasingly complex design space Vaswani et al. (2017) . Among these alternatives, sparse attention approaches like Native Sparse Attention (NSA) Yuan et al. (2025) employ hierarchical sparse strategies to achieve substantial speedups while maintaining model capabilities. Beyond sparse attention, three principal families have emerged with linear time complexity: Linear Attention, which uses linearizing feature maps Katharopoulos et al. (2020) ; Choromanski et al. (2020) ; Qin et al. (2022) ; State-Space Models (SSMs) like Mamba, employing structured state transition matrices Gu and Dao (2023) ; Dao and Gu (2024) ; and Linear RNNs such as RWKV, with matrix-valued recurrent states Peng et al. (2023) ; Qin et al. (2023) ; Qin et al. (2024b) . The current trajectory points toward synthesis and hybridization, with architectures like Jamba interleaving different model families to leverage their respective strengths Lieber et al. (2024) ; Qin et al. (2024a) . This evolution has transformed the landscape from a single dominant design to a vast combinatorial space where optimal architectures are highly dependent on specific tasks and constraints. While existing work focuses on manually designing individual architectural components or families, this process is often protracted, requiring months of iterative effort from human experts to yield a single state-of-the-art architecture. In contrast, ASI-Arch uniquely addresses the systematic exploration of this complex design landscape through automated multi-agent collaboration, enabling the discovery of novel architectures that transcend traditional family boundaries.
## 3 Methodology
ASI-Arch framework operates as a closed-loop system for autonomous architecture discovery, structured around a modular framework with three core roles. The Researcher module proposes novel architectures, the Engineer module conducts empirical evaluations by executing them in a real-world environment, and the Analyst module performs analytical summaries of the results to acquire new insights. All experimental data and derived insights are systematically archived in a central database, creating a persistent memory that drives the entire process.
To ensure the system progressively generates superior designs, we implement an evolutionary improvement strategy that enables the model to continuously learn from experience. This is realized through two key mechanisms: first, a comprehensive fitness score that holistically evaluates each new architecture, providing a clear optimization target; and second, the ability to leverage both distilled knowledge from human expert literature (cognition) and analytical summaries of its own past experiments (analysis) to inform subsequent design proposals. Given the resource-intensive nature of this evolutionary process, we adopt a two-stage exploration-then-verification strategy. The initial stage involves broad exploration on small-scale models to efficiently identify a large pool of promising candidates. In the final stage, these candidates are scaled up to larger models for rigorous validation, confirming their state-of-the-art performance.
### 3.1 The Fitness Function
ASI-Arch ’s model architecture evolution mirrors biological evolution, drawing insights from the principles of natural selection. In nature, fitness determines an organism’s survival and reproduction, and similarly, we define a fitness function that governs which architectures survive and propagate through our evolutionary process. A critical flaw in past approaches is their sole reliance on quantitative metrics like loss and benchmark scores. This narrow focus inevitably leads to reward hacking Amodei et al. (2016) , where the system learns to maximize scores without producing genuinely superior architectures. We expand this definition by incorporating a qualitative assessment of the architecture itself. Our composite fitness combines both quantitative and qualitative dimensions, holistically evaluating performance and design quality:
Fitness = Objective Performance ⏟ Quantitative + Architectural Quality ⏟ Qualitative \text{Fitness}=\underbrace{\text{Objective Performance}}_{\text{Quantitative}}+\underbrace{\text{Architectural Quality}}_{\text{Qualitative}} (1)
In our framework, the objective performance assessment evaluates both benchmark scores and loss performance relative to baseline architectures. Recognizing that scientific breakthroughs often emerge from incremental advances, we apply a sigmoid transformation to performance differences: σ ( Δ performance ) \sigma(\Delta_{\text{performance}}) . This transformation serves a dual purpose—amplifying small but potentially significant improvements while capping extreme values that could otherwise dominate the optimization process. For the architectural quality assessment, we introduce a separate LLM that acts as an expert evaluator, mimicking how a human specialist would judge architectural merit. This judge examines multiple dimensions: architectural innovation, structural complexity, implementation correctness, and convergence characteristics. By incorporating these qualitative assessments alongside quantitative metrics, we capture architectural qualities that resist simple numerical measurement. Our final composite fitness function thus takes the form:
Fitness = 1 3 [ σ ( Δ loss ) + σ ( Δ benchmark ) + LLM judge ] \text{Fitness}=\frac{1}{3}\left[\sigma(\Delta_{\text{loss}})+\sigma(\Delta_{\text{benchmark}})+\text{LLM}_{\text{judge}}\right] (2)
where σ ( Δ loss ) \sigma(\Delta_{\text{loss}}) and σ ( Δ benchmark ) \sigma(\Delta_{\text{benchmark}}) represent sigmoid-transformed performance improvements over baseline, and LLM j u d g e \text{LLM}_{judge} provides the subjective quality assessment normalized to [0,1].
### 3.2 Researcher: Propose New Architecture
The Researcher module serves as the creative engine of our system, where AI independently proposes novel model architectures based on historical experience and human expertise. Our design targets two critical objectives: ensuring high-quality architectural innovations while preventing repeated explorations that squander computational resources. To achieve these goals, we implement four key mechanisms that work together:
#### Seed Selection