Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns — part 1/4
Cleaned-text code points [0, 15967). Bibliography omitted here; source.md includes it.
# Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
###### Abstract
Neural scaling laws for transformer language models predict smooth improvements in pretraining loss with increasing parameters, but downstream capabilities such as in-context learning are known to emerge abruptly past a certain model scale. In this paper, we show that emergent capabilities arise stochastically throughout training, with larger models acquiring them earlier on average. We demonstrate that the emergence of capabilities such as pattern completion and indirect object identification corresponds to the abrupt learning of task-relevant attention patterns. To isolate this phenomenon, we train transformer models on synthetic linear map and cellular automata datasets, and we show that the difficulty of learning attention patterns depends on context length and pattern sparsity. Moreover, scaling the number of attention heads improves learning efficiency on our synthetic tasks, while increasing the head dimension yields diminishing returns past a minimum capacity. We additionally investigate architectures with alternative attention mechanisms, showing that MLP-Mixer outperforms a transformer on linear map tasks with complex attention patterns. Our findings provide a mechanistic insight into emergence, showing that downstream capabilities arise abruptly due to the intrinsic difficulty of learning sparse attention patterns in transformer models.
## 1 Introduction
As transformer language models grow in parameter count, they acquire emergent capabilities not seen in smaller models ( Wei et al., 2022 ) ; however, it remains unclear what changes inside a model when a capability emerges. At a fixed scale, whether a capability appears by the end of training depends on the initialization seed ( Zhao et al., 2026 ) , and skills such as syntax acquisition and in-context learning emerge abruptly within a short window of training ( Chen et al., 2025 ; Olsson et al., 2022 ) . Are there interpretable mechanisms behind emergence?
In this paper, we show that the emergence of capabilities in language models is driven by learning task-relevant attention patterns . At a fixed model scale, a capability may arise at variable points during training or even fail to arise depending on the model initialization. When emergence does occur, it coincides with the acquisition of task-relevant attention patterns, often across multiple heads (Figure 1 ). Larger models acquire these attention patterns earlier and more reliably across initializations, which we argue is the mechanism behind the higher emergence rates reported by Zhao et al. (2026) . We also demonstrate that attention learning is the key bottleneck to emergent capabilities (Section 2 ), as patching learned attention maps into an earlier training checkpoint can recover most of the performance for a given capability.
In natural language, there is no single ground-truth attention pattern, making it impossible to systematically isolate how factors such as context length and sparsity affect the difficulty of learning attention patterns. We study these factors in a controlled setting by training transformer models on synthetic linear map and cellular automata datasets (Section 3 ). Notably, we find that varying the context length or attention pattern sparsity dictates whether a model solves a task completely or makes no progress at all. We then investigate the role of architectures in learning attention patterns in Section 4 . Scaling width yields consistent benefits on both of our attention tasks, particularly when increasing the number of heads given a sufficiently high head dimension. We also investigate alternative architectures and find that MLP-Mixer ( Tolstikhin et al., 2021 ) , which uses an MLP for its token-mixing mechanism rather than dot-product attention, outperforms a transformer by an order of magnitude in learning the ground-truth attention pattern for our linear map task. Although we only observe this improvement on a single synthetic task, it suggests a tantalizing possibility of designing architectures with more sample-efficient token-mixing mechanisms than transformers.
Through our emergence analysis on Pythia models and training experiments on synthetic tasks, we observe that learning long-context, sparse attention patterns is a key bottleneck for eliciting downstream capabilities in transformer models. We see that various architectural choices such as scaling attention heads or applying entirely new token-mixing mechanisms can significantly improve a model’s ability to learn these patterns. Our findings prescribe multiple future research directions on designing datasets and architectures to improve a model’s ability to learn sparse attention patterns such as copying or induction heads, thereby removing a key bottleneck in the emergence of capabilities and enabling more efficient scaling of downstream performance.
## 2 Random Emergence of Capabilities in Language Models
While neural scaling laws ( Kaplan et al., 2020 ; Hestness et al., 2017 ) predict a smooth decrease in training loss as a function of model size, certain downstream capabilities are known to emerge both abruptly at certain model scales ( Wei et al., 2022 ) and randomly depending on the model initialization ( Zhao et al., 2026 ) . Specifically, Zhao et al. (2026) demonstrate that model performance on many downstream tasks follows a bimodal distribution: different models at a fixed scale will either reliably solve a task or fail entirely depending on the random initialization seed, with the success probability increasing at larger model scales. This observation suggests that although language model training dynamics are smooth across training steps, the acquisition of certain capabilities may also occur randomly throughout training. We confirm this hypothesis and demonstrate that sharp, random emergence of capabilities during training occurs when the model learns core attention patterns relevant to a certain task.
All experiments in this section are conducted with the Pythia suite of language models ( Biderman et al., 2023 ) . Each set of Pythia checkpoints contains model weights at initialization, at each step corresponding to powers of 2 between 1 1 and 512 512 , and then at every 1000 1000 steps up until the final checkpoint at step 143000 143000 . We study the models with 14M, 31M, 70M, 160M, and 410M parameters, with ten sets of checkpoints for each model scale corresponding to different random initializations.
### 2.1 Searching for Emergence During Training
We first evaluate how often Pythia models can solve a given task by the end of training. We construct a task based on a single sample, where the model is provided with a prefix and must predict the correct next token. Our analysis focuses on single-sample evaluations due to slight variations in emergence times when introducing multiple samples. We provide additional details on our motivation behind single-sample evaluations and confirm our findings apply to multi-sample evaluations in Appendix A.2 . We define a capability as having emerged for a given model size and initialization seed if the most likely next token (i.e., the model output under greedy sampling) matches the ground-truth output at the final training checkpoint. Formally, emergence occurs when arg max y ∈ 𝒱 p ( y | x ) = y ∗ \arg\max_{y\in\mathcal{V}}p(y\ |\ x)=y^{*} given a model with token vocabulary 𝒱 \mathcal{V} where y ∗ y^{*} is the ground-truth output given prefix x x . For the repetition task in Figure 1 , emergence corresponds to the model predicting “9” given the prefix “1 …9 1 …8”.
If a model does learn to solve a given task, we search for the point during training at which the capability first emerges. We define the point of emergence as the smallest t t such that arg max x ∈ 𝒱 p t ( y | x ) = y ∗ \arg\max_{x\in\mathcal{V}}p_{t}(y\ |\ x)=y^{*} , where p t ( x ) p_{t}(x) is the model prediction at training step t t . We assume that after the point of emergence, the model will not revert to another incorrect prediction at a later timestep t ′ > t t^{\prime}>t , and we verify this claim holds empirically. With this assumption, we apply binary search (rather than evaluating all 154 checkpoints) to find the first checkpoint where arg max x ∈ 𝒱 p t ( y | x ) = y ∗ \arg\max_{x\in\mathcal{V}}p_{t}(y\ |\ x)=y^{*} . Because we can only evaluate p t ( y | x ) p_{t}(y\ |\ x) at specific checkpoints, we estimate t t as the midpoint of the first checkpoint step with emergence and the checkpoint prior to it. We apply this approach to compute the point of emergence for all ten model seeds at each scale. Figure 1 (left) illustrates the results.
At smaller model scales, some models successfully learn the repetition task, but for specific initialization seeds the capability never emerges. As previously demonstrated by Zhao et al. (2026) , we see that the rate at which models acquire a capability increases with scale; at the 160M and 410M parameter scales, the task is solved by all ten models. Notably, we also see that the same capability can emerge randomly at different points throughout training, and larger models acquire capabilities earlier on average.
### 2.2 Learning Core Attention Patterns Enables Abrupt Emergence
In Figure 1 (right) we plot p t ( y ∗ | x ) p_{t}(y^{*}\ |\ x) , the next-token probability the model assigns to the ground-truth answer, as a function of training steps t t . We see that p t ( y ∗ | x ) p_{t}(y^{*}\ |\ x) remains very small up until the point of emergence, and then abruptly jumps between the point of emergence and the previous training checkpoint. Given that emergence occurs abruptly, we investigate how a language model’s internal representations change before and after acquiring this capability.
#### Causal Attention Head Ablation.
When training transformers on algorithmic tasks, there is often a fixed ground-truth attention pattern the model must learn. Gopalani and Hu (2025) show that learning this ground-truth attention map is a major bottleneck that leads to abrupt learning, and biasing the model’s attention map toward this optimal solution significantly accelerates training convergence. While there is no single ground-truth attention pattern for next-token prediction on natural language, we hypothesize that the abrupt learning we observe on individual capabilities is also due to sudden changes in the model’s attention weights. We validate this hypothesis by tracking how each attention head in the language model influences the sudden increase in p t ( y ∗ | x ) p_{t}(y^{*}\ |\ x) . We consider the first checkpoint that exhibits emergence as the post-emergence model, and the preceding checkpoint as the pre-emergence model. For each attention head at every layer, we observe the head’s attention map computed on the prefix x x in the post-emergence model. We then input the same prefix to the pre-emergence model, while fixing the attention pattern for that head to be that of the post-emergence model. This approach is effectively activation patching ( Heimersheim and Nanda, 2024 ) on the dot-product attention scores. We now test whether the attention patterns alone can explain the sudden boost in model capability.
We record the change in p t ( y ∗ | x ) p_{t}(y^{*}\ |\ x) resulting from patching each individual head in the pre-emergence model, and select the top K = 16 K=16 heads with the largest change Δ p \Delta p . We refer to these top K K heads as causal attention heads . Figure 1 (top right) visualizes the change in attention patterns of two such heads. We see both attention heads transition from a roughly uniform map to interpretable previous-token and copy patterns, which together enable the model to solve the repetition task. As we show in Figure 2 , patching the attention heads alone recovers the performance gap between the pre-emergence and post-emergence models. The effectiveness of patching depends on our choice of K K ; rather than attributing the rapid increase in p t ( y ∗ | x ) p_{t}(y^{*}\ |\ x) to a single head, we find that emergence jointly depends on multiple attention heads learning complementary patterns. For example, in Figure 1 both the previous-token and copy head are necessary to solve the repetition task. We provide additional details regarding our causal ablation implementation in Appendix A .
### 2.3 Emergence Across Multiple Tasks
We extend the string copying experiment illustrated in Figure 1 to three more tasks: in-context repetition, pattern completion, and indirect object identification. Figure 2 visualizes the results. We provide a complete interpretation of individual task results, along with their relevance to prior work ( Olsson et al., 2022 ; Michaud et al., 2024 ; Wang et al., 2022 ) in Appendix A.3 .
#### General Trends.
In all of our emergence experiments (Figures 1 and 2 ), we observe multiple behaviors consistent with prior work: the emergence of capabilities in a given model depends heavily on the model’s initialization ( Zhao et al., 2026 ) , the rate of emergence improves with model scale, and individual capabilities are acquired through discrete phase changes rather than continuously ( Michaud et al., 2024 ) . We additionally demonstrate that capabilities emerge at varying points throughout training depending on the initialization, emergence coincides with learning at least one interpretable task-relevant attention pattern, and larger models acquire capabilities more often and at earlier training steps. Our results indicate that emergent capabilities in language models arise randomly and abruptly over the course of training and depend on learning core attention patterns without which the model could not solve the task.
Our causal ablations additionally show that attention pattern learning is a bottleneck for language model capabilities, as patching learned attention patterns can elicit emergence in an earlier checkpoint. Moreover, the causal contribution is not isolated in any single head, but rather distributed across many heads. These ablations suggest that the improved rate of emergence in larger models may relate to their increased efficiency in learning attention patterns. We now investigate this hypothesis by extensively studying attention pattern learning in synthetic tasks where the ground-truth pattern is fully specified.
## 3 Learning Synthetic Attention-Based Tasks
Our experiments in Section 2 show that learning attention patterns drives the abrupt acquisition of capabilities in language models, and larger models learn these patterns faster and more reliably. However, there is no single ground-truth attention pattern in natural language tasks, so we cannot directly study the effects of factors such as attention sparsity and context length. We now train transformers on two synthetic attention-based tasks, linear maps and cellular automata, to study these factors in detail.
### 3.1 Task Definitions
We follow a standard language model training setup with a next-token-prediction objective, except instead of text we train models on one of two synthetic tasks. For both tasks, we define a state size S S , a trajectory length T T , and a number of colors C C . We sample an initial state x 0 = { 0 , … , C − 1 } S x_{0}=\{0,\dots,C-1\}^{S} and evolve the state T − 1 T-1 times with a recursive function x t + 1 = f ( x t ) x_{t+1}=f(x_{t}) to produce a trajectory [ x 0 , x 1 , … , x t − 1 ] [x_{0},x_{1},\dots,x_{t-1}] . We then train an autoregressive language model with vocabulary size C C on the flattened sequence of S T ST tokens, where the minimum loss is at least S − 1 S T ln C \frac{S-1}{ST}\ln C as the model cannot predict the next tokens of the initial random state. We provide a table with all the notation used for synthetic tasks in Appendix B.1 for clarity.
#### Linear Map.