Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns — part 3/4

Cleaned-text code points [31878, 47576). Bibliography omitted here; source.md includes it.

Prior work establishes that emergence is abrupt ( Wei et al., 2022 ) , random across initializations ( Zhao et al., 2026 ) , and connected to attention ( Olsson et al., 2022 ; Chen et al., 2025 ) . We unify and extend these observations in four directions. First, we show emergence is random across training steps , not only across initialization seeds, and we pinpoint the step at which it occurs for multiple models. Second, we provide causal evidence showing that patching a handful of attention heads elicits a capability before it arises naturally. Third, we design synthetic tasks with known ground-truth patterns to isolate context length and sparsity as the factors that govern how hard a pattern is to learn. Finally, we show that architectural choices such as attention head count and the token-mixing mechanism directly control attention-pattern learning, with MLP-Mixer learning our linear map task an order of magnitude faster than a transformer.

## 6 Conclusion

Emergent capabilities in language models arise randomly and abruptly because learning sparse attention patterns is inherently difficult for transformers. Our causal ablations on Pythia models show that patching a small set of attention heads can elicit a capability before it naturally emerges, demonstrating that attention pattern learning is a bottleneck for emergence. Our synthetic training experiments reveal that the difficulty of learning an attention pattern depends on context length and pattern sparsity. Scaling head count accelerates attention pattern learning, likely because more heads provide more candidates for discovering the correct pattern. We show alternative architectures like MLP-Mixer outperform transformers specifically when the task requires learning a complex positional attention pattern. Our findings prescribe multiple potential future directions for long-context training: attention interventions during training, attention mechanisms with an inductive bias towards sparsity, and distillation of attention maps from stronger models.

Our Pythia experiments in Section 2 are limited to models below 1B parameters, and the tasks we study involve relatively elementary capabilities. Whether our findings generalize to more sophisticated capabilities and larger models is an open question. Additionally, our synthetic tasks involve positional attention patterns; context-dependent patterns such as induction heads may exhibit qualitatively different properties. We also believe synthetic pre-pretraining ( Hu et al., 2025 ; Lee et al., 2026 ) may accelerate the attention pattern search by seeding the model with useful inductive biases before natural language training. We also provide preliminary results for testing whether pre-pretraining affects the acquisition of emergent capabilities in Appendix C . We hope these results motivate future work on designing datasets, architectures, and training procedures that make attention pattern learning faster and more reliable.

#### Acknowledgements.

We thank Lily Li and Martin Marek for helpful discussions. We also thank NSF CAREER IIS-2145492, NSF CDS&E-MSS 2134216, and DARPA AIQ HR00112590066 for support.

## Appendix A Emergent Capabilities in Pythia Language Models

### A.1 Causal Ablations and Attention Sinks

We ablate an individual attention head as follows. In the pre-emergence model, the output for an attention head at a given head and layer is S ​ V SV , where S ∈ ℝ N × N S\in\mathbb{R}^{N\times N} is the attention score matrix computed by softmax over key-value dot products, and V ∈ ℝ N × D V\in\mathbb{R}^{N\times D} is the set of value vectors for the given head. We instead compute S ′ ​ V S^{\prime}V , where S ′ S^{\prime} is the attention score matrix for the given head resulting from passing the prefix x x to the post-emergence model. We then compute the forward pass through the rest of the layers in the pre-emergence model, and measure the change in the output token probability for the ground-truth token. We select causal heads by applying this ablation to each individual head, and sorting by the change in correct token probability.

When applying our causal ablation, especially in early checkpoints, many top causal heads exhibit attention sink behavior [ Xiao et al., 2024 ] where every query attends to the first token. We filter these out from our analysis before applying the top- K K filtering to focus on heads that display interpretable behavior. One way to define attention heads is by testing whether every query token attends to key position 0 with the highest weight. However, we find that this is not sufficient to filter out sink-like behavior as attention sinks gradually emerge over time; this criteria does not include heads which are in the process of becoming attention sinks but do not yet place maximal attention weight on token 0. We instead define an attention sink in the context of our ablation as a head where the attention weight to key position zero increases for each query token between the pre-emergence and post-emergence checkpoints. If the attention weight to position zero for an attention head increases for all queries in our sample, we discard this head from our analysis. This allows us to find attention pattern transitions where a head learns to attend to some token position other than 0 for any query.

### A.2 Analyzing Emergence on Multiple Samples

In Section 2 we show that the model’s probability of predicting the correct next token for a single sample in a given task spikes abruptly throughout training. Although this is the case for individual samples, we find that emergence times vary for different samples within the same task, which makes it difficult to probe for causal attention heads and measure trends across model scale. We speculate that this may be due to distinct circuits in the model that decide when to apply an algorithm on a given sample and the specific algorithm to apply; the former may be learned at different rates even if the latter is learned at once. To test this hypothesis, we run our emergence analysis on multiple samples of the copying task, and observe how many causal attention heads are consistently shared across tasks. Specifically, we apply our forward causal head ablation on individual samples, track the K = 16 K=16 top heads for each sample, and measure the degree to which the top K = 16 K=16 heads overlap across pairs of samples.

We measure overlap using the intersection over union (IOU) of causal head sets; Table 1 shows the results. We see that there is a high degree of overlap for causal attention heads in the repetition task for most models — much higher than what we would expect if the model were utilizing random heads for each sample. Note we only include seeds for which the model exhibits the copying capability across all samples.

### A.3 Interpreting Causal Attention Mechanisms

In Figures 1 and 2 we apply our emergence analysis across four tasks: copying, in-context repetition, pattern completion, and indirect object identification. Below, we interpret the qualitative findings in each figure and how they relate to established prior results:

#### Copying.

In the copying task we provide the model with a string followed by all but the last token of the same string as context, and we evaluate whether the model can predict the last token of the repeated string. For models that learn to solve the copying task, the capability emerges sharply, and two of the causal attention heads display patterns corresponding to copying and attending to the previous token.

#### In-Context Repetition.

Olsson et al. [2022] empirically demonstrate the presence of induction heads in language models, which implement a primitive algorithm used for in-context learning. Induction heads are responsible for completing the sequence [A] [B] …[A] → \rightarrow [B] , by searching for previous instances of phrase [A] and predicting that [B] will follow after. In our example, [A] and [B] are the strings “atten-” and “tion” respectively. We see this repetition capability emerges rather early during training even for smaller models, and performance on this task follows a phase change-like transition as shown by Olsson et al. [2022] .

#### Pattern Completion.

The quantization model of neural scaling [ Michaud et al., 2024 ] posits that the smooth aggregate performance of neural networks can be decomposed into discrete skills (quanta) that are learned abruptly, and the authors find that one empirical quanta in Pythia models corresponds to predicting next entries of numbered lists. We confirm this skill is learned abruptly in Figure 2 . We additionally observe that larger models acquire this skill faster even when trained on the same data distribution and the discrete phase change is a result of the abrupt learning of attention patterns.

#### Indirect Object Identification.

We probe for indirect object identification using the example illustrated by Wang et al. [2022] . We visualize one selected attention head in Figure 2 , which roughly corresponds to the name-mover behavior demonstrated by Wang et al. [2022] in GPT-2-small. Even at the scale of GPT-2-small (124M), this capability is often not present in most model seeds. Note that the visualized head demonstrates the correction of an attention pattern; before emergence, the last query token places significant weight on “John”, but after emergence most of the weight collapses onto the correct token “Mary”.

## Appendix B Abrupt Learning on Synthetic Tasks

### B.1 Notation

In Table 2 we clarify notation for all of the symbols we refer to when describing our synthetic tasks.

### B.2 Attention Entropy

In Figure 8 we plot the loss and attention entropy across all eight heads in a single-layer transformer trained on the linear map task. We highlight a window of training steps in which the loss for a single row drops to zero. Within that same window, we see the attention entropy of two heads drop in tandem. As we show in Section 3.2 , these two heads learn interpretable attention patterns corresponding to a single row of the ground truth linear map.

### B.3 Attention Interventions

We apply an attention intervention similar to Gopalani and Hu [2025] by first defining a ground-truth attention pattern (see Figure 3 and Section 3 ) and artificially increasing a model’s attention scores based on this pattern. Figure 9 illustrates the results. Notably, with s = 8 s=8 and S = 16 S=16 a model makes almost no progress after 10,000 steps, while the intervention enables training convergence in under 1,000 steps. The same is true for cellular automata with S = 512 S=512 .

### B.4 What Variables Affect Loss Plateau Length?

In Section 3.3 we showed that long context lengths exacerbate loss plateaus during training, although we only ablated the effect of state size in Figure 5(c) . In this section, we also extend our context length analysis by varying the trajectory length T T , along with other cellular automata parameters which are orthogonal to the context length: number of rules N N , recursive depth k k , number of colors C C , and window size W W . Note we expect N , k , C , W N,k,C,W to still affect the overall difficulty of the task (for example, the amount of training required to achieve the minimum loss), but here we only study the initial loss plateau — the amount of training steps required to make any progress at all. Figure 10 illustrates our full sweep of the loss plateau length across various cellular automata configurations.

Unlike the relationship to state size shown in Figure 5(c) , loss plateau length does not always increase monotonically with trajectory length T T . With a state size of S = 64 S=64 and trajectory length T = 2 T=2 , for example, the model only ever sees S ⋅ ( T − 1 ) = 64 S\cdot(T-1)=64 cell transitions in a single sample, which is not nearly sufficient to exactly infer which of the N N rules is active from the context. Thus we see an initial drop in the loss plateau length in the top row of Figure 10 , as increasing T T initially provides more useful information to the model. However, increasing T T further makes the task more difficult through increasing the context length, hence the longer loss plateaus for large T T .

We see that in most cases, varying cellular automata parameters outside of S S and T T does not affect the loss plateau length. Any differences from the baseline trend (black lines) in Figure 10 are due to the loss plateau length saturating at 10,000 10,000 steps (the entire duration of training) as the task becomes too difficult to train on. For example, increasing the number of colors to C = 8 C=8 leads to C C W = 8 8 3 C^{C^{W}}=8^{8^{3}} possible rules, which makes the task too difficult except for the most extreme case of T = 2 T=2 . Outside of these unlearnable configurations, we see that increasing context length is the primary fa

### B.5 Depth Scaling

We study the effect of scaling depth when we fix the hidden dimension to D = 128 D=128 and the number of attention heads to H = 8 H=8 . In Figure 11 (left) we show that deeper models perform worse than shallow models on the linear map task. In fact, there is no significant benefit to scaling beyond a single layer model. We believe this is due to the attention-heavy nature of the linear map task; as visualized in Figure 4 , the model must express an intricate sparse pattern in its attention layers. Composing attention layers recursively may increase the difficulty of learning the ground truth pattern, despite the overall increase in model parameters.

In our cellular automata task, increasing layers has no benefit in the single-rule setting ( N = 1 N=1 ), even if we increase the recursive depth k k . However, in the multi-rule setting with N = 64 N=64 , we see a clear improvement with more layers. Additionally, increasing k k with N = 64 N=64 widens the gap between deeper and shallower models. In Figure 11 (right) we visualize the effect of scaling layers with N = 64 N=64 rules and depth k = 4 k=4 . The multi-rule cellular automata task inherently requires recursive computation, in contrast to the linear map task which does not benefit from increased depth. The cellular automata task resembles a more practical setting for scaling depth, as a task like language modeling requires both learning key attention patterns and applying recursive computation.

### B.6 Comparing Alternative Attention Mechanisms

Transformers model dependencies between tokens through dot-product attention, and in Section 4.3 we compare this mechanism to MLP-Mixer [ Tolstikhin et al., 2021 ] , which learns an MLP over the sequence dimension. In this section, we also evaluate Linear RNN [ Orvieto et al., 2023 ] , xLSTM [ Beck et al., 2024 ] , Mamba [ Gu and Dao, 2024 ] , Gated DeltaNet [ Yang et al., 2025 ] , and RWKV [ Peng et al., 2023 ] as alternatives to the attention-based transformer. Figure 12 illustrates the results.

As previously demonstrated in Figure 7(a) , MLP-Mixer outperforms transformers on the most difficult linear map tasks, but not on the cellular automata task. In Figure 12(a) we see that all other architectures also yield worse training performance than a transformer on the cellular automata task. When trained on the linear map task with state size N = 16 N=16 and sparsity s = 3 s=3 , only transformer and MLP-Mixer achieve nontrivial performance, and MLP-Mixer learns the task completely. However, when we increase N N to 32 32 , thus extending the context length, all architectures except for MLP-Mixer fail to make any progress at all.

## Appendix C Does Pre-pretraining Accelerate Emergence of Capabilities?