Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns — part 4/4

Cleaned-text code points [47576, 51957). Bibliography omitted here; source.md includes it.

Our results in Section 4 show how architectural interventions improve a model’s ability to learn attention patterns. We now explore how data interventions accelerate emergence of practical capabilities in language models. Following prior work, we apply pre-pretraining (PPT) with two synthetic datasets: 33 million tokens of k k -shuffle Dyck [ Hu et al., 2025 ] and 164 million tokens of neural cellular automata (NCA, Lee et al. [2026] ). Our training experiments consist of two main phases. In the PPT phase, we train a randomly initialized model on synthetic data, and in the second phase we train on natural language pretraining data using the final weights from the first phase as an initialization. We compare to a natural language baseline where we only train on standard pretraining data from scratch, starting with the same random initialization we would use for PPT.

We train transformer models with 160 million parameters, following the Pythia training recipe exactly [ Biderman et al., 2023 ] . We train on 10 billion tokens of C4 data (5,000 steps) since the original dataset used to train Pythia models is no longer available. For pre-pretraining, we follow the implementations from Hu et al. [2025] and Lee et al. [2026] for Dyck and NCA respectively, except we extend the context length of NCA data to 2048 to match that of standard Pythia language training. Note pre-pretraining hyperparameters differ from those used in the natural language training phase.

#### Hyperparameters.

Natural language training uses 5,000 steps with 1,430 warmup steps. For Dyck pre-pretraining, we train for 500 steps (30M tokens) with context length 2048 and batch size 32, matching the hyperparameters of Hu et al. [2025] for 160M parameter models. For NCA pre-pretraining, we use the same hyperparameters except with batch size 16, yielding 164M tokens over 5,000 steps. We also re-initialize the token embeddings for NCA pre-pretraining.

We compare training from scratch, PPT with Dyck, and PPT with NCA across 4 different random initialization seeds. As previously demonstrated by Hu et al. [2025] and Lee et al. [2026] , both Dyck and NCA pre-pretraining result in lower final loss after 5,000 steps compared to training from scratch. During training, we probe for the emergence of the capabilities illustrated in Figure 2 every ten steps. We also track the time to emergence using the same approach described in Section 2.1 .

Figure 13(a) shows emergence results for in-context repetition, pattern completion, and indirect object identification. Pre-pretraining tends to result in in-context capabilities emerging earlier , though this trend does not hold for pattern completion. In Figure 13(b) , NCA pre-pretraining leads to the emergence of indirect object identification for one seed, whereas the capability does not emerge for training from scratch or Dyck PPT after 5,000 steps. However, we caution that these results are based on only four random seeds, which limits statistical power given the high inter-run variance of emergence. Copying does not emerge in any model after 5,000 steps; we believe this may be due to the change in pretraining dataset from the original Pythia setup. These results suggest that the effect of pre-pretraining on emergent capabilities depends on the specific capability and the nature of the synthetic data, but a larger sample is needed to draw concrete conclusions.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .