Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

arxiv:2606.25010

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-06-23

Cites (4) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns
• Abstract
• 1 Introduction
• 2 Random Emergence of Capabilities in Language Models
• 2.1 Searching for Emergence During Training
• 2.2 Learning Core Attention Patterns Enables Abrupt Emergence
• Causal Attention Head Ablation.
• 2.3 Emergence Across Multiple Tasks
• General Trends.
• 3 Learning Synthetic Attention-Based Tasks
• 3.1 Task Definitions
• Linear Map.
• Cellular Automata.
• 3.2 Abrupt Learning with Linear Maps
• 3.3 Long-Context Attention Patterns are Difficult to Learn
• 4 The Role of Architecture in Learning Attention Patterns
• 4.1 Scaling Width
• 4.2 Scaling Attention Heads
• 4.3 Alternative Architectures
• 5 Related Work
• Emergent Capabilities.
• Abrupt Learning.
• Neural Scaling Laws.
• Our Contributions.
• 6 Conclusion
• Acknowledgements.
• Appendix A Emergent Capabilities in Pythia Language Models
• A.1 Causal Ablations and Attention Sinks
• A.2 Analyzing Emergence on Multiple Samples
• A.3 Interpreting Causal Attention Mechanisms
• Copying.
• In-Context Repetition.
• Pattern Completion.
• Indirect Object Identification.
• Appendix B Abrupt Learning on Synthetic Tasks
• B.1 Notation
• B.2 Attention Entropy
• B.3 Attention Interventions
• B.4 What Variables Affect Loss Plateau Length?
• B.5 Depth Scaling
• B.6 Comparing Alternative Attention Mechanisms
• Appendix C Does Pre-pretraining Accelerate Emergence of Capabilities?
• Hyperparameters.
• Instructions for reporting errors