Progress measures for grokking via mechanistic interpretability

arxiv:2301.05217

Original source · Paper JSON

Reasoning and the "physics" of language models · 2023-01-12

Cites (1) · Cited by (17)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Progress measures for grokking via mechanistic interpretability
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Setup and Background
• 3.1 The Fourier multiplication algorithm
• 4 Reverse engineering a one-layer transformer
• 4.1 Suggestive evidence: surprising periodicity
• 4.2 Mechanistic Evidence: Composing Model Weights
• 4.3 Zooming In: Approximating Neurons with Sines and Cosines
• 4.4 Correctness checks: ablations
• 5 Understanding grokking behavior using progress measures
• 5.1 Progress measures
• 5.2 Phases of grokking: memorization, circuit formation, and cleanup
• 5.3 Grokking and Weight Decay
• 6 Conclusion and discussion
• Reproducibility Statement
• Author Contributions
• Acknowledgments
• Appendix A Mathematical Structure of the Transformer
• A.1 Empirical Model Simplifications
• Appendix B Why use constructive intereference?
• Appendix C Supporting evidence for mechanistic analysis of modular arithmetic networks
• C.1 Further analysis of the specific training run discussed in the paper
• C.1.1 Periodicity in the activations of other attention heads
• C.1.2 Approximating attention heads with Sines and Cosines
• C.1.3 The attention pattern weights are well approximated by differences of sines and cosines of a single frequency.
• Mechanistic Analysis of Attention Patterns.
• C.1.4 Periodicity in the activations of additional neurons
• C.1.5 Additional grokking figures for mainline run
• C.2 Additional results from different runs
• C.2.1 Additional results for different runs with the same architecture
• C.2.2 Results for other experimental setups
• 1-Layer Transformers with Varying Fractions of Training Data.
• 2-Layer Transformers.
• Smaller and larger primes.
• C.2.3 Generalizing models consistently use the Fourier Multiplication Algorithm
• Appendix D Additional results on grokking
• D.1 Both regularization and limited data are necessary for grokking
• D.2 The slingshot mechanism often occurs, but is unnecessary for grokking
• D.3 Additional evidence from other algorithmic tasks
• 5 Digit Addition
• Repeated subsequence
• Skip trigram
• Appendix E Further speculations on grokking
• E.1 An intuitive explanation of grokking
• E.2 Hypothesis: Phase Transitions are inherent to composition
• Appendix F Further discussion on using mechanistic interpretability and progress measures for studying emergent phenomena
• Instructions for reporting errors