{"id":"arxiv:2301.05217","title":"Progress measures for grokking via mechanistic interpretability","date":"2023-01-12","section":"Reasoning and the \"physics\" of language models","quality":0.649164,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/","source_url":"https://arxiv.org/abs/2301.05217","text_available":true,"authors":["Neel Nanda","Lawrence Chan","Tom Lieberum","Jess Smith"],"outgoing":[{"source":"arxiv:2301.05217","target":"arxiv:1803.03635","evidence":"id","id":"arxiv:1803.03635","title":"The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a9d7f0235205afcc709e3e74f28cc7707831bcb1d46239c418cc7e4b624d23bd/"}],"incoming":[{"source":"arxiv:2305.13673","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2305.13673","title":"Physics of Language Models: Part 1, Learning Hierarchical Language Structures","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/a72219d4bd65d1e84ce31079b6c788d7e6c297e71d4b083af5d4cffccfdcff72/"},{"source":"arxiv:2405.15071","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2405.15071","title":"Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/1cd7fa83eca0b480c3bf9e2490c043b70e937777252a40b196ae7801a0620141/"},{"source":"arxiv:2405.20233","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2405.20233","title":"Grokfast: Accelerated Grokking by Amplifying Slow Gradients","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b62d50c44fbd3554702d96f717a5ee25029a28ceed9346ab27ff64d1d08ba07e/"},{"source":"arxiv:2406.03445","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2406.03445","title":"Pre-trained Large Language Models Use Fourier Features to Compute Addition","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/169a3e64467f1e0862508b86b2198187fc8eeb3ab114a47d2bd4caaea4a9ee2b/"},{"source":"arxiv:2410.21272","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2410.21272","title":"Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/397b826ba289bd4f14f457562942cb092c38550a4c49451e95ffcc7b376c8cb0/"},{"source":"arxiv:2502.00873","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2502.00873","title":"Language Models Use Trigonometry to Do Addition","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7766b2640c6b5ab53a295e8d60920cb2419bfbbd812810d9b35d4b7c323922ed/"},{"source":"arxiv:2502.05171","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2502.05171","title":"Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/24a6ab865777290627cecebd3f06ea3daef01622f102d32a95f7fb2a5a42880f/"},{"source":"arxiv:2502.19981","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2502.19981","title":"The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/fc024e63f0774183f41cf84f17bb5a4238ff43a8ec18a6f8afb9a1abcd113488/"},{"source":"arxiv:2503.21676","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2503.21676","title":"How do language models learn facts? Dynamics, curricula and hallucinations","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/2ef59e7bcfcfc0e4bab638835504ad8a0d2f50665a9797edc09e8628e9b3c0b6/"},{"source":"arxiv:2504.20571","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2504.20571","title":"Reinforcement Learning for Reasoning in Large Language Models with One Training Example","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/b2b21c3df49619588c371411dba9467a41f81b3fd4da5441898b5145bb0de120/"},{"source":"arxiv:2508.02513","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2508.02513","title":"Modular Arithmetic: Language Models Solve Math Digit by Digit","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/83aede757707b76dc76fc15431a436e5dec05563a9f2f28ebcc650775c878bcc/"},{"source":"arxiv:2510.00184","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2510.00184","title":"Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7e395b2ae5bfc2a4d8bfc1e8669b4f324739b6a9b11161cba150bfdffc31746c/"},{"source":"arxiv:2512.16902","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2512.16902","title":"In-Context Algebra","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/"},{"source":"arxiv:2606.03982","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2606.03982","title":"Language Models Compare Quantities Using Number-specific and Unit-specific Heuristics","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/48f07d03c9308ca8fb43ecd2ac22978f6dcbd4bc081056f022ca511711942dbf/"},{"source":"arxiv:2606.25010","target":"arxiv:2301.05217","evidence":"id","id":"arxiv:2606.25010","title":"Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f87db387bad54ae4fbc0589d2a702f78aeb0817b28f20844f6502bc552a89c2f/"},{"source":"arxiv:2608.23875","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2608.23875","title":"AI Finds A Way","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/afe34dd5ee58c2e9739f9651349c4c9180d95dabff51994a13339f48dfd0bb27/"},{"source":"url:alignmentforum.org/posts/nffst5mio7bcaqhpa/an-extremely-opinionated-annotated-list-of-my-favourite-1","target":"arxiv:2301.05217","evidence":"id","id":"url:alignmentforum.org/posts/nffst5mio7bcaqhpa/an-extremely-opinionated-annotated-list-of-my-favourite-1","title":"An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 (Alignment Forum)","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/28f1f482044fd8bff4e1190eded455473b6882f5493ef64f40f8ab78ec9ba79b/"}],"text_parts":[{"start":0,"end":15546,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html"},{"start":15546,"end":31275,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html"},{"start":31275,"end":46874,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html"},{"start":46874,"end":62497,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html"},{"start":62497,"end":78210,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html"},{"start":78210,"end":80061,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-6.html"}],"contents":[{"title":"Progress measures for grokking via mechanistic interpretability","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"Abstract","offset":67,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"1 Introduction","offset":1399,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"2 Related Work","offset":5780,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"3 Setup and Background","offset":7563,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"3.1 The Fourier multiplication algorithm","offset":10169,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"4 Reverse engineering a one-layer transformer","offset":12438,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"4.1 Suggestive evidence: surprising periodicity","offset":14510,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-1.html#text"},{"title":"4.2 Mechanistic Evidence: Composing Model Weights","offset":16495,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"4.3 Zooming In: Approximating Neurons with Sines and Cosines","offset":22115,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"4.4 Correctness checks: ablations","offset":23829,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"5 Understanding grokking behavior using progress measures","offset":26478,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"5.1 Progress measures","offset":26846,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"5.2 Phases of grokking: memorization, circuit formation, and cleanup","offset":28402,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"5.3 Grokking and Weight Decay","offset":30243,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"6 Conclusion and discussion","offset":31243,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-2.html#text"},{"title":"Reproducibility Statement","offset":33041,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Author Contributions","offset":33255,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Acknowledgments","offset":34689,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Appendix A Mathematical Structure of the Transformer","offset":36418,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"A.1 Empirical Model Simplifications","offset":39487,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Appendix B Why use constructive intereference?","offset":41373,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Appendix C Supporting evidence for mechanistic analysis of modular arithmetic networks","offset":42883,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"C.1 Further analysis of the specific training run discussed in the paper","offset":42974,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"C.1.1 Periodicity in the activations of other attention heads","offset":43133,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"C.1.2 Approximating attention heads with Sines and Cosines","offset":43702,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"C.1.3 The attention pattern weights are well approximated by differences of sines and cosines of a single frequency.","offset":45087,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"Mechanistic Analysis of Attention Patterns.","offset":45881,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-3.html#text"},{"title":"C.1.4 Periodicity in the activations of additional neurons","offset":50477,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"C.1.5 Additional grokking figures for mainline run","offset":50724,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"C.2 Additional results from different runs","offset":51436,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"C.2.1 Additional results for different runs with the same architecture","offset":52285,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"C.2.2 Results for other experimental setups","offset":55025,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"1-Layer Transformers with Varying Fractions of Training Data.","offset":55277,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"2-Layer Transformers.","offset":55898,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"Smaller and larger primes.","offset":56297,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"C.2.3 Generalizing models consistently use the Fourier Multiplication Algorithm","offset":57811,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"Appendix D Additional results on grokking","offset":59039,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"D.1 Both regularization and limited data are necessary for grokking","offset":59085,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"D.2 The slingshot mechanism often occurs, but is unnecessary for grokking","offset":61978,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-4.html#text"},{"title":"D.3 Additional evidence from other algorithmic tasks","offset":62950,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"5 Digit Addition","offset":63938,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"Repeated subsequence","offset":64559,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"Skip trigram","offset":64922,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"Appendix E Further speculations on grokking","offset":65341,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"E.1 An intuitive explanation of grokking","offset":65389,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"E.2 Hypothesis: Phase Transitions are inherent to composition","offset":70214,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"Appendix F Further discussion on using mechanistic interpretability and progress measures for studying emergent phenomena","offset":76229,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-5.html#text"},{"title":"Instructions for reporting errors","offset":79097,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/part-6.html#text"}],"text_code_points":80061,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/source.md"}