{"id":"arxiv:2512.16902","title":"In-Context Algebra","date":"2025-12-18","section":"Reasoning and the \"physics\" of language models","quality":0.332516,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/","source_url":"https://arxiv.org/abs/2512.16902","text_available":true,"authors":["Eric Todd","Jannik Brinkmann","Rohit Gandikota","David Bau"],"outgoing":[{"source":"arxiv:2512.16902","target":"arxiv:2301.05217","evidence":"title","id":"arxiv:2301.05217","title":"Progress measures for grokking via mechanistic interpretability","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/f71e3b65a8b3f35beb171d17f6c65878e0914227a0042ccba59b88952554baef/"},{"source":"arxiv:2512.16902","target":"arxiv:2406.03445","evidence":"title","id":"arxiv:2406.03445","title":"Pre-trained Large Language Models Use Fourier Features to Compute Addition","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/169a3e64467f1e0862508b86b2198187fc8eeb3ab114a47d2bd4caaea4a9ee2b/"},{"source":"arxiv:2512.16902","target":"arxiv:2407.20311","evidence":"id","id":"arxiv:2407.20311","title":"Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/de5db7bc6b929af401b5b83c845823505a7823c25987b0e455231d2e4fcec3e4/"},{"source":"arxiv:2512.16902","target":"arxiv:2410.21272","evidence":"title","id":"arxiv:2410.21272","title":"Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/397b826ba289bd4f14f457562942cb092c38550a4c49451e95ffcc7b376c8cb0/"},{"source":"arxiv:2512.16902","target":"arxiv:2502.00873","evidence":"id","id":"arxiv:2502.00873","title":"Language Models Use Trigonometry to Do Addition","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7766b2640c6b5ab53a295e8d60920cb2419bfbbd812810d9b35d4b7c323922ed/"},{"source":"arxiv:2512.16902","target":"arxiv:2510.00184","evidence":"id","id":"arxiv:2510.00184","title":"Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/7e395b2ae5bfc2a4d8bfc1e8669b4f324739b6a9b11161cba150bfdffc31746c/"},{"source":"arxiv:2512.16902","target":"openreview:hcQfTsVnBo","evidence":"url","id":"openreview:hcQfTsVnBo","title":"Grokking Group Multiplication with Cosets","url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/daa23510ca0765cf7cb802f4ab922f167481195d3b31bd012b5e4298470c342d/"}],"incoming":[],"text_parts":[{"start":0,"end":15856,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html"},{"start":15856,"end":31693,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html"},{"start":31693,"end":47688,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html"},{"start":47688,"end":63435,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html"},{"start":63435,"end":76891,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html"}],"contents":[{"title":"In-Context Algebra","offset":0,"level":1,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"Abstract","offset":22,"level":6,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"1 Introduction","offset":1296,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"2 Task Description","offset":3750,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"3 Can transformers learn in-context algebra?","offset":6924,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"4 Hypothesizing Model Mechanisms","offset":9646,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"4.1 Measuring Coverage and Performance on Targeted Distributions","offset":13371,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-1.html#text"},{"title":"5 Causal Verification of Learned Mechanisms","offset":16471,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"5.1 Causal Interventions","offset":16811,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"5.2 Copying and Commutative Copying","offset":18577,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"5.3 Identity Recognition","offset":22208,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"5.4 Closure-Based Cancellation","offset":26157,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"6 Phase Transitions Correspond to Learning of Discrete Skills","offset":31627,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-2.html#text"},{"title":"7 Related Work","offset":34076,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"8 Conclusion","offset":37694,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"Ethics Statement","offset":38764,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"Acknowledgments","offset":39050,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"Appendix A Group Theory","offset":39453,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"Appendix B Discussion: Contextual Reasoning is Richer Than Just Copying","offset":42715,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"Appendix C Model Architecture and Training","offset":47641,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-3.html#text"},{"title":"C.1 Model Architecture Hyperparameters","offset":49897,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"C.2 Group Sampling Probability and Task Diversity","offset":51642,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"C.3 Performance on Groups and Non-Group Algebraic Structures","offset":55220,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"C.4 Additional Performance on Data Subsets","offset":56670,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"Appendix D Additional Results on Copying","offset":56816,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"Appendix E Identity Recognition Details","offset":59851,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-4.html#text"},{"title":"Appendix F Closure and Elimination Subspaces","offset":63835,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"F.1 How are Closure and Cancellation Sets Computed?","offset":65588,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"Cancellation Set Construction.","offset":67308,"level":5,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"F.2 Subspace Construction","offset":68674,"level":3,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"F.2.1 Subspace Training Details","offset":70257,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"F.2.2 Probing the Closure and Elimination Subspaces","offset":73301,"level":4,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"Appendix G Data Coverage Pseudocode","offset":75055,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"Appendix H Use of Large Language Models","offset":75325,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"},{"title":"Instructions for reporting errors","offset":75927,"level":2,"url":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/part-5.html#text"}],"text_code_points":76891,"source_markdown":"https://vinnik-dmitry07.github.io/favorite-papers/agents/papers/5aa7087a88619eafd156aad9217e16f6d1ce284c65534322239be0e02f04e553/source.md"}