Grokking Group Multiplication with Cosets — part 4/7

Cleaned-text code points [38998, 53060). Bibliography omitted here; source.md includes it.

Grokking Group Multiplication with Cosets
we train. Though our setting is too small to be directly
relevant to real-world use cases, we hope that similar tech-
niques will be able to test, audit, and monitor deep neural
networks that have been deployed in the real world. We
also present results that urge caution and humility when
attempting to interpret neural networks. We believe that
robust and effective interpretability techniques may miti-
gate some societal harms that could arise from the use of
deep neural networks, but that mistakenly trusting illusory
interpretability techniques could be disastrous.
Acknowledgements
We would like to thank Coreweave for donating the comput-
ing resources that we used to run all of our experiments, to
Bilal Chughtai for helpful discussions we had throughout the
project, and to Neel Nanda for telling us to “not hold back
for fear of offending [him].” We would also like to thank
Nora Belrose, Neils uit de Bos, Aidan Ewart, Sara Price,
Hailey Schoelkopf, C´edric Simal, and Benjamin Wright for
their helpful feedback on earlier drafts of this paper.
References
[1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I.,
Hardt, M., and Kim, B. Sanity checks for saliency
maps, 2020.
[2] Bolukbasi, T., Pearce, A., Yuan, A., Coenen, A., Reif,
E., Vi´egas, F., and Wattenberg, M. An interpretability
illusion for bert, 2021.
[3] Casper, S., Li, Y ., Li, J., Bu, T., Zhang, K., Hariharan,
K., and Hadfield-Menell, D. Red teaming deep neural
networks with feature synthesis tools, 2023.
[4] Chughtai, B., Chan, L., and Nanda, N. A
Toy Model of Universality: Reverse Engineering
How Networks Learn Group Operations. Tech-
nical Report arXiv:2302.03025, arXiv, May 2023.
URL http://arxiv.org/abs/2302.03025.
arXiv:2302.03025 [cs, math] type: article.
[5] Clausen, M. and Baum, U. Fast Fourier Trans-
forms for Symmetric Groups: Theory and Imple-
mentation. Mathematics of Computation , 61(204):
833–847, 1993. ISSN 0025-5718. doi: 10.2307/
2153256. URL https://www.jstor.org/
stable/2153256. Publisher: American Mathe-
matical Society.
[6] Cohen, T. and Welling, M. Group Equivari-
ant Convolutional Networks. In Proceedings
of The 33rd International Conference on Ma-
chine Learning , pp. 2990–2999. PMLR, June
2016. URL https://proceedings.mlr.
press/v48/cohenc16.html. ISSN: 1938-
7228.
[7] Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimer-
sheim, S., and Garriga-Alonso, A. Towards Auto-
mated Circuit Discovery for Mechanistic Interpretabil-
ity, October 2023. URL http://arxiv.org/
abs/2304.14997. arXiv:2304.14997 [cs].
[8] Diaconis, P. Group Representations in Probability
and Statistics, volume 11 of Institute of Mathemati-
cal Statistics Lecture Notes. Insitute of Mathematical
Statistics, Hayward, CA, 1988. ISBN 0-940600-14-5.
[9] Doshi-Velez, F. and Kim, B. Towards a rigorous sci-
ence of interpretable machine learning, 2017.
[10] Dummit, D. S. and Foote, R. M. Abstract Algebra.
Wiley, 3rd edition, July 2003. ISBN 978-0-471-43334-
7.
[11] Elias M. Stein, R. S. Fourier analysis: an introduction.
Princeton lectures in analysis 1. Princeton University
Press, 2003. ISBN 069111384X,9780691113845.
[12] Ellis, D., Friedgut, E., and Pilpel, H. Intersecting
Families of Permutations, July 2017. URL http://
arxiv.org/abs/1011.3342. arXiv:1011.3342
[math].
[13] Erdos, P. On an elementary proof of some asymptotic
formulas in the theory of partitions. Annals of Mathe-
matics, 43(3):437–450, 1942. ISSN 0003486X. URL
http://www.jstor.org/stable/1968802.
[14] Friedman, D., Lampinen, A., Dixon, L., Chen, D., and
Ghandeharioun, A. Interpretability illusions in the
generalization of simplified models, 2023.
[15] Fulton, W. and Harris, Joe. Representation Theory.
Graduate Texts in Mathematics. Springer, New York,
NY , October 1991. ISBN 978-0-387-97495-8.
[16] GAP. Gap – groups, algorithms, and program-
ming, version 4.12.2, 2023. URL https://www.
gap-system.org.
[17] Geiger, A., Potts, C., and Icard, T. Causal abstraction
for faithful model interpretation, 2023.
[18] Geva, M., Schuster, R., Berant, J., and Levy, O. Trans-
former feed-forward layers are key-value memories,
2021.
[19] Goldowsky-Dill, N., MacLeod, C., Sato, L., and Arora,
A. Localizing model behavior with path patching,
2023.
10

Grokking Group Multiplication with Cosets
[20] Gromov, A. Grokking modular arithmetic, January
2023. URL http://arxiv.org/abs/2301.
02679. arXiv:2301.02679 [cond-mat].
[21] Hanna, M., Liu, O., and Variengien, A. How does
GPT-2 compute greater-than?: Interpreting mathemat-
ical abilities in a pre-trained language model, Novem-
ber 2023. URL http://arxiv.org/abs/2305.
00586. arXiv:2305.00586 [cs].
[22] Harris, C. R., Millman, K. J., van der Walt, S. J.,
Gommers, R., Virtanen, P., Cournapeau, D., Wieser,
E., Taylor, J., Berg, S., Smith, N. J., Kern, R., Pi-
cus, M., Hoyer, S., van Kerkwijk, M. H., Brett,
M., Haldane, A., del R ´ıo, J. F., Wiebe, M., Peter-
son, P., G ´erard-Marchant, P., Sheppard, K., Reddy,
T., Weckesser, W., Abbasi, H., Gohlke, C., and
Oliphant, T. E. Array programming with NumPy.
Nature, 585(7825):357–362, September 2020. doi:
10.1038/s41586-020-2649-2. URL https://doi.
org/10.1038/s41586-020-2649-2 .
[23] Hase, P., Bansal, M., Kim, B., and Ghandeharioun,
A. Does localization inform editing? surprising dif-
ferences in causality-based localization vs. knowledge
editing in language models, 2023.
[24] Huang, J., Guestrin, C., and Guibas, L. Fourier Theo-
retic Probabilistic Inference over Permutations. Jour-
nal of Machine Learning Research, 10(37):997–1070,
2009. ISSN 1533-7928. URL http://jmlr.org/
papers/v10/huang09a.html.
[25] Jain, S. and Wallace, B. C. Attention is not
explanation. In North American Chapter of the
Association for Computational Linguistics , 2019.
URL https://api.semanticscholar.org/
CorpusID:67855860.
[26] Janusz, G. and Rotman, J. Outer automorphisms of
S6. The American Mathematical Monthly, 89(6):407–
410, 1982. ISSN 00029890, 19300972. URL http:
//www.jstor.org/stable/2321657.
[27] Karjol, P., Kashyap, R., and Ap, P. Neural Dis-
covery of Permutation Subgroups. In Proceedings
of The 26th International Conference on Artificial
Intelligence and Statistics , pp. 4668–4678. PMLR,
April 2023. URL https://proceedings.mlr.
press/v206/karjol23a.html. ISSN: 2640-
3498.
[28] Kingma, D. P. and Ba, J. Adam: A method for
stochastic optimization. CoRR, abs/1412.6980, 2014.
URL https://api.semanticscholar.org/
CorpusID:6628106.
[29] Kondor, R. Group theoretical methods in machine
learning. PhD thesis, Columbia University, New
York, NY , 2008. URL https://dl.acm.org/
doi/abs/10.5555/1570977. Archive Location:
world.
[30] Kondor, R. and Trivedi, S. On the General-
ization of Equivariance and Convolution in Neu-
ral Networks to the Action of Compact Groups.
In Proceedings of the 35th International Confer-
ence on Machine Learning , pp. 2747–2755. PMLR,
July 2018. URL https://proceedings.mlr.
press/v80/kondor18a.html. ISSN: 2640-
3498.
[31] Kumar, T., Bordelon, B., Gershman, S. J., and
Pehlevan, C. Grokking as the Transition from
Lazy to Rich Training Dynamics, October 2023.
URL http://arxiv.org/abs/2310.06110.
arXiv:2310.06110 [cond-mat, stat].
[32] Lieberum, T., Rahtz, M., Kram´ar, J., Nanda, N., Irving,
G., Shah, R., and Mikulik, V . Does circuit analysis
interpretability scale? evidence from multiple choice
capabilities in chinchilla, 2023.
[33] Lundberg, S. and Lee, S.-I. A unified approach to
interpreting model predictions, 2017.
[34] Makelov, A., Lange, G., and Nanda, N. Is this the sub-
space you are looking for? an interpretability illusion
for subspace activation patching, 2023.
[35] McGrath, T., Rahtz, M., Kramar, J., Mikulik, V ., and
Legg, S. The hydra effect: Emergent self-repair in
language model computations, 2023.
[36] Meng, K., Bau, D., Andonian, A., and Belinkov, Y .
Locating and editing factual associations in gpt, 2023.
[37] Merrill, W., Tsilivis, N., and Shukla, A. A
Tale of Two Circuits: Grokking as Competition
of Sparse and Dense Subnetworks, March 2023.
URL http://arxiv.org/abs/2303.11873.
arXiv:2303.11873 [cs].
[38] Morwani, D., Edelman, B. L., Oncescu, C.-A., Zhao,
R., and Kakade, S. Feature emergence via margin
maximization: case studies in algebraic tasks, 2024.
[39] Nanda, N. and Bloom, J. Transformerlens.
https://github.com/neelnanda-io/
TransformerLens, 2022.
[40] Nanda, N., Chan, L., Lieberum, T., Smith,
J., and Steinhardt, J. Progress measures for
grokking via mechanistic interpretability. Techni-
cal Report arXiv:2301.05217, arXiv, January 2023.
11

Grokking Group Multiplication with Cosets
URL http://arxiv.org/abs/2301.05217.
arXiv:2301.05217 [cs] type: article.
[41] Nanda, N., Lee, A., and Wattenberg, M. Emer-
gent Linear Representations in World Models of
Self-Supervised Sequence Models, September 2023.
URL http://arxiv.org/abs/2309.00941.
arXiv:2309.00941 [cs].
[42] OEIS Foundation Inc. The On-Line Encyclopedia of
Integer Sequences, 2023. Published electronically at
http://oeis.org.
[43] Olah, C., Cammarata, N., Schubert, L., Goh, G.,
Petrov, M., and Carter, S. Zoom In: An In-
troduction to Circuits. Distill, 5(3):e00024.001,
March 2020. ISSN 2476-0757. doi: 10.23915/
distill.00024.001. URL https://distill.pub/
2020/circuits/zoom-in.
[44] Olsson, C., Elhage, N., Nanda, N., Joseph, N., Das-
Sarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y .,
Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-
Dodds, Z., Hernandez, D., Johnston, S., Jones, A.,
Kernion, J., Lovitt, L., Ndousse, K., Amodei, D.,
Brown, T., Clark, J., Kaplan, J., McCandlish, S., and
Olah, C. In-context Learning and Induction Heads,
September 2022. URL https://arxiv.org/
abs/2209.11895v1.
[45] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury,
J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N.,
Antiga, L., Desmaison, A., K ¨opf, A., Yang, E., De-
Vito, Z., Raison, M., Tejani, A., Chilamkurthy, S.,
Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch:
An Imperative Style, High-Performance Deep Learn-
ing Library, December 2019. URL http://arxiv.
org/abs/1912.01703. arXiv:1912.01703 [cs,
stat].
[46] Plumb, G., Pachauri, D., Kondor, R., and Singh, V .
SnFFT: A Julia Toolkit for Fourier Analysis of Func-
tions over Permutations. Journal of Machine Learning
Research, 16(107):3469–3473, 2015. URL http:
//jmlr.org/papers/v16/plumb15a.html.
[47] Power, A., Burda, Y ., Edwards, H., Babuschkin, I.,
and Misra, V . Grokking: Generalization Beyond
Overfitting on Small Algorithmic Datasets, January
2022. URL http://arxiv.org/abs/2201.
02177. arXiv:2201.02177 [cs].
[48] Pyber, L. Enumerating finite groups of given or-
der. Annals of Mathematics, 137(1):203–220, 1993.
ISSN 0003486X. URL http://www.jstor.
org/stable/2946623.
[49] Quirke, P. and Barez, F. Understanding addition in
transformers, 2024.
[50] Rubin, N., Seroussi, I., and Ringel, Z. Droplets
of Good Representations: Grokking as a First Or-
der Phase Transition in Two Layer Networks, Octo-
ber 2023. URL http://arxiv.org/abs/2310.
03789. arXiv:2310.03789 [cond-mat, stat].
[51] Shrikumar, A., Greenside, P., Shcherbina, A., and Kun-
daje, A. Not just a black box: Learning important
features through propagating activation differences,
2017.
[52] Simonyan, K., Vedaldi, A., and Zisserman, A. Deep
inside convolutional networks: Visualising image clas-
sification models and saliency maps, 2014.
[53] Stein, W. et al. Sage Mathematics Software (Version
10.0.0). The Sage Development Team, 2023. URL
http://www.sagemath.org.
[54] Tigges, C., Hollinsworth, O. J., Geiger, A., and Nanda,
N. Linear representations of sentiment in large lan-
guage models, 2023.
[55] Varma, V ., Shah, R., Kenton, Z., Kram ´ar, J., and
Kumar, R. Explaining grokking through circuit ef-
ficiency, September 2023. URL http://arxiv.
org/abs/2309.02390. arXiv:2309.02390 [cs].
[56] Vink, R., Gooijer, S. d., Beedie, A., Gorelli, M. E.,
Zundert, J. v., Hulselmans, G., Grinstead, C., Santa-
maria, M., Guo, W., Heres, D., Magarick, J., Mar-
shall, ibENPC, Peters, O., Leitao, J., Wilksch, M.,
Heerden, M. v., Borchert, O., Jermain, C., Haag, J.,
Peek, J., Russell, R., Pryer, C., Castellanos, A. G.,
Goh, J., illumination-k, Brannigan, L., Conradt, M.,
and Robert. pola-rs/polars: Python Polars 0.19.0, Au-
gust 2023. URL https://doi.org/10.5281/
zenodo.8301818.
[57] Wang, K., Variengien, A., Conmy, A., Shlegeris,
B., and Steinhardt, J. Interpretability in the Wild:
a Circuit for Indirect Object Identification in GPT-
2 small, November 2022. URL http://arxiv.
org/abs/2211.00593. arXiv:2211.00593 [cs].
[58] Wang, K., Variengien, A., Conmy, A., Shlegeris, B.,
and Steinhardt, J. Interpretability in the wild: a circuit
for indirect object identification in gpt-2 small, 2022.
[59] Wei, J., Tay, Y ., Bommasani, R., Raffel, C., Zoph, B.,
Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D.,
Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O.,
Liang, P., Dean, J., and Fedus, W. Emergent abilities
of large language models, 2022.
12

Grokking Group Multiplication with Cosets
[60] Wen, K., Li, Y ., Liu, B., and Risteski, A. Transform-
ers are uninterpretable with myopic methods: a case
study with bounded dyck grammars. In Thirty-seventh
Conference on Neural Information Processing Sys-
tems, 2023. URL https://openreview.net/
forum?id=OitmaxSAUu.
[61] Wold, S., Esbensen, K., and Geladi, P. Principal com-
ponent analysis. Chemometrics and Intelligent Labo-
ratory Systems, 2(1-3):37–52, 1987.
[62] Xu, Z., Wang, Y ., Frei, S., Vardi, G., and Hu,
W. Benign Overfitting and Grokking in ReLU
Networks for XOR Cluster Data, October 2023.
URL http://arxiv.org/abs/2310.02541.
arXiv:2310.02541 [cs, stat].
[63] Zhang, S. D., Tigges, C., Biderman, S., Ra-
ginsky, M., and Ringer, T. Can Transformers
Learn to Solve Problems Recursively?, June 2023.
URL http://arxiv.org/abs/2305.14699.
arXiv:2305.14699 [cs].
[64] Zhong, Z., Liu, Z., Tegmark, M., and Andreas, J.
The Clock and the Pizza: Two Stories in Mecha-
nistic Explanation of Neural Networks, June 2023.
URL http://arxiv.org/abs/2306.17844.
arXiv:2306.17844 [cs].
13