Language Models Use Trigonometry to Do Addition

arxiv:2502.00873

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-02-02

Cites (5) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Language Models Use Trigonometry to Do Addition
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• 3 Problem Setup
• 4 LLMs Represent Numbers as a Helix
• 4.1 Investigating Numerical Structure
• 4.2 Parameterizing the Structure as a Helix
• 4.3 Fitting a Helix
• 4.4 Evaluating the Quality of the Helical Fit
• 4.5 Is the Helix the Full Picture?
• 5 LLMs Use the Clock Algorithm to Compute Addition
• 5.1 Introducing the Clock Algorithm
• 5.2 Investigating Attention Heads
• 5.3 Looking at MLPs
• 5.4 Zooming in on Neurons
• 5.4.1 Modeling Neuron Preactivations
• 5.4.2 Understanding MLP Inputs
• 5.5 Limitations of Our Understanding
• 6 Conclusion
• Acknowledgments
• Appendix A Performance of all models on a + b = a+b=
• Appendix B Additional Results on the Structure of Numbers
• Appendix C Additional Helix Fitting Results
• C.1 Helix Properties
• C.2 Additional Causal Experiments for Helix Fits
• Appendix D Additional Clock algorithm evidence
• D.1 Attention Heads
• D.2 MLPs and Neurons
• Appendix E Why Use the Clock Algorithm at All?
• Appendix F Investigating Model Errors
• Appendix G Tooling and Compute
• Instructions for reporting errors