Modular Arithmetic: Language Models Solve Math Digit by Digit

arxiv:2508.02513

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-08-04

Cites (7) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Modular Arithmetic: Language Models Solve Math Digit by Digit
• Abstract
• 1 Introduction
• 2 Method
• 2.1 Overview of our Approach
• 2.2 Data for Neuron Identification Experiments
• 2.3 Data for Intervention Experiments
• 2.4 Models
• 3 Circuit Localization
• 3.1 Step 1: Identifying Digit-Specific Neurons
• Intuition.
• Neuron Selection for Digit-Position Circuits.
• Statistics on Selected Circuits.
• 3.2 Step 2: Causal Verification via Interventions
• Circuit Depth.
• Experiment.
• Results.
• 4 Supplementary Experiments
• 4.1 Addition and Subtraction Circuits are Largely Distinct
• 4.2 Additions with Carry Bits Employ the Same Mechanisms
• (1) Carry from Units to Tens.
• (2) Carry from Tens to Hundreds.
• Results.
• 4.3 MLP Similarity Reflects Task Overlap
• 5 Structured Arithmetic Processing or a Bag of Heuristics?
• 6 Related Work
• 7 Conclusion
• Limitations
• Acknowledgements
• Appendix A Implementation Details
• Appendix B Results on Single Digit Tokenization Models
• Intuition: Single- vs Multi-digit Numeric Tokenization.
• Localizing Digit-Specific Neurons.
• Intervention.
• Appendix C Layers Involved in Arithmetic
• Appendix D Fisher Score for Digit-Sensitivity of Neurons.
• Appendix E Statistics on Arithmetic Circuits
• Appendix F Effect of chosen Threshold on Digit-Circuit Intervention
• Appendix G Intervention Results on D a ​ d ​ d , o ​ p ​ 1 D_{add,op1} and D s ​ u ​ b , o ​ p ​ 1 D_{sub,op1}
• Appendix H Ablation: Deeper Circuits
• Appendix I Similarity of Addition and Subtraction Circuits
• Appendix J Examples of Neuron Heuristics
• Instructions for reporting errors