Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls

arxiv:2510.00184

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-09-30

Cites (6) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls
• Abstract
• 1 Introduction
• 2 Experiment Setup, Training ICoT, Notations
• Notations.
• 3 Comparing the Mechanisms of ICoT versus SFT
• 3.1 Long-range dependencies in multi-digit multiplication
• 3.2 Evidence of Long-Range Dependencies in ICoT
• 3.3 Encoding Long-Range Dependencies via Attention Trees
• 4 Feature Geometry of ICoT
• 4.1 Digit-wise Multiplications as Minkowski Sums
• 4.2 Embedding Digits on a Pentagonal Prism via Fourier Bases
• 5 Pitfalls of Learning: Lack of Long-Range Dependency
• 6 Learning Multiplication Without ICoT
• 6.1 Does the model with auxiliary loss learn the same mechanism as ICoT?
• 7 Related Work
• Studying Transformers with Arithmetic Tasks.
• 8 Conclusion
• Acknowledgments
• Reproducibility Statement
• Appendix A Training Details
• A.1 ICoT Training
• A.2 Standard Fine-Tuning
• Appendix B Fourier structure in model’s weights, activations
• Appendix C Per Token Gradients and Losses: 12-Layer Model
• Appendix D Attention Patterns of All Models
• Appendix E LLM Usage
• Instructions for reporting errors