The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs

arxiv:2502.19981

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-02-27

Cites (4) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs
• Abstract
• 1 Introduction
• 2 LLMs Struggle with Multi-Operand Addition
• 2.1 Models and Data
• Models.
• Data.
• 2.2 LLM Accuracy on Addition Tasks
• 3 Probing LLMs on Digits in Two-Operand Addition Tasks
• 3.1 Methodology and Experiments
• Data.
• Probing Setup.
• 3.2 Results
• 4 The Carry Heuristic of LLMs
• 4.1 Formalization of Left-to-Right Addition in Base 10
• 4.2 A Naive Heuristic for Solving Addition Left-to-Right
• Examples.
• 147 + 293.
• 147 + 255.
• 5 H1 Predicts Difficulties of LLMs in Two-Operand Addition
• 5.1 Predicted Accuracy
• 5.2 Finegrained Analysis
• Data.
• Results.
• 6 H1 Predicts the Deterioration of Accuracy in Multi-Operand Addition
• 6.1 Multi-Operand Performance Predicted by H1
• 6.2 Empricial Evidence on Multi-Operand Addition
• 7 Multi-Digit Tokenization Models Employ the Same Heuristic
• Experimental Setup.
• Carry Scenarios.
• Expected Outcomes.
• Results.
• 8 Related Work
• 9 Conclusion
• Limitations
• Acknowledgements
• Appendix A Example Addition According to Formalization
• Appendix B Generation Accuracies for 2-Operand, 3-Digit Addition
• Appendix C Example: H1 Failure on 4-Operand Addition
• Appendix D Zero-shot Generation Accuracy
• Appendix E Accuracy Prediction of Heuristic
• Appendix F Generation Accuracy on All Datasets
• Appendix G Probing Accuracy on Carry Scenarios
• Results.
• Instructions for reporting errors