The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs — part 2/3
Cleaned-text code points [15820, 31626). Bibliography omitted here; source.md includes it.
See Figure 4 . We need T 2 h T_{2}^{h} and thus c 2 h c_{2}^{h} to generate the first result digit s 2 h s_{2}^{h} . c 2 h ∈ { ⌊ 4 + 9 + c m i n 10 ⌋ , ⌊ 4 + 9 + c m a x 10 ⌋ } c_{2}^{h}\in\{\left\lfloor\frac{4+9+c_{min}}{10}\right\rfloor,\left\lfloor\frac{4+9+c_{max}}{10}\right\rfloor\} = { ⌊ 13 10 ⌋ , ⌊ 14 10 ⌋ } = { 1 , 1 } =\{\left\lfloor\frac{13}{10}\right\rfloor,\left\lfloor\frac{14}{10}\right\rfloor\}=\{1,1\} therefore c 2 h = 1 c_{2}^{h}=1 , T 2 h = 4 T_{2}^{h}=4 , and s 2 h = 4 s_{2}^{h}=4 . H1 succeeds in predicting the first digit s 2 s_{2} for 147 + 293 .
#### 147 + 255.
See Figure 5 . c 2 h ∈ { ⌊ 4 + 5 + c m i n 10 ⌋ , ⌊ 4 + 5 + c m a x 10 ⌋ } c_{2}^{h}\in\{\left\lfloor\frac{4+5+c_{min}}{10}\right\rfloor,\left\lfloor\frac{4+5+c_{max}}{10}\right\rfloor\} = { ⌊ 9 10 ⌋ , ⌊ 10 10 ⌋ } = { 0 , 1 } =\{\left\lfloor\frac{9}{10}\right\rfloor,\left\lfloor\frac{10}{10}\right\rfloor\}=\{0,1\} therefore c 2 h c_{2}^{h} is chosen uniformly at random between 0 0 and 1 1 . The heuristic fails in predicting the first digit s 2 s_{2} for 147 + 255 with a 50% chance.
## 5 H1 Predicts Difficulties of LLMs in Two-Operand Addition
In this section we show that single-digit token LLMs struggle exactly in those cases in which the heuristic H1 is insufficient.
### 5.1 Predicted Accuracy
For two-operand addition, there are 19 possible values for each t i t_{i} (ranging from 0 to 18, because this is the range of sums between two digits). In 18 out of these 19 cases, H1 reliably determines the correct carry value. Only if t i = 9 t_{i}=9 , H1 must randomly choose between two possible carry values, thus failing with a 50% chance. This results in an overall predicted accuracy of 18 × 1.0 + 1 × 0.5 19 = 0.974 \frac{18\times 1.0+1\times 0.5}{19}=0.974 for the first result digit s 2 s_{2} in two-operand addition: H1 achieves 97.4% accuracy in correctly predicting the first result digit s 2 s_{2} . This corresponds almost exactly to Gemma’s and Mistral’s accuracies for generating s 2 s_{2} during zero-shot and one-shot inference (Gemma: 0-shot: 97.12 % 97.12\% , 1-shot: 98.04 % 98.04\% ; Mistral: 0-shot: 94.60 % 94.60\% , 1-shot: 97.46 % 97.46\% ). Table 3 in Appendix F provides all generation accuracies for the data described in Section 2.1 .
### 5.2 Finegrained Analysis
We further investigate whether it is true that especially cases with t i = 9 t_{i}=9 are challenging for LLMs.
#### Data.
To this end, we evaluate prediction accuracy across five distinct newly introduced datasets, each containing 100 queries with distinct carry scenarios. The datasets follow the zero-shot template described in Section 2.1 and are designed to exhaustively capture all cases of carries affecting s 2 s_{2} in two-operand addition of triple-digit numbers. • Dataset 1 (DS1): No carry. The addition does not produce any carry (e.g., 231 + 124 = 355 231+124=355 ). 2 2 2 We employ the additional constraint that the sum of the 10 1 10^{1} operand digits ≠ 9 \neq 9 , i.e., ( s 1 ≠ 9 s_{1}\neq 9 ) .
• Dataset 2 (DS2): Carry in position 10 0 10^{0} , no cascading. A carry is generated in the 10 0 10^{0} ( s 0 s_{0} ) digit but does not cascade to the 10 2 10^{2} ( s 2 s_{2} ) digit (e.g., 236 + 125 = 361 236+125=361 ).
• Dataset 3 (DS3): Cascading carry from 10 0 10^{0} to 10 2 10^{2} . A carry originates in the 10 0 10^{0} ( s 0 s_{0} ) digit and cascades to the 10 2 10^{2} ( s 2 s_{2} ) digit (e.g., 246 + 155 = 401 246+155=401 ).
• Dataset 4 (DS4): Direct carry in position 10 1 10^{1} . A carry is generated in the 10 1 10^{1} ( s 1 s_{1} ) digit and directly affects the 10 2 10^{2} ( s 2 s_{2} ) digit (e.g., 252 + 163 = 415 252+163=415 ).
• Dataset 5 (DS5): No carry, but position 10 1 10^{1} digits sum to 9. There is no carry in any digit, but the sum of the 10 1 10^{1} operand digits is 9, i.e., ( s 1 = 9 s_{1}=9 ) (e.g., 256 + 142 = 398 256+142=398 ).
DS1 to DS5 can be neatly categorized according to whether the heuristic can accurately predict s 2 s_{2} :
• DS1 and 2: t 1 = ∑ j = 1 2 n j , 1 < 9 → c 2 h = 0 t_{1}=\sum_{j=1}^{2}n_{j,1}<9\rightarrow c_{2}^{h}=0
• DS4: t 1 = ∑ j = 1 2 n j , i > 9 → c 2 h = 1 t_{1}=\sum_{j=1}^{2}n_{j,i}>9\rightarrow c_{2}^{h}=1
• DS3 and 5: t 1 = ∑ j = 1 2 n j , 1 = 9 → c 2 h = ? t_{1}=\sum_{j=1}^{2}n_{j,1}=9\rightarrow c_{2}^{h}=?
#### Results.
Figure 6 shows that LLMs struggle with DS3 and DS5, which are precisely the cases where H1 predicts issues. As H1 suggests, predicting the first result digit s 2 s_{2} at position 10 2 10^{2} is particularly error-prone in these scenarios. The difficult datasets are the ones where a lookahead of one digit position does not suffice to determine the value of the carry needed to generate s 2 s_{2} . Simply put: Overall, addition results tend do be predicted correctly by LLMs, if and only if a lookahead of one digit is sufficient to determine the value of the carry bit affecting s 2 s_{2} . Prediction is often incorrect if a lookahead of two or more digits is needed to determine the value of the carry bit affecting s 2 s_{2} .
In cases where a lookahead of one digit is enough to accurately determine the value of s 2 s_{2} (DS1, DS2, DS4), the models succeed. However, when a lookahead of one digit is insufficient to determine the value of s 2 s_{2} (DS3 and DS5), the model struggles with predicting s 2 s_{2} correctly. Table 1 in Appendix B provides the generation accuracy of s 2 s_{2} for Gemma and Mistral, in addition to the plot. Additionally, Appendix G presents probing experiments that yield the same results.
## 6 H1 Predicts the Deterioration of Accuracy in Multi-Operand Addition
As shown in the last section, H1 is a good approximator for LLM behaviour on two-operand addition: In the majority of cases, a lookahead of one digit is sufficient to accurately determine the value of the carry bit affecting s 2 s_{2} . With a look-ahead of one digit, H1 predicts a failure of the generation of s 2 s_{2} , if and only if the value of s 1 s_{1} does not suffice to determine the value of the carry bit. In two-operand addition in base 10, this is the case if and only if t 1 = 9 t_{1}=9 . We now show that H1 can also account for model performance on multi -operand addition.
### 6.1 Multi-Operand Performance Predicted by H1
The possible value of a carry increases with increasing numbers of operands. For instance in 4-operand addition ( k = 4 k=4 ) the maximal value of a carry is 3 3 : c m a x ( 4 ) = ⌊ ∑ j = 1 4 9 10 ⌋ = 3 c_{max}(4)=\left\lfloor\frac{\sum_{j=1}^{4}9}{10}\right\rfloor=3
Therefore the carry heuristic c i h c_{i}^{h} is unreliable in 4-operand addition whenever t i − 1 = ∑ j = 1 k n j , i − 1 ∈ { 7 , 8 , 9 , 17 , 18 , 19 , 27 , 28 , 29 } t_{i-1}=\sum_{j=1}^{k}n_{j,i-1}\in\{7,8,9,17,18,19,27,28,29\} .
Put simply, because the value of the carry can be larger for more operands, the proportion of values of s 1 s_{1} for which the heuristic is insufficient (with its lookahead of one) increases with an increasing number of operands .
Consider an example in which the heuristic fails in 4-operand addition for clarification (see Figure 9 in Appendix C ):
186 + 261 + 198 + 256. t 1 = 8 + 6 + 9 + 5 = 28 c 2 h ∈ { ⌊ c m i n + 28 10 ⌋ , ⌊ c m a x + 28 10 ⌋ } \begin{split}t_{1}=8+6+9+5=28\\ c_{2}^{h}\in\{\left\lfloor\frac{c_{min}+28}{10}\right\rfloor,\\ \left\lfloor\frac{c_{max}+28}{10}\right\rfloor\}\end{split}
with c m a x = 3 c_{max}=3 c 2 h ∈ { ⌊ 28 10 ⌋ , ⌊ 31 10 ⌋ } = { 2 , 3 } c_{2}^{h}\in\{\left\lfloor\frac{28}{10}\right\rfloor,\left\lfloor\frac{31}{10}\right\rfloor\}=\{2,3\} therefore c 2 h c_{2}^{h} is chosen uniformly at random between 2 2 and 3 3 . The heuristic thus fails in solving 186 + 261 + 198 + 256 with a chance of 50%.
For 4-operand addition, there are 37 possible sums for the second digits (ranging from 0 to 36). In 28 out of these 37 cases, the heuristic reliably determines the correct carry bit. However, when t 1 ∈ { 7 , 8 , 9 , 17 , 18 , 19 , 27 , 28 , 29 } t_{1}\in\{7,8,9,17,18,19,27,28,29\} , the heuristic must randomly choose between two possible carry values, leading to a 50% chance of selecting the correct one. This results in an overall accuracy of: 28 × 1.0 + 9 × 0.5 37 = 0.878 \frac{28\times 1.0+9\times 0.5}{37}=0.878 Thus, the heuristic only achieves 88% accuracy in correctly predicting the first result digit s 2 s_{2} in 4-operand addition, compared to the 97% accuracy in two-operand addition. In Appendix E , we provide exact values for s 2 s_{2} accuracy as predicted by H1 , for addition tasks between 2 and 11 operands.
### 6.2 Empricial Evidence on Multi-Operand Addition
Intuitively, according to H1 , Mistral and Gemma with their one-digit tokenization should fail at multi-operand addition at a certain rate: The amount of instances in which a lookahead of one digit is sufficient to accurately predict s i s_{i} gets smaller and smaller because the carry bit value can get larger and larger for multiple operands. We test if H1 holds in predicting the first generated digit s d s_{d} in Mistral and Gemma for multiple operands. We evaluate prediction accuracy on the multi-operand datasets described in Section 2.1 . H1 should provide an upper bound for the performance of LLMs 3 3 3 Autoregressive LLMs with single-digit tokenization of numbers. for predicting the first result digit s d s_{d} . Figure 7 shows that H1 is a good predictor for the accuracy of the one-shot 4 4 4 Results for the zero-shot setting are in Appendix D . generation of the first result digit s d s_{d} by Mistral and Gemma. We take this as further evidence that these LLMs make use of H1 .
## 7 Multi-Digit Tokenization Models Employ the Same Heuristic
While Levy and Geva (2024) demonstrate that all LLMs, regardless of the tokenization strategy, internally represent numbers as individual digits, it remained unclear whether models with multi-digit tokenization also rely on a one-digit lookahead when generating addition results. In this section, we show that perhaps surprisingly multi-digit tokenization models, such as Llama-3, also employ a lookahead of one digit when predicting carry bits. To show this, we design 3 controlled datasets that force the multi-digit tokenization model Llama-3 to generate results across multiple tokens.
#### Experimental Setup.
To examine whether Llama-3 employs a one-digit lookahead, we use six-digit numbers in two-operand addition (e.g., “231234 + 124514 = ”), where each operand is tokenized into two three-digit tokens by the model’s tokenizer, such as: [“ 231”,“ 234”, “ +”, “ 124”, “ 514”, “ =”] and the result is generated as two triple-digit tokens as well, in this example [“ 355”, “ 748”]. The first generated triple-digit token s 5 s 4 s 3 s_{5}s_{4}s_{3} corresponds to digit base positions 10 5 10^{5} , 10 4 10^{4} , and 10 3 10^{3} . If Llama-3 did employ H1 it would look ahead to digit position 10 2 10^{2} , but ignore digit positions 10 1 10^{1} and 10 0 10^{0} , as they fall outside the lookahead window.
#### Carry Scenarios.
We evaluate model behavior in three datasets with six-digit operands (ranging from 100,000 to 899,999) and results between 200,000 and 999,999. We use a zero-shot prompt template. Each dataset consist of 100 samples: • DS6: No carry. The addition does not produce any carry and no digits sum to 9. (e.g., 111,234 + 111,514 = 222,748 111,234+111,514=222,748 ).
• DS7: Direct carry in position 10 2 10^{2} . A carry is generated at 10 2 10^{2} and directly affects 10 3 10^{3} (e.g., 111,721 + 111,435 = 223,156 111,721+111,435=223,156 ).
• DS8: Cascading carry from 10 1 10^{1} to 10 3 10^{3} . A carry originates at 10 1 10^{1} , cascades to 10 2 10^{2} and then affects 10 3 10^{3} (e.g., 111,382 + 111,634 = 223,016 111,382+111,634=223,016 ).
#### Expected Outcomes.
If Llama-3 employs H1 , we expect that DS6 should be easy, as no carry propagation is required. DS7 should also be easy, since the carry affecting 10 3 10^{3} is within the one-digit lookahead window. DS8 in contrast should be challenging, as the carry originates from 10 1 10^{1} , from beyond the model’s lookahead range. We expect a lower accuracy in generating 10 3 10^{3} , the result digit that is affected by the potentially inaccurate carry.
#### Results.
Figure 8 shows that Llama-3 exhibits the expected pattern predicted by H1 . The sharp drop in accuracy in dataset DS8 on digit 10 3 10^{3} provides evidence that Llama-3, regardless of its multi-digit tokenization strategy, relies on the same one-digit lookahead for solving addition left to right.
## 8 Related Work
Recent work has benchmarked the arithmetic capabilities of LLMs using text-based evaluations and handcrafted tests Yuan et al. (2023) ; Lightman et al. (2023) ; Frieder et al. (2023) ; Zhuang et al. (2023) . Numerous studies consistently show that LLMs struggle with arithmetic tasks Nogueira et al. (2021) ; Qian et al. (2022) ; Dziri et al. (2023) ; Yu et al. (2024) .
Zhou et al. (2023) and Zhou et al. (2024) examine transformers’ ability to learn algorithmic procedures and find challenges in length generalization Anil et al. (2022) . Similarly, Xiao and Liu (2024) propose a theoretical explanation for LLMs’ difficulties with length generalization in arithmetic. Gambardella et al. (2024) find that LLMs can reliably predict the first digit in multiplication but struggle with subsequent digits.
The focus of research has recently shifted from mere benchmarking of LLMs to trying to understand why LLMs struggle with arithmetic reasoning. Using circuit analysis, Stolfo et al. (2023) and Hanna et al. (2023) explore internal processing in arithmetic tasks, while Nikankin et al. (2024) reveal that LLMs use a variety of heuristics managed by identifiable circuits and neurons. In contrast, Deng et al. (2024) argue that LLMs rely on symbolic pattern recognition rather than true numerical computation. Recently, Kantamneni and Tegmark (2025) showed that LLMs represent numbers as generalized helixes and perform addition using a “Clock” algorithm Nanda et al. (2023) .
Related work has also examined how LLMs encode numbers. Levy and Geva (2024) demonstrate that numbers are represented digit-by-digit, extending Gould et al. (2023) , who find that LLMs encode numeric values modulo 10. Zhu et al. (2025) suggest that numbers are encoded linearly, while Marjieh et al. (2025) indicate that number representations can blend string-like and numerical forms.
Another line of research explores how tokenization influences arithmetic capabilities. Garreth Lee and Wolf (2024) show that single-digit tokenization outperforms other methods in simple arithmetic tasks. Singh and Strouse (2024) highlight that right-to-left (R2L) tokenization—where tokens are right-aligned—improves arithmetic performance. Additionally, the role of embeddings and positional encodings is emphasized by McLeish et al. (2024) , who demonstrate that suitable embeddings enable transformers to learn arithmetic, and by Shen et al. (2023) , who show that positional encoding improves arithmetic performance.
## 9 Conclusion
Our study shows that LLMs, regardless of their numeric tokenization strategy, rely on a simple one-digit lookahead heuristic for anticipating carries when performing addition tasks. While this strategy is fairly effective for two-operand additions, it fails in the multi-operand additions due to the increasingly unpredictable value of cascading carry bits. Through probing experiments and targeted evaluations of digit-wise result accuracy, we demonstrate that model accuracy deteriorates precisely at the rate the heuristic predicts.
These findings highlight an inherent weakness in current LLMs that prevents them from robustly generalizing to more complex arithmetic tasks.