The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs — part 3/3

Cleaned-text code points [31626, 38555). Bibliography omitted here; source.md includes it.

Our work contributes to a broader understanding of LLM limitations in arithmetic reasoning and highlights increasing LLMs’ lookahead as a promising approach to enhancing their ability to handle complex numerical tasks.

## Limitations

Our work highlights limited lookahead as a key challenge for LLMs when adding multiple numbers. However, it remains unclear whether this limitation extends to other arithmetic operations, such as subtraction. Additionally, we cannot determine whether the limited lookahead is a heuristic explicitly learned for arithmetic tasks, or if it could also affect general language generation tasks as thus hinder performance of other tasks that require long-range dependencies. Future work should explore the depth of lookahead in tasks beyond arithmetic.

While the lookahead heuristic offers a straightforward explanation for the upper performance limit of LLMs on addition, it does not fully account for why LLMs still somewhat underperform relative to the heuristic in addition tasks with many operands (e.g., adding 8–11 numbers). We suspect this discrepancy may be related to limited training exposure to these many-operand addition tasks, but further investigation is needed to confirm this.

Our work also does not address whether larger models within the same family (e.g., 70B parameter models) exhibit a deeper lookahead. Future studies should examine whether scaling model size leads to improved performance by enabling a deeper lookahead.

Finally, we do not tackle methods to overcome the shallow lookahead. Future work should investigate whether targeted training on tasks requiring deeper lookahead can encourage models to deepen their lookahead.

## Acknowledgements

We thank Patrick Schramowski for his helpful feedback on the paper draft. This work has been supported by the German Ministry of Education and Research (BMBF) as part of the project TRAILS (01IW24005).

## Appendix A Example Addition According to Formalization

We show a concrete example for two-operand addition according to the formalization defined in Section 4 . For 147 + 255 147+255 , we have:

k = 2 , d = 3 , n ​ 1 = [ 1 , 4 , 7 ] , n ​ 2 = [ 2 , 5 , 5 ] k=2,d=3,n1=[1,4,7],n2=[2,5,5] .

We then compute:

T 2 = c 2 + 1 + 2 T_{2}=c_{2}+1+2 T 1 = c 1 + 4 + 5 T_{1}=c_{1}+4+5 T 0 = c 0 + 7 + 5 = 0 + 7 + 5 = 12 T_{0}=c_{0}+7+5=0+7+5=12 s 0 = 12 mod 10 = 2 , c 1 = ⌊ 12 10 ⌋ = 1 s_{0}=12\mod 10=2,\quad c_{1}=\left\lfloor\frac{12}{10}\right\rfloor=1 T 1 = 1 + 4 + 5 = 10 T_{1}=1+4+5=10 s 1 = 10 mod 10 = 0 , c 2 = ⌊ 10 10 ⌋ = 1 s_{1}=10\mod 10=0,\quad c_{2}=\left\lfloor\frac{10}{10}\right\rfloor=1 T 2 = 1 + 1 + 2 = 4 T_{2}=1+1+2=4 s 2 = 4 mod 10 = 4 , c 3 = ⌊ 4 10 ⌋ = 0 s_{2}=4\mod 10=4,\quad c_{3}=\left\lfloor\frac{4}{10}\right\rfloor=0 S = [ 0 , 4 , 0 , 2 ] S=[0,4,0,2]

The result of the addition is 402 402 .

## Appendix B Generation Accuracies for 2-Operand, 3-Digit Addition

We show the generation accuracy of the full result S S and the digit-wise accuracy of s 2 s_{2} , compared across the different carry bit datasets, as referenced in Section 4 . Table 1 shows that Gemma and Mistral struggle with the generation of the correct result digit s 2 s_{2} , exactly in the datasets that H1 predicts to be difficult. DS3 and DS5 contain addition tasks in which a lookahead of one digit is insufficient ot determine the value of s 2 s_{2} .

## Appendix C Example: H1 Failure on 4-Operand Addition

Below is an example in which the heuristic H1 fails in 4-operand addition, visualized in Figure 9 :

186 + 261 + 198 + 256. t 1 = 8 + 6 + 9 + 5 = 28 c 2 h ∈ { ⌊ c m ​ i ​ n + 28 10 ⌋ , ⌊ c m ​ a ​ x + 28 10 ⌋ } \begin{split}t_{1}=8+6+9+5=28\\ c_{2}^{h}\in\{\left\lfloor\frac{c_{min}+28}{10}\right\rfloor,\\ \left\lfloor\frac{c_{max}+28}{10}\right\rfloor\}\end{split}

with c m ​ a ​ x = 3 c_{max}=3 c 2 h ∈ { ⌊ 28 10 ⌋ , ⌊ 31 10 ⌋ } = { 2 , 3 } c_{2}^{h}\in\{\left\lfloor\frac{28}{10}\right\rfloor,\left\lfloor\frac{31}{10}\right\rfloor\}=\{2,3\} therefore c 2 h c_{2}^{h} is chosen uniformly at random between 2 2 and 3 3 . The heuristic thus fails in solving 186 + 261 + 198 + 256 with a chance of 50%.

## Appendix D Zero-shot Generation Accuracy

We test if H1 holds up in predicting the generation accuracy on s d s_{d} of Mistral and Gemma for multiple operands. Figure 10 shows that H1 provides an upper bound for the generation accuracy of s d s_{d} in a zero-shot setting for Mistral and Gemma on s d s_{d} .

## Appendix E Accuracy Prediction of Heuristic

Table 2 contains, for addition tasks with different numbers of operands k k , the maximum value of the carry c m ​ a ​ x ​ ( k ) c_{max}(k) . Based on c m ​ a ​ x c_{m}ax it list those values of t i t_{i} in which H1 is insufficient to accurately predict s 2 s_{2} . Based on the proportion of values of t i t_{i} for which H1 is sufficient to the total number of possible values, it lists the predicted accuracy for s 2 s_{2} .

## Appendix F Generation Accuracy on All Datasets

See Table 3 .

## Appendix G Probing Accuracy on Carry Scenarios

We evaluate probing accuracy of the probes trained in Section 3 across the five distinct carry scenarios, introduced in Section 5 .

#### Results.

Figure 11 shows that LLMs struggle with DS3 and DS5, which are exactly the cases where H1 would predict problems. The difficult datasets are the ones where a lookahead of one digit position does not suffice to determine the value of the carry needed to generate s 2 s_{2} . Simply put: In cases where a lookahead of one digit is enough to accurately determine the value of s 2 s_{2} (DS1, DS2, DS4), the models have a relatively good internal representation of the value of the second result digit s 1 s_{1} . This results in high performance on the currently generated digit s 2 s_{2} . However, when a lookahead of one digit is insufficient to determine the value of s 2 s_{2} (DS3 and DS5), the model struggles with representing digits s 1 s_{1} and s 2 s_{2} correctly.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .