AI-rithmetic — part 2/2

Cleaned-text code points [15863, 25503). Bibliography omitted here; source.md includes it.

Any mistakes that do not match the conditions for these three error types are classified as “other”. For Claude Opus 4.1, Gemini 2.5 Pro, GPT-4o, and GPT-5, the majority of their mistakes match at least one of the three mistake types. Most of the mistakes made by Claude Opus 4.1 are close carrying errors, while most of the mistakes made by Gemini 2.5 Pro and GPT-5 are misalignment errors.

Figure 3 shows histograms of the edit distance between the extracted model answer and the true answer for mistakes on problems of length 40 to 80. Each histogram bar is colored according to the types of the mistakes. Most models exhibit a bimodal distribution with a significant fraction of the low edit distance mistakes classified as close carry errors and many of the longer edit distance mistakes classified as misalignment errors. This is intuitive since close carry errors are local in nature, while a single misalignment can distort the entire result.

Figure 4 shows that close carry errors are significantly more prevalent than other carrying errors. The heat maps show the frequency of the model’s leftmost error delta (the difference between model’s digit and answer’s digit) and the column sum in the next position. Close carry errors are characterized by having a delta of -1 and next column sum of 10, or a delta of +1 and a next column sum of 9. We see that these combinations are much more frequent than any other combination, although each model has its own unique behaviors as well.

Finally, for misalignment errors, Figure 5 shows histograms of the offsets producing the longest prefix match. Recall that positive offsets correspond to padding the first argument on the right with 0s. With the exception of GPT-5, the models all nearly always have a positive offset. A possible explanation is that it is common practice to write the longer argument first when adding numbers of different lengths. We see that Gemini 2.5 Flash and Pro, and Gemma 3 27B all typically misalign the arguments by a single digit position. On the other hand, GPT-4o and Claude Opus 4.1 favor a 3 3 -digit misalignment. Note that these plots also show some evidence of periodicity in the offset frequencies for some models; we will return to this observation in Section 4.1 .

## 4 Explanation of Mistakes

### 4.1 Tokenization and Misalignment

As observed in Figure 5 , misalignment errors tend to favor certain argument offsets, which vary by model. Here we investigate further and present evidence that misalignment errors may be related to tokenization.

Figure 6 shows the unsmoothed accuracy of GPT-5 (which makes primarily misalignment errors) as a function of the argument length. Particularly for longer arguments, accuracy seems to behave in a periodic way. The right plot splits this curve into three parts, corresponding to lengths that are equivalent to 0, 1, or 2 mod 3. With the shaded regions indicating standard error, it is apparent that the accuracy of GPT-5 is significantly higher for arguments whose length is a multiple of three.

GPT-5 uses the tiktoken tokenizer ( OpenAI, 2025a ; OpenAI, 2025b ) , in which a pre-tokenization regular expression breaks spans of digits into groups of 3 digits, followed by a group containing the remaining 1 or 2 digits if the span length is not a multiple of three. Each of these groups is then represented by a single token. As a consequence, the tokenization of a d d -digit number begins with ⌊ d / 3 ⌋ \lfloor d/3\rfloor tokens that each represent 3-digit strings, followed by a token representing a 1- or 2-digit string if d mod 3 ≠ 0 d\mod 3\neq 0 . Figure 6 therefore suggests that GPT-5 is more likely to make mistakes precisely when 1- or 2-digit tokens are present.

To investigate whether this phenomenon explains misalignment errors more generally, we plot in Figure 7 the discrete Fourier transform of the misalignment error rate as a function of length for all of the models. As expected, we see a large magnitude for the frequency 1 / 3 \nicefrac{{1}}{{3}} for GPT-5, corresponding to the reduced error rate on lengths divisible by 3. In addition, we see pronounced spikes at 1 / 3 \nicefrac{{1}}{{3}} for GPT-4o, which also uses a 3-digit tokenizer ( OpenAI, 2025b ) , and Claude Opus 4.1, for which we find strong evidence of using a similar tokenization scheme. 4 4 4 Specifically, when querying the count_tokens API for claude-opus-4-1-20250805 ( Anthropic, 2025 ) , the increase in returned token count as a function of the number of digits d d is precisely ⌈ d / 3 ⌉ \lceil d/3\rceil , matching GPT-5 and GPT-4o. Meanwhile, we see no spikes at 1 / 3 \nicefrac{{1}}{{3}} for the other models, for which we find strong evidence that they tokenize digits individually. 5 5 5 Gemma 3’s tokenizer tokenizes digits individually ( Google, 2025 ) . For evidence that Gemini 2.5 tokenizes digits individually: Gemma Team (2025) reports using the same tokenizer as Gemini 2; and the count_tokens API for Gemini 2.5 Pro and Flash reports an increase of d d tokens when passed a d d -digit number.

These results suggest that misalignment errors may be exacerbated when operands are encoded with tokens containing variable numbers of digits. However, we also note that models using single-digit tokenizers tended to exhibit more misalignment errors overall, so it may be that longer tokens actually help to reduce misalignment errors, and this effect is muted when mixed-length tokens are used.

### 4.2 Independence and Close Carry Errors

As shown by Figure 3 , close carry errors are “local” in the sense that they tend to produce small overall edit distance between the model output and the correct answer. This suggests that an error made on one close carry may be relatively independent of other mistakes during the computation. Here we propose a simple one-parameter stochastic model of close carry errors, and show that it often matches the observed patterns of mistakes.

Assume a given addition problem involves n n close carries. We imagine that the model proceeds from left to right, producing the correct output digit at each position that does not involve a close carry. Upon reaching a close carry position, the model flips a coin and makes a close carry error with probability p p . If it makes an error, then the answer will be incorrect. If it does not make an error, it proceeds and, on reaching the next close carry position, flips another independent coin with the same bias p p . Continuing in this way, the model will ultimately produce a correct answer with probability ( 1 − p ) n (1-p)^{n} .

To assess whether this stochastic process is representative of the behavior of real models, we first choose a target of n = 15 n=15 close carries and then select only addition problems with exactly n n close carries. (In general, approximately 20% of columns produce a close carry, so the selected problems have a typical length of d = 75 d=75 .) For each model, we set p p so that the predicted error matches the observed error rate on these problems, including only problems for which the model either made a close carry error or was correct. We then examine, for each selected problem, the number of successful close carries before the first carry error, comparing the distribution observed empirically with the one predicted by the stochastic model. As shown in Figure 8 , the two distributions are often closely (though not perfectly) aligned.

These results are consistent with the idea that some models make close carry errors in a simple, independent way, and do not correlate errors across positions. This is particularly true for Gemma 3 27B. However, not every model’s behavior is fully explained in this way. In particular, several of the models show a lower propensity to make a mistake at the final close carry position, which is consistent with a general increase in accuracy for the lowest-order digits.

## 5 Discussion

Recent work has studied the representational power of neural networks (and transformer architectures in particular) ( Weiss et al., 2021 ; Schnabel et al., 2025 ; Sanford et al., 2024 ) . While the specific modeling choices vary, basic arithmetic generally falls in the class of tasks that can be solved exactly using modern architectures. In fact, addition is one of the examples used to explain how to “think like a transformer” ( Weiss et al., 2021 ) .

However, our evidence suggests that modern LLMs are not finding reliable implementations for addition, instead resorting to error-prone strategies that give poor results and exhibit design-dependent idiosyncrasies. Bigger models tend to perform better, but are dominated by a simple calculator, especially as the number of digits grows beyond 50. If we expect intelligent systems to perform this basic operation without falling back on specialized tools, then we will need new model designs, data sources, and training strategies.

Of course, arithmetic is itself just an example. Success at arithmetic is easy to measure, and we have ready expectations about how an intelligent system ought to perform. But today’s AI systems fail in many unique ways, some of which may be similarly easy to characterize and understand, and others which may be hard to delineate or even see clearly at all. Perhaps LLMs are simply not the right tools for tasks where it is important to identify a single correct answer. Or perhaps even “soft” probabilistic reasoning is too much to expect in complex settings. But we will need a clearer understanding of these boundaries and how to move beyond them if we want to develop robust AI deployments that consistently meet our expectations.