AI-rithmetic — part 1/2

Cleaned-text code points [0, 15863). Bibliography omitted here; source.md includes it.

# AI-rithmetic Thanks: Authors are listed in alphabetical order. Correspondence to {alexbie, tdick, kulesza, pragh, vinodraman, sergeiv}@google.com .

###### Abstract

Modern AI systems have been successfully deployed to win medals at international math competitions, assist with research workflows, and prove novel technical lemmas. However, despite their progress at advanced levels of mathematics, they remain stubbornly bad at basic arithmetic, consistently failing on the simple task of adding two numbers. We present a systematic investigation of this phenomenon. We demonstrate empirically that all frontier models suffer significantly degraded accuracy for integer addition as the number of digits increases. Furthermore, we show that most errors made by these models are highly interpretable and can be attributed to either operand misalignment or a failure to correctly carry; these two error classes explain 87.9%, 62.9%, and 92.4% of Claude Opus 4.1, GPT-5, and Gemini 2.5 Pro errors, respectively. Finally, we show that misalignment errors are frequently related to tokenization, and that carrying errors appear largely as independent random failures.

## 1 Introduction

Dramatic improvements in AI systems have led to surprising new applications in many areas, including advanced mathematics. While computer-assisted proofs have been around since the 4-color theorem of Appel and Haken (1976) , there has been an explosion of progress in the recent years. Large language models from Google and OpenAI have won medals in international competitions, developing proofs to new problems in algebra, geometry, and analysis ( Luong and Lockhart, 2025 ; Wei, 2025 ) . They have been used in research mathematics to find new lemmas, or improve existing proofs ( Gans, 2025 ; Novikov et al., 2025 ; Nagda et al., 2025 ; Georgiev et al., 2025 ) . And mathematicians of all levels of experience, from graduate students to Fields medals winners, are using them to solve open problems and advance the state of the art ( Fountoulakis, 2025 ; Sellke and Yin, 2025 ) .

At the same time, it is widely known (and we confirm here) that these systems are remarkably unreliable at basic arithmetic, sometimes failing to correctly add even two- or three-digit numbers. Similar failures such as an inability to count the number of ‘r’s in “strawberry” or solve simple equations ( shaun.ralston, 2025 ) have also been observed. Of course, this is not inherently contradictory: arithmetic is different from advanced mathematics in many ways. But for humans, these skills are both clearly related and strongly ordered, arithmetic being a simple but critical part of the foundation on which deeper mathematics is built. Therefore, it may seem surprising that AI systems can outperform experts at more “advanced” tasks but fail to keep pace with kindergartners in the “simple” ones.

Accepting that AI systems do not exhibit human-like patterns of performance, we might instead re-imagine them as algorithmic machines performing complex behaviors by composing simple ones (as in, for example, long addition). Indeed, this is the idea behind the current wave of reasoning models trained to perform a series of intermediate operations to reach a final answer. But this compositional view also fails to describe arithmetic performance, which we find is not even monotonic: AI systems are sometimes better at adding longer numbers than shorter ones. Moreover, while bigger models generally perform better than smaller models, all current models make significant numbers of mistakes as the problem size grows, suggesting that scaling alone is not likely to solve the problem, even though only a finite number of simple behaviors need to be learned and orchestrated by the reasoner.

Arithmetic, then, is an interesting case study in how modern AI systems can confound intuitions. In this work, we systematically investigate model behavior on simple integer addition problems, characterizing mistakes and identifying patterns. We find that errors are generally explainable in intuitive terms, offering a view of AI systems that is neither strictly human-like nor purely compositional, but idiosyncratic and dependent on design choices such as tokenization and auto-regression.

While tool use or task-focused training can probably be used to address the specific arithmetic difficulties we discuss here, the larger point is that today’s models still have considerable weaknesses that we cannot fully anticipate. Manually correcting individual deficiencies as they are discovered does not seem like a viable path toward reliable general-purpose systems (if that is the goal). Instead, our aim is to use arithmetic as a convenient probe to understand some of today’s AI failures, and hopefully begin to illuminate larger problems that have yet to be solved.

### 1.1 Summary of Findings

We conduct systematic experiments on a collection of frontier large language models, finding that: • When adding two numbers, all frontier models perform poorly as the length of the numbers increases (Figure 1 ). We also find that performance is not necessarily monotone in length.

• Most observed mistakes are due either to misalignment or close carry errors (Figure 2 ). Misalignment errors are when one of the input operands is incorrectly shifted. Close carry errors are when the model incorrectly carries or fails to carry in borderline cases (that is, where the sum of digits is 9 but the model still carries a 1, or the sum is 10 and the model fails to carry). Misalignment and close carry errors comprise 87.9%, 62.9%, and 92.4% of Claude Opus 4.1, GPT-5, and Gemini 2.5 Pro errors, respectively.

• Further investigation into error classes reveals that: – Misalignment errors are often periodic with respect to argument length, a fact that may be explained by tokenization, especially for models that use multi-digit tokens (Section 4.1 ).

– The occurrence of close carry errors is largely consistent with an independent error model where models err on each close carry independently with probability p p (Section 4.2 ). If an addition problem contains n n close carries, this predicts a ( 1 − p ) n (1-p)^{n} chance of getting the problem correct (i.e., the LeCun (2023) model), and a geometrically distributed first error position.

### 1.2 Related Work

Although language models can perform arithmetic tasks with non-trivial accuracy in certain settings ( Zoph et al., 2022 ; Yang et al., 2023 ; Maltoni and Ferrara, 2024 ) , their failure to generalize and overall poor performance have been widely observed ( Saxton et al., 2019 ; Nogueira et al., 2021 ; Dziri et al., 2023 ; Qian et al., 2023 ; Gambardella et al., 2024 ; Testolin, 2024 ; Yan et al., 2025 ; Loeber, 2024 ) .

Recent work has focused on understanding how AI systems represent numbers or implement arithmetic operations ( Stolfo et al., 2023 ; Dziri et al., 2023 ; Nikankin et al., 2025 ; Deng et al., 2024 ; Zhang et al., 2024 ; Baeumel et al., 2025b ; Levy and Geva, 2025 ; Kantamneni and Tegmark, 2025 ) , as well as how they can be improved. Techniques include prompting or training with chain-of-thought ( Wei et al., 2022 ; Lee et al., 2023 ) , augmenting the model’s abilities with symbolic systems ( Yang et al., 2024 ; Dugan et al., 2024 ) , adding hints or restructuring the problem ( Nogueira et al., 2021 ; Zhou et al., 2023 ) , increasing numerical precision ( Feng et al., 2024 ) , and utilizing improved positional encodings ( Shen et al., 2023 ; McLeish et al., 2024 ; Zhou et al., 2024 ) .

In terms of error analysis, Singh and Strouse (2024) find that enforcing right-to-left tokenization with separators improves accuracy for GPT-4 (which employs multi-digit tokenization), and that errors are highly position-dependent rather than problem difficulty-dependent. In this vein, our analysis uncovers that accuracy is periodic with respect to length (Figure 6 ) for GPT models; furthermore we identify that these tokenization-induced errors are due to misalignment. Baeumel et al. (2025a) analyze close carry errors, attributing them to the inability of models to anticipate cascading carries. They conduct an in-depth analysis on multi-operand, 3-digit addition on smaller models, while we examine two-operand, n n -digit addition and frontier reasoning models. Our focus on operand length leads us to uncover the independent nature of close carry errors. Nikankin et al. (2025) use tools from mechanistic interpretability to identify the exact circuits that pretrained LLMs use to add, observing that an arithmetic problem activates several independent (flawed) mechanisms that contribute to promoting the correct answer. Nikankin et al. (2025) primarily study Llama models, and restrict their focus to answers computed in a single forward pass (e.g. input operands and the output answer are a single 3-digit token).

## 2 Model Performance

In this section, we investigate the performance of frontier LLMs on adding two numbers.

#### Setup.

We use prompts of the form: What is A A + B B ? Write just the answer.

For each length d ∈ { 1 , … , 100 } d\in\{1,\ldots,100\} , we generate 100 prompts by setting A A and B B to be d d -digit numbers chosen independently and uniformly at random from { 10 d − 1 , … , 10 d − 1 } \{10^{d-1},\ldots,10^{d}-1\} . We send each prompt to each model and collect their responses.

To extract an answer from the model’s response, we remove all commas (to account for different numerical formatting conventions) and then extract the span of digits with the lowest edit distance to the correct sum A + B A+B . We say that the model is correct if the match is exact, and incorrect otherwise. This test is lenient: if the model response includes multiple spans of digits, we consider the response correct if any of them is correct. Table 1 gives examples of several correct and incorrect model responses when the true sum is 1234 .

#### Models.

We test a variety of frontier LLMs, including Claude Opus 4.1, GPT-4o, GPT-5, Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 2.5 Pro with Code Execution, and Gemma 3 27B. All models tested have been consumer facing at some point in their life cycle.

#### Results.

Figure 1 shows the fraction of correct responses a function of the length d d . With the exception of Gemini 2.5 Pro Code Execution, the accuracy of all models drops significantly below 100% as the number of digits increases. Even at a moderate length of 20, most models exhibit a nontrivial error rate. This suggests that the varieties of architectures, data mixtures, and training practices used by current models are not sufficient for learning to accurately perform basic addition.

Gemini 2.5 Pro with Code Execution’s stronger performance can be attributed to the fact that it usually calculates the sum using short python scripts. Nonetheless, we still observe errors. These occur when the model fails to use the tool or fails to correctly copy the answer calculated by the tool.

## 3 Exploration of Mistakes

Next, we explore the types of mistakes evident in Figure 1 . We find that a significant fraction of mistakes can be explained intuitively.

Recall that when adding a pair of numbers, the canonical long addition algorithm aligns the operands vertically and then computes a sequence of digit sums, one for each column, proceeding from least to most significant position. When a column sum exceeds 9, a one is “carried” to the next (more significant) column.

A large proportion of model mistakes are consistent with one of two natural failure modes during the execution of this algorithm: (1) misalignment, where the digits of one operand are shifted by a few positions in either direction, and (2) close carry failure, where a carry is performed in error when the column sum is 9, or not performed when the column sum is 10. (See below for the details of how we identify such errors.)

Table 2 shows the proportion of all incorrect answers given by each model that are consistent with one of these two error types. Notably, these two types cover 92.4% of Gemini 2.5 Pro’s mistakes, 87.9% of Claude Opus 4.1’s mistakes, and at least 55.6% of every model’s mistakes.

#### Misalignment Errors.

We say that a model’s extracted answer for the sum of A A and B B contains a misalignment error if at least the first 6 digits of the extracted answer match the sum when adding A A and B B with the arguments offset by up to 10 digits in either direction. We limit the test to six digits because models tend to compound their errors as they go, making long misaligned matches unlikely. We require at least six digits to avoid false detections.

More specifically, for each offset s ∈ { − 10 , … , − 1 , 1 , … , 10 } s\in\{-10,\ldots,-1,1,\ldots,10\} , we calculate offsetsum ⁡ ( A , B , s ) = A ⋅ 10 max ⁡ ( 0 , s ) + B ⋅ 10 max ⁡ ( 0 , − s ) \operatorname{offsetsum}(A,B,s)=A\cdot 10^{\max(0,s)}+B\cdot 10^{\max(0,-s)} and calculate the length of the shared prefix between the extracted model answer and offsetsum ⁡ ( A , B , s ) \operatorname{offsetsum}(A,B,s) . We identify the response as a misalignment error if the length is at least 6 and the extracted answer is not equal to one of the arguments. 2 2 2 When the model outputs A A or B B , we do not consider this to be a misalignment error, even though offsetsum ⁡ ( A , B , s ) \operatorname{offsetsum}(A,B,s) typically begins with | s | |s| digits from A A or B B . Note that matching 6 digits for an offset of size at most 10 is unlikely by chance: there are at most 20 distinct 6-digit prefixes obtained from the offset sums, yet approximately 10 6 10^{6} possible prefixes, so the probability of a uniformly drawn answer matching a six-digit prefix is ≈ 0.002 % \approx 0.002\% .

Table 3 shows examples of the prefixes that qualify as misalignment errors for A = 555555 A=555555 and B = 123456 B=123456 .

#### Close Carry Errors.

We use the following procedure to identify close carry errors. If the correct sum and the extracted model answer are different lengths, we pad the shorter to the left with zeros. Then we compare the extracted answer to the correct answer digit by digit from left to right—although long addition is performed from right to left, LLMs generate tokens autoregressively in the order they appear. When we reach the first digit at which an error occurs, we check if it is consistent with a close carry error: • If the digit is too large by 1 and the column to the right has a correct column sum of 9, then we say it is a close carry error.

• If the digit is too small by 1 and the column to the right has a correct column sum of 10, then it is also a close carry error.

If the response contains no errors, or the first error does not satisfy either case above, then we do not identify it as a close carry error. Table 4 show some examples of the errors that would be considered close carries for small addition problems.

#### Runaway errors.

In addition to misalignment and close carry errors, we observed that some of the models frequently output answers that are much longer than the correct answer (sometimes thousands of digits long). We say that an extracted answer is a runaway error if its length is at least 50% longer than the true answer.

#### Results.

Figure 2 shows how the mistakes made by each model break down into aforementioned mistake types. Since the error types we consider are not mutually exclusive, we classify a model’s response according to the first mistake type it matches in the following order: runaway, misalignment, close carry. 3 3 3 The overlap between error types is low, and the order here does not have a large impact on the results. Among all models, we observe 8081 responses that are only identified as misalignment errors, 9347 that are only identified as close carries, and 543 that are identified as both.