Modular Arithmetic: Language Models Solve Math Digit by Digit — part 2/3

Cleaned-text code points [15964, 31810). Bibliography omitted here; source.md includes it.

We assess position-specificity of the intervention by checking whether the probability of the expected result variant increases selectively (e.g. for an intervention on hundreds digits sbb ). If other variants like ssb or sbs were to also increase, this would indicate that the intervention affects multiple digit positions. The specificity of the probability shift provides key evidence for digit-position-specific circuits in the MLP layers.

#### Results.

Table 2 contains results for the intervention on the best candidate digit-position circuits C m , o , d , t ∗ C_{m,\mathrm{o},d,t*} , where t ∗ t* is the best value for threshold t t . For all 4 datasets D o , o ​ p D_{\mathrm{o},op} ( o ∈ { a ​ d ​ d , s ​ u ​ b } ; op ∈ { o ​ p ​ 1 , o ​ p ​ 2 } \mathrm{o}\in\{add,sub\};\text{op}\in\{op1,op2\} ) and all models, we observe a consistent and substantial increase of the probability of the expected result variant – bbs for unit, bsb for tens, and sbb for hundreds – demonstrating that the interventions on the hypothesized digit-position circuits indeed only affect the desired digit position. This is evidence for the digit-position-specific nature of the identified circuits, and thus the causal role of digit-position specific processing in solving arithmetic tasks . For instance, an intervention on C Llama 3 8B , add , hundreds , 0.9 C_{\text{Llama 3 8B},\text{add},\text{hundreds},0.9} on D a ​ d ​ d , o ​ p ​ 2 D_{add,op2} yields a 45.56% probability point increase on the sbb result variant (correct hundreds digit from the source, with tens and unit digits from the base), from 0.23% before intervention to 45.79% after intervention. Specificity of the digit-position circuits is further supported by the low probability changes in result variants that contain multiple source digits ( bss , sbs , ssb , and sss ).

Additionally, Table 2 shows an expected reduction in the probability of the base result ( bbb ), indicating effective suppression of the default behavior. Importantly, non-target digits remain largely unaffected, as indicated by minimal changes observed in other result variants, confirming the specificity of the identified digit-position circuits.

We show detailed results on C Llama 3 8B , add , d , t C_{\text{Llama 3 8B},\text{add},d,t} in Figure 3 to show the effect of the chosen threshold t t . Additionally, Table 3 shows that the intervention is often successful in flipping the model’s prediction from the bbb result to the intended digit-specific result variant.

The results support our hypothesis: Intervening on the hypothesized circuits for the unit, tens, and hundreds digits leads to targeted and isolated changes in the corresponding digit of the model’s output. This is strong evidence that distinct circuits underlie the generation of arithmetic results in individual digits, and that the identified digit-position addition circuits are causally involved in generating digit-position specific sub-results.

## 4 Supplementary Experiments

To further strengthen our findings we present results from a series of supplementary experiments conducted on Llama 3 8B.

### 4.1 Addition and Subtraction Circuits are Largely Distinct

We are interested in whether subtraction and addition circuits are similar in their functionality or rather very distinct. To assess the similarity between addition and subtraction circuits, we investigate the overlap of selected MLP neurons for each digit position in each layer. Table 8 shows a low average overlap (Overlap in the top 100 MLP neurons: unit 19%, tens 9.2%, hundreds 19.8%) in selected neurons, indicating that addition and subtraction circuits rely on mostly distinct subsets of neurons.

### 4.2 Additions with Carry Bits Employ the Same Mechanisms

All previous experiments explicitly avoided carry operations. However, carries are a key reason why fully modular digit-wise processing does not always suffice in arithmetic: the result at one digit position can influence another. For example, 9 + 5 9+5 in the units position yields a carry of 1 1 to the tens position.

To examine how our digit-wise arithmetic circuits interact with carry propagation, we test whether targeted interventions remain effective when they introduce a carry. Specifically, we ask: Does a digit-specific circuit encode carry information, or is it handled elsewhere in the network?

We consider two scenarios involving carry-induced interventions (200 samples each):

#### (1) Carry from Units to Tens.

We intervene on the units-digit circuit of a base prompt with no carry, such as 347 + 231 = 578 347+231=578 , using activations from a source prompt that induces a carry from the unit to the tens position, such as 347 + 415 = 762 347+415=762 . After intervention on the units-digit circuit, we evaluate whether the output reflects (a) the base tens and hundreds digits with the source units digit (i.e., 572 = b ​ b ​ s 572=bbs ), or (b) if in addition to this a carry is propagated into the tens digit (i.e., 582 = b ​ b + 1 ​ s 582=bb_{+1}s ).

#### (2) Carry from Tens to Hundreds.

We intervene on the tens-digit circuit of a no-carry base prompt ( 347 + 231 = 578 347+231=578 ) with a source prompt that causes a carry from tens to hundreds ( 347 + 482 = 829 347+482=829 ). We compare whether the model outputs (a) the base hundreds and units digits with the source tens digit (i.e., 528 = b ​ s ​ b 528=bsb ), or (b) if the hundreds digit is carry-adjusted (i.e., 628 = b + 1 ​ s ​ b 628=b_{+1}sb ).

These interventions help determine whether carry information is localized within digit-specific circuits or handled separately by the model.

#### Results.

Figures 4 and 5 show that the bbs variant and the bsb variant, for the unit and tens carry interventions, respectively, exhibit the strongest increase in probability. At the same time, the effect on the result variants b ​ b + 1 ​ s bb_{+1}s and b + 1 ​ s ​ b b_{+1}sb is significantly smaller. This provides further evidence that digit-position-specific arithmetic circuits operate largely independently of one another and suggests that carry information is likely determined and processed by separate mechanisms, rather than being embedded in the digit-position circuits.

### 4.3 MLP Similarity Reflects Task Overlap

We investigate how similar digit-specific MLP activations are for prompts sharing the same digit-level arithmetic task at a given position (e.g., 154 + 635 154+635 and 137 + 611 137+611 share the same hundreds digit addition: 1 + 6 1+6 ). Specifically, we compute pairwise cosine similarity between activations of MLP neurons in the digit-position circuit d d , for samples with the same subcomputation at digit position d d . As a baseline, we compute cosine similarity in these neurons across 5000 random sample pairs.

As expected, Table 4 shows consistently high within-label similarity across digit positions. For example, in layer 15, pairs sharing the same unit-digit computation have an average similarity of 0.84 in unit-digit MLP neurons. This provides additional evidence for digit position-specific processing in mid-late MLP neurons.

## 5 Structured Arithmetic Processing or a Bag of Heuristics?

We are interested in how digit-position specific arithmetic processing fits in with previous findings that mathematical processing is solved by a sparse set of “heuristic” neurons that respond to specific operand or result patterns Nikankin et al. (2024) . These heuristics include, for example, results in a specific numerical range (e.g., results between 200 and 290), or results that share arithmetic properties (e.g., results congruent to 4 mod 5 4\mod 5 ).

We hypothesize that such heuristics may in part be fragments of digit-wise circuits, e.g., a neuron responding to results ≡ 4 mod 5 \equiv 4\mod 5 may reflect unit-digit sensitivity, while neurons responding to result ranges (e.g., 200–290) may reflect sensitivity to higher digit positions.

To qualitatively test this idea, we perform a small-scale exploratory analysis. In the spirit of Nikankin et al. (2024) we examine digit-specific activation patterns (instead of analyzing neurons across full operand values) to look for heuristic patterns in individual MLP neurons. We analyze the 20 highest Fisher Score MLP neurons per digit circuit (units, tens, hundreds) per layer across layers 15–24 in LLaMA 3 8B. We then generate digit-wise heatmaps for each neuron, plotting the mean activation as a function of the digit values in operand 1 and operand 2, for each digit position.

Our small qualitative study shows that many high-Fisher-score neurons exhibit highly structured and interpretable patterns, which mirror the kinds of heuristics described by Nikankin et al. (2024) : Certain neurons (Figures 45 and 44 ) respond to specific operand digit values, while others are sensitive to certain result ranges (Figure 7 ), result digits (Figure 46 ), or result parity (Figure 6 ) 3 3 3 All plots in https://github.com/tbaeumel/transformer-digit-arithmetic/tree/main/Digit-Heuristics_Top_Digit-Circuit_Neurons .

Importantly, the type of heuristic implemented by a neuron is typically aligned with the digit circuit that neuron belongs to, such that a range 900-999 result heuristic is implemented by a neuron within the hundreds-digit circuit, and the 0 mod 2 0\mod 2 result heuristic neuron is located within the unit-digit circuit.

While these are preliminary results they suggest that heuristic neurons may reflect digit-specific processing embedded within broader modular arithmetic circuits. In this view, heuristic neurons may be a manifestation of modular arithmetic circuits, rather than an alternative to them: they may be single-neuron approximations of low-level arithmetic subroutines tied to individual digit positions. While our analysis is qualitative and limited in scope, it highlights that compositional mechanisms and heuristic rules need not be mutually exclusive.

## 6 Related Work

Mechanistic Interpretability. Mechanistic interpretability (MI) aims to reverse-engineer the internal mechanisms of language models by analyzing weights and components. In transformer LLMs, a circuit refers to a small set of interconnected components (e.g., MLPs or attention heads) that collectively perform a specific computation Olah et al. (2020) ; Elhage et al. (2021) . Causal mediation techniques such as activation patching Vig et al. (2020) ; Meng et al. (2023) enable demonstrating the causal role of these circuits during generation. In this work, we identify digit-position-specific arithmetic circuits and verify their causal contribution through targeted interventions.

Arithmetic Reasoning Interpretability. Recent studies have begun to uncover how LLMs tackle arithmetic tasks Stolfo et al. (2023) ; Nikankin et al. (2024) ; Zhang et al. (2024) ; Lindsey et al. (2025) ; Baeumel et al. (2025) ; Zhou et al. (2024) . For example, Lindsey et al. (2025) describe a dual-pathway mechanism in a model, where one pathway estimates the rough magnitude of results and the other generates the precise unit digit. Nikankin et al. (2024) argue that LLMs rely on a “bag of heuristics” rather than a single coherent algorithm, with individual neurons implementing simple heuristics like “result % 5 = 0.” Stolfo et al. (2023) and others have used circuit analysis to reveal internal processing during arithmetic, while Deng et al. (2024) suggest LLMs mainly perform symbolic pattern recognition, not true numerical computation. Kantamneni and Tegmark (2025) recently proposed that LLMs represent numbers as generalized helices and perform addition using a “Clock” algorithm Nanda et al. (2023) . Despite this progress, a unified understanding of how LLMs perform basic arithmetic remains open.

Number Representation. Understanding how LLMs represent numbers internally has attracted considerable attention. Levy and Geva (2024) show that probes on LLM hidden states fail to recover exact numeric values directly, but succeed in recovering each digit in base 10, indicating digt-wise representation in of numbers in LLMs. This extends findings by Gould et al. (2023) , who showed LLMs encode numeric values modulo 10. Related work Zhu et al. (2025) ; Marjieh et al. (2025) suggests number representations can blend string-like and numerical forms or be encoded linearly. Our work is inspired by these representation-focused studies and shows that numbers are not only represented digit digit but also processed this way in simple arithmetic tasks.

## 7 Conclusion

We identify and validate digit-position-specific arithmetic circuits in LLMs. Through targeted interventions, we demonstrate that these circuits are not only highly specific and modular but also causally involved in generating individual digit outputs.

Our results suggest LLMs perform simple arithmetic digit by digit, with distinct circuits operating independently at the units, tens, and hundreds positions. This modular arithmetic structure is compatible with prior findings that LLMs use heuristic pathways and distributed neuron groups for arithmetic ( Nikankin et al., 2024 ) , however our results show LLMs solve arithmetic tasks in a far more organized way than previously thought.

## Limitations

While our results provide strong evidence for digit-position-specific arithmetic circuits in LLMs, our analysis is limited to addition and subtraction. More complex operations, such as multiplication and division, are not addressed in this work. Extending our framework to these tasks is an important direction for future work. We also leave the analysis of circuits responsible for composing digit-level results into final outputs to future work.

Our analysis focuses on MLP layers, which contain the neurons that directly control digit outputs. This captures the core computation but does not account for the role of attention heads or other residual stream components.

Finally, we use Fisher Score to identify neurons involved in arithmetic, as it is simple and effective across model scales. However, other methods—such as gradient-based attribution—may yield more precise circuits. Despite this, our identified circuits are causal, interpretable, and robust, demonstrating a strong proof of concept for neuron-level circuit discovery in language models.

## Acknowledgements

We thank Cennet Oguz for her helpful feedback on the paper draft. This work has been supported by the German Ministry of Education and Research (BMBF) as part of the project TRAILS (01IW24005).

## Appendix A Implementation Details

For our intervention experiments, we use the pyvene library Wu et al. (2024) to perform interchange interventions, where we intervene at a specific layer on a specific module at the last token position of a base prompt, with the corresponding activation from a source prompt ( Interchange Intervention ).

## Appendix B Results on Single Digit Tokenization Models

We present results from Gemma 2 9B, a single-digit tokenization model, to investigate whether it also exhibits digit-position-specific processing pathways.

#### Intuition: Single- vs Multi-digit Numeric Tokenization.

Before presenting results on Gemma 2 9B, we clarify why arithmetic processing may differ in models that tokenize numbers into single digits (e.g., “3”, “4”, “7”) versus a single multi-digit token (e.g., “347”).

The key distinction lies in the model’s output, not its input. A multi-digit tokenization model, such as those in the Llama 3 family, tokenizes an input like “347 + 231 = ” as [347, +, 231, =], and it is trained to generate a full multi-digit output token, such as “578”. In contrast, a single-digit tokenization model, such as from the Gemma 2 family, tokenizes the same input as [3, 4, 7, +, 2, 3, 1, =] and can only generate single-digit output tokens. It will therefore output “5”, not “578”.