Modular Arithmetic: Language Models Solve Math Digit by Digit — part 1/3
Cleaned-text code points [0, 15964). Bibliography omitted here; source.md includes it.
# Modular Arithmetic: Language Models Solve Math Digit by Digit
###### Abstract
While recent work has begun to uncover the internal strategies that Large Language Models (LLMs) employ for simple arithmetic tasks, a unified understanding of their underlying mechanisms is still lacking. We extend recent findings showing that LLMs represent numbers in a digit-wise manner and present evidence for the existence of digit-position-specific circuits that LLMs use to perform simple arithmetic tasks, i.e. modular subgroups of MLP neurons that operate independently on different digit positions (units, tens, hundreds). Notably, such circuits exist independently of model size and of tokenization strategy, i.e. both for models that encode longer numbers digit-by-digit and as one token. Using Feature Importance and Causal Interventions, we identify and validate the digit-position-specific circuits, revealing a compositional and interpretable structure underlying the solving of arithmetic problems in LLMs. Our interventions selectively alter the model’s prediction at targeted digit positions, demonstrating the causal role of digit-position circuits in solving arithmetic tasks.
## 1 Introduction
The emergence of mathematical abilities in large language models (LLMs) has sparked growing interest in uncovering the internal mechanisms that underlie their arithmetic reasoning capabilities Stolfo et al. (2023) ; Nikankin et al. (2024) ; Zhang et al. (2024) ; Lindsey et al. (2025) . Understanding whether LLMs solve arithmetic by applying generalizable strategies, relying on superficial heuristics, or merely memorizing training examples can provide crucial insights into the general nature of reasoning abilities in LLMs.
Despite recent progress, a unified account of how LLMs perform basic arithmetic has yet to be established. Nikankin et al. (2024) argue that LLMs rely on a bag of heuristics —a sparse set of neurons that are sensitive to simple operand or result patterns—rather than implementing coherent algorithms for arithmetic. Lindsey et al. (2025) describe a dual-pathway mechanism for Claude-3.5 Haiku, with separate pathways for estimating rough result magnitude and the result unit digit. At the same time, Levy and Geva (2024) and Gould et al. (2023) find that digit values – but not full numeric values – can be extracted from the residual stream via probing, suggesting that LLMs internally represent numbers in a digit-wise manner. This finding is suprising, as many LLMs represent numbers up to a certain value (e.g. 3 digits) as single tokens (e.g., “347” is one token in Llama 3 8B). Their work however does not investigate whether these numeric representations are also exploited for solving arithmetic tasks.
In this work, we take the next step in understanding how LLMs solve arithmetic tasks by presenting causal evidence for digit-position-specific arithmetic circuits, i.e., modular subgroups of MLP neurons that independently generate results for different digit positions (Figure 1 ). For instance, to solve 347 + 231 = 347+231= , the model generates results for 7 + 1 7+1 (units), 4 + 3 4+3 (tens), and 3 + 2 3+2 (hundreds) using three distinct circuits, formed by groups of MLP neurons that are selectively sensitive to individual digit positions and distributed across mid-to-late layers.
We identify the digit-position-specific circuits using Fisher Score-based feature selection Gu et al. (2012) ; Sun et al. (2021) and validate their causal role via targeted interventions Vig et al. (2020) ; Meng et al. (2023) : Altering the activation of the units-digit circuit for instance selectively changes only the predicted units digit while leaving tens and hundreds unchanged (Figure 2 ). Our findings offer important novel evidence that LLMs solve arithmetic tasks not merely through heuristics, but through structured, compositional arithmetic processes.
Our main contributions are:
(1) We identify digit-position-specific arithmetic circuits by using supervised Fisher Score-based feature selection to detect MLP neuron groups responsible for solving arithmetic digit-level-subtasks.
(2) We validate these circuits via causal interventions, demonstrating that they selectively control individual output digits.
(3) We confirm the existence of digit-position-specific arithmetic circuits across a wide range of models, tasks, and tokenization schemes.
We release all code and data to support reproducibility 1 1 1 https://github.com/tbaeumel/transformer-digit-arithmetic .
## 2 Method
### 2.1 Overview of our Approach
We investigate whether LLMs perform arithmetic in a digit-wise, position-specific manner by a two-step approach, following recent work in mechanistic interpretability:
1. Identify Digit-Specific Neurons through Feature Selection: We identify neurons that are involved in generating digit position specific subresults. We use a Fisher Score-based feature selection method to measure neuron sensitivity to sub-tasks at different digit positions.
2. Causal Verification via Interventions : To verify the causal role of the identified neuron groups, we perform targeted interventionsby replacing the activation of a digit-specific neuron group in a base prompt with that of a source prompt. This allows us to test whether the altered activations lead to predictable changes in the model’s output. Our results demonstrate that these neuron groups indeed implement digit-position-specific circuits: Only the digit corresponding to the intervened circuit changes, while the rest of the output remains unaffected. As illustrated in Figure 2 , intervening on the unit digit circuit of the base prompt 347 + 231 (expected output 578 ) with activations from the source prompt 261 + 512 (expected output 773 ) yields the output 573 . This output combines the hundreds and tens digits from the base result ( 57 ) with the unit digit from the source result ( 3 ), confirming the selective influence of the intervened circuit.
### 2.2 Data for Neuron Identification Experiments
We generate a simple addition dataset D a d d D_{add} and subtraction dataset D s u b D_{sub} by generating 1000 addition and 1000 subtraction prompts respectively. This data is used to identify digit-specific neurons (Step 1). The prompts are formulated in a one-shot setting of the form ‘‘157 o \mathrm{o} 431 = 588; A A o \mathrm{o} B B = ’’ , where o ∈ { + , − } \mathrm{o}\in\{+,-\} and A , B ∈ { 100 , … , 999 } A,B\in\{100,\dots,999\} . All addition and subtraction tasks are sampled such that the result is also a 3-digit integer, i.e., ∈ { 100 , … , 999 } \in\{100,\dots,999\} . To isolate digit-wise computations and prevent interactions between digit positions, we construct all data such that no carry occurs in any digit position. We present an investigation of carry bits in Section 4.2 .
### 2.3 Data for Intervention Experiments
For the causal intervention experiments , we construct paired prompts in a one-shot setting consisting of a base (e.g., ‘‘157 o \mathrm{o} 431 = 588; A A o \mathrm{o} B B = ’’ ) and a source (e.g., ‘‘157 o \mathrm{o} 431 = 588; C C o \mathrm{o} D D = ’’ ) where operator o ∈ { + , − } \mathrm{o}\in\{+,-\} and operands A , B , C , D ∈ { 100 , … , 999 } A,B,C,D\in\{100,\dots,999\} ),.
We construct two addition datasets D a d d , o p 1 D_{add,op1} and D a d d , o p 2 D_{add,op2} where operator o = + \mathrm{o}=+ , such that one operand is shared between base and source while the other varies across all digits ( D a d d , o p 1 D_{add,op1} : B = D B=D , A d i ≠ C d i ∀ digit positions d i A_{d_{i}}\neq C_{d_{i}}\forall\text{ digit positions }d_{i} ; D a d d , o p 2 D_{add,op2} : A = C A=C , B d i ≠ D d i ∀ d i B_{d_{i}}\neq D_{d_{i}}\forall d_{i} ). In the same way we construct D s u b , o p 1 D_{sub,op1} and D s u b , o p 1 D_{sub,op1} where o = − \mathrm{o}=- .
All datasets contain 200 pairs of unique arithmetic problems. To isolate digit-wise computations and prevent interactions between digit positions, we construct all data such that no carry occurs in any digit position.
### 2.4 Models
We evaluate four decoder-only transformer models: LLaMA3-8B, LLaMA3-70B Grattafiori et al. (2024) , OLMo 2 7B OLMo et al. (2024) , and Gemma 2 9B Team (2024) all of which achieve high performance on simple addition and subtraction tasks ( > > 95%, Table 1 ).
We focus our main analyses on models that employ multi-digit numeric tokenization strategies, i.e., their vocabularies contain multi-digit tokens like ‘‘147’’, i.e., LLaMA3-8B, Olmo 2 7B and LLaMA3-70B 2 2 2 All operands and correct results from all datasets appear as unique vocabulary tokens in all multi-digit tokenization models. . Any observed digit-wise processing must be a property of the model’s internal structure rather than a simple consequence of tokenization for such models, making it a more surprising and informative phenomenon. To demonstrate the generalizability of our findings, we also include results from a single-digit tokenization model (Gemma 2 9B) in Appendix B , which similarly exhibits digit-position-specific processing pathways.
To study the mechanisms that emerge from standard language model pretraining, we evaluate base models without any instruction tuning or arithmetic-specific fine-tuning.
## 3 Circuit Localization
We now describe the circuits localization steps in detail along with the results. We focus our analyses on LLaMA3-8B in the main text and report detailed analyses showing similar results for Olmo 2 7B and LLaMA3-70B in Appendices E and F .
### 3.1 Step 1: Identifying Digit-Specific Neurons
#### Intuition.
To identify MLP neurons that are sensitive to digit-level arithmetic subtasks, we perform supervised feature selection. Intuitively, we identify hich neurons are most discriminative for different arithmetic subtasks in individual digit positions.
We employ Fisher Score Duda and Stork (2001) ; Venkatesh and Anuradha (2019) as our feature selection method, which is an established information-theoretic method for assessing individual feature importance in high-dimensional spaces. Fisher Score measures how well a feature (in our case, a neuron’s activation) separates data points belonging to different classes (in our case, digit-level arithmetic tasks). A high Fisher score F i , d F_{i,d} of neuron i i at digit position d d indicates that the activation of neuron i i reliably discriminates between subtasks targeting that digit position. We interpret such neurons as likely members of digit-position-specific arithmetic subcircuits. For instance, if a neuron is similarly activated for the addition tasks ‘‘15 7 + 43 1 = ’’ , ‘‘23 7 + 36 1 = ’’ , and ‘‘76 7 + 21 1 = ’’ , chances are that it is involved in the generation of the unit result digit, as all examples have the same subtask 7 + 1 7+1 in the unit position.
We collect MLP activations across layers, for all prompts in D a d d D_{add} and D s u b D_{sub} and compute Fisher Scores for each neuron and digit position. We then rank neurons according to their Fisher Score, reflecting how strongly their activations vary with changes to specific digit positions in the input operands. By thresholding the scores (as a hyperparameter), we obtain candidate circuits composed of neurons sensitive to each digit position.
Following Stolfo et al. (2023) , we focus on MLP layers that could plausibly contribute to producing arithmetic output. We identify the earliest layer at which operand information has been propagated to the residual stream at the final token, as earlier layers lack relevant input context and are unlikely to contribute to the result generation. We therefore restrict our search for arithmetic circuits to layers at which operand information has been propagated. Appendix C provides the identified “operand injection” layers for all models and tasks, alongside a more detailed rationale.
#### Neuron Selection for Digit-Position Circuits.
Appendix D presents a detailed formalization of how Fisher Score F i , d F_{i,d} is calculated for each neuron.
For each MLP layer l l in model m m and o ∈ a d d , s u b \mathrm{o}\in{add,sub} in each digit position d d : 1. Compute average F i , d F_{i,d} for all neurons i i in l l over the dataset D o D_{\mathrm{o}} .
2. Select the neurons with F i , d F_{i,d} above a threshold t t as candidates for the digit-specific arithmetic circuit C m , o , d , t C_{m,\mathrm{o},d,t} .
For all models, we explore circuits C m , o , d , t C_{m,\mathrm{o},d,t} based on different thresholds t ∈ { 0.4 , 0.5 , 0.6 , 0.7 , 0.8 , 0.9 , 1.0 } t\in\{0.4,0.5,0.6,0.7,0.8,0.9,1.0\} . Fisher Scores are a relative metric, therefore choosing a reasonable threshold for circuit membership is a hyperparameter tuning problem. Generally speaking, Fisher scores close to 0 indicate a lack of discriminative power, scores between 0.1 and 1.0 weak to moderate discriminability, and values above 1.0 strong discriminability.
#### Statistics on Selected Circuits.
We provide detailed statistics for all identified circuits C m , o , d , t C_{m,\mathrm{o},d,t} across all models, operators, and digit positions in Appendix E and provide a brief summary of the findings here.
We find that digit-position-specific circuits consist of a significant portion available MLP neurons, for instance for C Llama3 8B , + , d , t C_{\text{Llama3 8B},\mathrm{+},d,t} the average number of MLP neurons per layer responsible for one of the digit-position specific circuits is 60.3% of all MLP neurons. Further, the sets of digit-position-specific MLP neurons are highly sufficient for representing digit-specific arithmetic subtasks, indicated by strong classificiation performance using only the digit circuit neurons. Importantly, we also find that neuron sets in different digit-position circuits are largely distinct (Figure 21 ), providing a first indication of digit-positional modularity in LLM arithmetic.
### 3.2 Step 2: Causal Verification via Interventions
We perform targeted interchange interventions Vig et al. (2020) ; Meng et al. (2023) on candidate digit-position circuits C m , o , d , t C_{m,\mathrm{o},d,t} replacing MLP activations of digit-position-specific neurons in a base prompt with those from a source prompt at selected layers.
#### Circuit Depth.
Based on the observations in Section 3.1 and Appendix C , we choose the MLP layer set considered for digit-position specific arithmetic circuits to be L L l a m a 38 B , a d d , o p 2 = { 15 , … , 24 } L_{Llama38B,add,op2}=\{15,...,24\} for D a d d , o p 2 D_{add,op2} and L L l a m a 38 B , a d d , o p 1 = { 16 , … , 24 } L_{Llama38B,add,op1}=\{16,...,24\} for D a d d , o p 1 D_{add,op1} . Ablation experiments (Appendix H ) show that choosing deeper circuits L L l a m a 38 B , a d d , o p 2 = { 15 , … , 28 } L_{Llama38B,add,op2}=\{15,...,28\} and L L l a m a 38 B , a d d , o p 2 = { 16 , … , 28 } L_{Llama38B,add,op2}=\{16,...,28\} does not make a significant difference. This supports our conclusion that late MLP layers are less responsible for generating digit-position specific arithmetic results.
#### Experiment.
Given a source ( 123 + 562 = 685 ) and base prompt ( 123 + 456 = 579 ), we intervene on the neurons in C m , o , d , t C_{m,\mathrm{o},d,t} . We then measure the model’s output distribution over all 8 possible digit-wise combinations of the base and source results (e.g., bbb , sbb , bsb , etc., where e.g. sbb indicates the the hundreths digits correspond to the source result and the other two digits to the base result). If our independent digit-wise circuit hypothesis is correct, a position-specific circuit should shift probability mass only toward the expected variant (e.g., sbb for a hundreds-digit intervention), with minimal effects on other positions.