Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics — part 3/5

Cleaned-text code points [29819, 45629). Bibliography omitted here; source.md includes it.

Does a competing arithmetic mechanism exist mid-training? Finally, we determine if the heuristics appear as the main arithmetic mechanism from early on in training, or if they co-exist with an unrelated mechanism that becomes vestigial in later checkpoints. We repeat the prompt-guided neuron knockout experiment ( Section 4.2 ) for each checkpoint; i.e., in each checkpoint, we sample 50 50 correctly completed prompts for each operator. For each prompt, we ablate 5 5 , 10 10 , and 25 25 heuristic neurons associated with the prompt in that checkpoint. We test if this targeted ablation significantly impairs the model’s accuracy, even in earlier stages of training, and compare this to a baseline, where we ablate a similar amount of randomly chosen heuristic neurons. The results ( Figure 10(c) ) demonstrate that removing any amount of neurons from heuristics associated with a prompt substantially reduces the model’s accuracy on these prompts even at earlier checkpoints, much more than ablating a random set of important neurons. We also observe that ablating 25 25 heuristic neurons per layer is enough to cause near-zero accuracy in all stages of training. This finding asserts that the causal link between a prompt’s associated heuristics and its correct completion exists throughout training.

## 6 Related work

Mechanistic interpretability (MI) aims to reverse-engineer mechanisms implemented by LMs by analyzing model weights and components. Causal mediation techniques ( Pearl, 2001 ) like activation patching ( Vig et al., 2020 ; Geiger et al., 2021 ) , path patching ( Wang et al., 2022 ) , and attribution patching ( Nanda, 2022 ; Syed et al., 2023 ; Hanna et al., 2024b ) allow localizing model behaviors to specific model components. Other studies have also presented techniques to explain the effect of specific weight matrices on input tokens ( Elhage et al., 2021 ; Dar et al., 2023 ) , or to analyze activations ( nostalgebraist, 2020 ; Geva et al., 2021 ) . Many studies have aimed to use these techniques to reverse-engineer specific behaviors of pre-trained LMs ( Wang et al., 2022 ; Hanna et al., 2024a ; Gould et al., 2024 ; Hou et al., 2023 ) . We leverage MI techniques to reverse-engineer the arithmetic mechanisms implemented by pre-trained LLMs and explain them at a single-neuron resolution.

##### Memorization and generalization in LLMs.

Whether models memorize training data or generalize to unseen data has been extensively studied in deep learning (e.g., Zhang et al., 2021 ) and specifically in LLMs ( Tänzer et al., 2022 ; Carlini et al., 2023 ; Antoniades et al., 2024 ) , but not many studies have observed this question through the lens of model internals. Among those that do, Bansal et al. (2022) attempt to predict this trade-off by observing the diversity of internal activations; Dankers & Titov (2024) show memorization in language classification tasks is not local to specific layers, and Varma et al. (2023) explain grokking using memorizing and generalizing circuits. We use this lens to observe how model internals operate in arithmetic reasoning—a task that could theoretically be solved either through extensive memorization or by learning a robust algorithm. Concurrent work ( jylin et al., 2024 ) has shown that a LM trained to predict legal board game moves ( Li et al., 2022 ) does so by implementing many heuristics. While heuristics would suffice to robustly predict legal moves in a board game setting, we find that the extent to which LLMs rely on heuristics is greater than prior work suggests: sets of heuristics are used to accomplish even generic tasks like arithmetic, where no heuristic is likely to generalize to all possible results.

##### Arithmetic reasoning interpretability.

Recent studies on how LMs process arithmetic prompts ( Stolfo et al., 2023 ; Zhang et al., 2024 ) reveal the general structure of arithmetic circuits, but do not fully explain how they combine operand information to produce correct answers. Our research bridges this gap by revealing the mechanism used for promoting the correct answer. Some studies show the emergence of mathematical algorithms for modular addition ( Nanda et al., 2023 ; Zhong et al., 2024 ; Ding et al., 2024 ) and binary arithmetic ( Maltoni & Ferrara, 2023 ) in simple, specialized toy LMs, but it is unclear if these findings extend to larger, general-purpose LMs or other operators. In pre-trained LLMs, Zhou et al. (2024) found that Fourier space features are used for addition. However, we claim this is only a partial view, as many additional types of features and heuristics relying on these features are involved in calculating answers across arithmetic operations. In this work, we give a wide view of these heuristics and how they combine to generate arithmetic answers.

## 7 Conclusions

Do LLMs rely on a robust algorithm or on memorization to solve arithmetic tasks? Our analysis suggests that the mechanism behind the arithmetic abilities of LLMs is somewhere in the middle: LLMs implement a bag of heuristics —a combination of many memorized rules—to perform arithmetic reasoning. To reach this conclusion, we performed a set of causal analysis experiments to locate a circuit, i.e., a subset of model components, responsible for arithmetic calculations. We examined the circuit at the level of individual neurons and pinpointed the arithmetic calculations to a sparse set of MLP neurons. We showed that each neuron acts as a memorized heuristic, activating for a specific pattern of inputs, and that the combination of many such neurons is required to correctly answer the prompts. In addition, we found that this mechanism gradually evolves over the course of training, emerging steadily rather than appearing abruptly or replacing other mechanisms.

Our results, showing LLMs’ reliance on the bag of heuristics, suggest that improving LLMs’ mathematical abilities may require fundamental changes to training and architectures, rather than post-hoc techniques like activation steering ( Subramani et al., 2022 ; Turner et al., 2023 ) . Additionally, the evolution of this mechanism across training indicates that models learn these heuristics early and reinforce them over time, potentially overfitting to early simple strategies; it is unclear if regularization can improve this, and this is a possible avenue for future research.

## 8 Limitations and discussion

Interpretability work is often fundamentally limited by human biases. As researchers, we often impose human abstractions onto models, whereas the goal of interpretability is to understand the abstractions that models learn and apply in a way that we can understand. Our work is also subject to this limitation, namely with respect to the definition of heuristic types: We define heuristic abstractions based on our human-identifiable definitions. A possible improvement would be to develop a method to identify these abstractions without human bias. Another important detail is that our analysis focuses on LLMs that combine digits in tokenization. That is, every token can contain more than one digit. The robust algorithms used by humans depend on our ability to separate larger numbers to single digits. Thus, a similar analysis might lead to different conclusions for models that perform single-digit tokenization.

#### Acknowledgments

We are grateful to Dana Arad and Alessandro Stolfo for providing feedback for this work. This research was supported by the Israel Science Foundation (grant No. 448/20), an Azrieli Foundation Early Career Faculty Fellowship, and an AI Alignment grant from Open Philanthropy. AM is supported by a postdoctoral fellowship under the Zuckerman STEM Leadership Program. AR is supported by a postdoctoral fellowship under the Azrieli International Postdoctoral Fellowship Program and the Ali Kaufman Postdoctoral Fellowship. This research was funded by the European Union (ERC, Control-LM, 101165402). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.

## Appendix A Llama3-8B circuit evaluation results

In Figure 11 , we present an analysis of the faithfulness of the identified circuit in Llama3-8B ( Section 2.1 ) as a function of the number of attention heads within the circuit. The circuit includes all MLPs (the neuron-level analysis in Section 3.1 does not apply in this context). Our goal is to find a minimal subset of the model with the fewest attention heads possible while maintaining high faithfulness. We assess faithfulness independently for each operator to obtain a more nuanced understanding of the necessary attention heads. For addition, subtraction, and division operations, we observe that 6 6 heads suffice to attain high faithfulness (97% on average). In contrast, in multiplication, 20 20 attention heads are required to achieve a faithfulness score exceeding 90%. The faithfulness of the full arithmetic circuit for each operator, corresponding to the Pareto-optimal number of heads (in terms of achieving high faithfulness with as few heads as possible), is documented in Table 1 . We explore the attention patterns of these attention heads in the next section.

## Appendix B Llama3-8B circuit additional components

To provide a more comprehensive understanding of the arithmetic circuit in Llama3-8B, we analyze the additional components that compose it, namely the first MLP layer (MLP0) and the high-effect attention heads.

### B.1 MLP0

To analyze the role of MLP0 in the arithmetic circuit, we first test if—similarly to the middle- and late-layer MLPs—only a sparse set of neurons within the MLP is required. We measure the intervention effect of each neuron, averaged across the operands and operator positions ( p ∈ [ 1 , 2 , 3 ] p\in[1,2,3] ). The results, shown in Figure 12(a) , reveal that few MLP0 neurons have a high effect, similar to the middle- and late-layer MLPs. To verify these neurons are sufficient for arithmetic calculations, we repeat the experiment from Section 3.1 . Specifically, we measure the faithfulness of the circuit—consisting of the top 1% of neurons in each of the middle and late layers ( l ∈ [ 16 , 32 ] l\in[16,32] ) as well as a varying number of neurons in MLP0. The results, shown in Figure 12(b) , reveal that also in the first MLP layer, as little as 1% of neurons is sufficient for the circuit to achieve high faithfulness, similarly to the middle- and late-layer MLPs.

Because Llama3-8B applies a positional embedding only at attention layers, the activation of MLP0 is not affected by the position of any token. Additionally, due to a lack of attention heads that move information between the operand and operator positions before MLP0, we can analyze its effect directly on single tokens. Thus, we view MLP0 as an “effective embedding” ( McDougall et al., 2023 ) , and hypothesize that the role of each MLP0 neuron is to incorporate additional numerical information into each token embedding. To verify this, we pass a list of numerical tokens, each representing an operand in our analyzed operand range ( t ∈ [ 0,300 ] t\in[0,300] ), through the model. We measure the activation of each high-effect neuron for each token. As exemplified in Figure 13 , the activations of these high-effect neurons correspond to varied numerical features. For example, Neuron 6206 6206 ( Figure 13(a) activates for numbers near 170 170 or 17 17 ; Neuron 7101 7101 activates for numbers greater than 100 100 ; Neuron 8969 8969 activates for numbers that are congruent to 8 ( mod 10 ) 8\pmod{10} . Overall, the set of patterns identified by these neurons can be used by the middle- and late-layer heuristic neurons to perform their more complex functionalities.

### B.2 Attention head patterns

We present the attention patterns of the attention heads that are contained in the arithmetic circuit. In Section 2.1 , for simplicity, we consider the arithmetic circuit as a single circuit for all arithmetic operators, containing all attention heads used across the four operators. We observe that to achieve high faithfulness of the circuit with a minimal number of heads ( Appendix A ), some general arithmetic heads, that significantly improve faithfulness across all operators, are required. Other heads, while contributing to increased faithfulness, are operator-specific. They exhibit varying levels of importance for different operators. A full description of the attention heads used in the arithmetic circuit for each operator is provided in Table 2 .

To better understand the role of the general arithmetic heads (L2H2, L15H13, L16H21), we compute their attention patterns, averaging them across our prompt dataset ( Section 2.1 ). These patterns ( Figure 14 ) reveal a clear signal: each of the three heads attends to a single input token, copying the representation from that position and projecting it to the last position. Specifically, L16H21 attends to the first operand, L2H2 attends to the operator and L15H13 attends to the second operand. This implies the role of each such head is to move the representation from that position, which includes to the last position, where it is further processed by the bag of heuristics implemented in the middle- and late-layer MLPs. To confirm that the information copied from each position to the last position consists solely of data from the token at that position, we conduct an ablation study. For each general arithmetic head, we zero out all preceding attention patterns that move information to its attended position (e.g., for L15H13 we zero out all patterns that move information to the o ​ p 2 op_{2} position), preventing any influence from previous positions. We observe this does not affect the circuit’s performance, indicating that the representation copied by each general arithmetic head to the final position contains information only regarding the token at its original position.

## Appendix C Multi-layer perceptron implementation details

Across the models we analyze, different implementations exist for the MLP layer in a transformer block. More specifically, Pythia-6.9B and GPT-J use simple MLP layers, consisting of two matrix multiplications. For a layer l l , it can be described as: 𝐡 p ​ o ​ s ​ t l = σ ⁡ ( 𝐡 i ​ n l ​ 𝐖 i ​ n l ⊤ ) \mathbf{h}_{post}^{l}=\sigma\left(\mathbf{h}_{in}^{l}\mathbf{W}_{in}^{{l}^{\top}}\right) (4) 𝐡 o ​ u ​ t l = 𝐡 p ​ o ​ s ​ t l ​ 𝐖 o ​ u ​ t l \mathbf{h}_{out}^{l}=\mathbf{h}_{post}^{l}\mathbf{W}_{out}^{l} (5)

where 𝐡 i ​ n l , 𝐡 o ​ u ​ t l ∈ ℝ d \mathbf{h}_{in}^{l},\mathbf{h}_{out}^{l}\in\mathbb{R}^{d} are the input and output representations of the MLP at layer l l , respectively. 𝐡 p ​ o ​ s ​ t l ∈ ℝ d m ​ l ​ p \mathbf{h}_{post}^{l}\in\mathbb{R}^{d_{mlp}} is the post-activation vector, 𝐖 i ​ n l , 𝐖 o ​ u ​ t l ∈ ℝ d m ​ l ​ p × d \mathbf{W}_{in}^{l},\mathbf{W}_{out}^{l}\in\mathbb{R}^{d_{mlp}\times d} are parameter matrices, and σ \sigma is a non-linearity function. Llama3-8B and Llama3-70B use a Gated MLP layer ( Liu et al., 2021 ) , described in the following two equations for layer l l : 𝐡 p ​ o ​ s ​ t l = σ ⁡ ( 𝐡 i ​ n l ​ 𝐖 g ​ a ​ t ​ e l ⊤ ) ∘ ( 𝐡 i ​ n l ​ 𝐖 i ​ n l ⊤ ) \mathbf{h}_{post}^{l}=\sigma(\mathbf{h}_{in}^{l}\mathbf{W}_{gate}^{{l}^{\top}})\circ(\mathbf{h}_{in}^{l}\mathbf{W}_{in}^{{l}^{\top}}) (6) 𝐡 o ​ u ​ t l = 𝐡 p ​ o ​ s ​ t l ​ 𝐖 o ​ u ​ t l \mathbf{h}_{out}^{l}=\mathbf{h}_{post}^{l}\mathbf{W}_{out}^{l} (7) where 𝐖 g ​ a ​ t ​ e l ∈ ℝ d m ​ l ​ p × d \mathbf{W}_{gate}^{l}\in\mathbb{R}^{d_{mlp}\times d} is an additional parameter matrix and ∘ \circ is Hadamard product. Biases are omitted in both presentations.