Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics β part 4/5
Cleaned-text code points [45629, 61621). Bibliography omitted here; source.md includes it.
While the key-value view that was used in Section 3 was devised for simple MLPs ( Geva et al., 2021 ) , it can be applied to the Gated MLP mechanism of Llama3 models as well. For the n th n^{\text{th}} neuron, we treat the element π‘ p β o β s β t l , n \mathbf{h}_{post}^{l,n} of π‘ p β o β s β t l \mathbf{h}_{post}^{l} as the activation of the n th n^{\text{th}} key vector. Each such activation is multiplied with π― o β u β t l , n \mathbf{v}_{out}^{l,n} , a row vector of π o β u β t l \mathbf{W}_{out}^{l} , resulting in the same multiplication as performed in simple MLPs.
## Appendix D Neuron intersection between operators
Figure 15 presents the IoU (intersection over union) of causally important neuron sets in the arithmetic circuits, identified separately for each operator. Each circuit consists of the top- 200 200 neurons from each layer l β [ 16 , 32 ] l\in[16,32] , ranked based on their mean effect ( Section 3.1 ), totaling 3,200 3,200 neurons. The average IoU between any two operator-specific circuits is relatively low (54%), indicating that a substantial proportion of key neurons are unique to each operator. Consequently, we define a distinct circuit for each arithmetic operator at the neuron level ( Section 3.1 ).
## Appendix E Heuristic types descriptions
In this section, we present the various heuristic types that were manually identified.
Most heuristic types are defined with parameters. A neuron is defined as a heuristic H H , if it matches the heuristic definition given a parameter value. For example, the neuron in Figure 1 b is a range heuristic with a range parameter [ 150,180 ] [150,180] . In Algorithm 1 , we describe the process of matching between each important neuron and each heuristic H H . The parameters are sub-sampled evenly to cover most observed possibilities, which causes some inaccuracies (for example, a neuron that fires when o β p 2 β [ 105,205 ] op_{2}\in[105,205] will be classified as a range heuristic with o β p 2 β [ 100,200 ] op_{2}\in[100,200] due to our parameter choices).
Several types of heuristics (Range, Modulo, Pattern) can apply either to an operand in a prompt or to the promptβs result. For example, an βoperand rangeβ heuristic is triggered when o β p 1 op_{1} or o β p 2 op_{2} falls within a specific numerical range, while a βresult rangeβ heuristic activates when the promptβs ground truth result is within a defined range.
The full list of heuristic types is as follows:
β’ Range heuristic: A neuron that activates when a value (either an operand or result) falls within a specified range [ a , b ] [a,b] . The parameters a , b β β a,b\in\mathbb{N} are chosen such that the range length l β e β n = ( b β a ) β [ 10 , 30 , 50 , 100 ] len=(b-a)\in[10,30,50,100] for addition, subtraction, and multiplication, and l β e β n = ( b β a ) β [ 2 , 10 , 100 ] len=(b-a)\in[2,10,100] for division (due to a different distribution of potential results for integer division). The range start, a a , is defined as a β x n : x n = x 0 + n β
max β‘ ( β len 3 β , 10 ) , n β β , x 0 = 0 a\in{x_{n}:x_{n}=x_{0}+n\cdot\max(\lfloor\frac{\text{len}}{3}\rfloor,10),n\in\mathbb{N},x_{0}=0} , provided a a remains below the maximum prompt result. This definition of range start and length allows for ranges to intersect or overlap.
β’ Modulo heuristic: A neuron that activates when a value (either operand or result) is congruent to m m modulo n n . The parameters are n β { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 11 , 13 , 15 } n\in\{2,3,4,5,6,7,8,9,11,13,15\} and m β [ 0 , n β 1 ] m\in[0,n-1] . We exclude n = 10 n=10 or n = 100 n=100 as these are specific cases of the broader βpatternβ heuristics, discussed next. Additionally, n = 12 , 14 n=12,14 , and n β₯ 16 n\geq 16 are excluded, as no neurons were classified under these parameter values.
β’ Pattern heuristic: A neuron that activates when a value (either operand or result) matches a specific regular expression p p . The parameter p p is a 3-digit regular expression, potentially padded with leading zeros. For example, a neuron classified under the βoperand pattern 1.2 1.2 β heuristic activates when one of the operands has 1 1 in the hundreds place and 2 2 in the units place.
β’ Identical operands heuristic: A neuron that activates when both operands are equal ( o β p 1 op_{1} == o β p 2 op_{2} ). The tokens embedded in the value vector change depending on the operator for which the neuron activates. For instance, in subtraction, such neurons have been observed to promote the β0β token (the result of subtracting a number from itself).
β’ Multi-result heuristic: A neuron that promotes a set π \mathbf{S} of several unrelated results (For example, a neuron that promote the results [ 4 , 5 , 7 ] [4,5,7] ). This heuristic type is defined exclusively for division, where the parameter S S can consist of several values, where | π | β [ 2 , 4 ] \left|\mathbf{S}\right|\in[2,4] . This range was chosen based on observations of activation patterns. The values included in π \mathbf{S} are chosen according to the result tokens that the neuron contributes the most to: we determine π \mathbf{S} by examining the top prompts for which the neuron promotes their answer. From these top answers, we identify a minimal set of 2 2 to 4 4 distinct result values. This set of result values is chosen if it accounts for more than a threshold percentage of the results. This approach captures the most significant result values that the neuron consistently promotes in division operations. If no such set of different results exists, a neuron is not classified as this heuristic.
## Appendix F Heuristic Neuron Classification Algorithm
We present the full algorithm used to match between each neuron and each heuristic type in Algorithm 1 (as exemplified in Figure 6 ). The goal of this process is to check if a neuron implements a specific heuristic. We repeat this process for each pair of neuron n n in layer l l and heuristic H H . This method allows a single neuron to be classified as several heuristics. The matching of each neuron to heuristics is done separately for each arithmetic operator.
Inputs: Heuristic H H , Layer l β [ 1 , l m β a β x ] l\in[1,l_{max}] , Neuron Index n β [ 1 , n m β a β x ] n\in[1,n_{max}] , Operator o β { + , β , Γ , Γ· } o\in\{+,-,\times,\div\} , Threshold t β [ 0 , 1.0 ] t\in[0,1.0]
The algorithm first calculates the neuronβs activations h p β o β s β t l , n h_{post}^{l,n} for all prompts, yielding a 2D activation pattern ( 1 ), as those seen in Appendix J . When checking if a neuron implements a direct heuristic (heuristics that directly promote relevant result tokens ( Section 3.2 )), we also need to take into consideration the tokens that are embedded in the neuronβs value vector π― o β u β t l , n \mathbf{v}_{out}^{l,n} . This is not done in indirect heuristics because we do not expect relevant result tokens to be promoted in such heuristics. Thus, we extract the logits of all numerical tokens (β 0 0 β, β 1 1 β, β 2 2 β, β¦ β 999 999 β) from the value vector π― o β u β t l , n \mathbf{v}_{out}^{l,n} ( 3 ), using Logit Lens ( nostalgebraist, 2020 ) . To match the logit of each result token to the prompts that result in it (e.g., match the logit of β 151 151 β with the prompts β 1 + 150 = 1+150= β,β 2 + 149 = 2+149= β, etc.), we convert the logits to a 2D pattern π \mathbf{L} , where π i β j \mathbf{L}_{ij} is the logit of applying the operator on i i and j j ( 4 ). Multiplying the 2D activation pattern and 2D logit pattern element-wise ( 5 ) yields the neuronβs effective logit contribution to the correct answer of each prompt, i.e., the value at each index i , j i,j marks how much the neuron promotes the answer token to the result of applying the operator on i i and j j . We then create two lists of prompts. One list contains prompts associated with the heuristic ( 7 ), to be used as the βground truthβ. We mark the number of associated prompts as k k ( 8 ), and find a second list, containing the top- k k prompts that are most contributed to by the neuron ( 9 ). We measure if the intersection of these two lists, normalized by k k , is larger than the threshold t = 0.6 t=0.6 ( 10 β 12 ) to decide if the neuron implements the heuristic.
## Appendix G Additional implementation details
All experimental procedures were executed using the TransformerLens library Nanda & Bloom (2022) . Experiments involving Llama3-8B, GPT-J, and Pythia-6.9B were conducted on a single Nvidia-L40 GPU with 48GB of GPU memory. Llama3-70B was loaded in 16-bit precision across four Nvidia A100 GPUs, each with 80GB of GPU memory.
##### Arithmetic prompts.
Unless otherwise specified, our experiments utilize prompts of the form o β p 1 β o β p 2 op_{1}\circ op_{2} =β, where o β p 1 , o β p 2 β [ 0,300 ] op_{1},op_{2}\in[0,300] and β β { + , β , Γ , Γ· } \circ\in\{+,-,\times,\div\} . The division operator ( Γ· \div ) is interpreted as integer division, considering only the integer part of the result (e.g., the ground truth result for 45 Γ· 4 = 45\div 4= β is 11 11 ). For each prompt, we compute the ground-truth result and exclude prompts whose results are not tokenized to a single token. This filtering process eliminates prompts with negative results and prompts for which the correct answer exceeds the single-token limit (The highest token from which the model splits numbers into two tokens). The single-token limit s s for Llama3-8B and Llama3-70B is s l β l β a β m β a β 3 = 1,000 s_{llama3}=1,000 , while for GPT-J and Pythia-6.9B, it is s g β p β t β j = s p β y β t β h β i β a = 520 s_{gptj}=s_{pythia}=520 .
##### Circuit discovery via activation patching.
In Section 2.1 , we quantify the effect of patching on probabilities. We find that applying our effect measure ( Equation 1 ) on logits instead of probabilities, does not alter the result significantlyβthe most effective components remain consistent, differing only in the scale of the effect.
##### Circuit faithfulness evaluation.
Thus far, to measure the faithfulness of the circuit, we have performed mean ablation on all non-circuit components. For that, we calculate the mean activation output of each component, across all arithmetic prompts. In this context, we do not filter correct prompts exclusively, but instead use all prompts of the form β o β p 1 β o β p 2 op_{1}\circ op_{2} =β, for o β p 1 , o β p 2 β [ 0,300 ] op_{1},op_{2}\in[0,300] . This leads to an equal amount of prompts per operator, thus maintaining the mean activation balanced across the different operators. When evaluating a circuit with partial MLP layers ( Section 3.1 ), we mean ablate lower-effect neurons in middle- and late-layer MLPs only, starting and ending in the earliest and latest layers from which the answer is extractable via linear probe ( Section 2.2 ). These layer ranges are [ 16 , 32 ] , [ 39 , 80 ] , [ 14 , 32 ] , [ 17 , 28 ] [16,32],[39,80],[14,32],[17,28] In Llama3-8B, Llama3-70B, Pythia-6.9B, and GPT-J, respectively.
##### Linear probing for correct answers.
We define a linear probe f l , p : β β β s f_{l,p}:\mathbb{R}\to\mathbb{R}^{s} , where s s is the single-token limit, to predict the correct answer token from the output representation at layer l l and position p p . We compute these outputs by using all correctly completed prompts, with 80% used for training the classifier and 20% for evaluation. Each probing classifier is implemented as a one-layer fully connected model. We train it using the Adam optimizer ( Kingma & Ba, 2015 ) with a learning rate of 0.0003 0.0003 and a batch size of 32 32 , optimizing a cross-entropy loss function.
##### Heuristic classification.
In applying the heuristic classification algorithm ( Section 4.1 ), we use all activations from the last position p = 4 p=4 and employ a classification score threshold t = 0.6 t=0.6 . Due to a lack of ground truth data for neuron-to-heuristic matches, the threshold is chosen to achieve a Pareto-optimal balance of false positives, that occur more for a lower threshold, and false negatives, that occur more in a higher threshold. The amount of false classifications is estimated manually.
## Appendix H Model accuracies on arithmetic prompts
Table 3 presents the accuracy of the analyzed models on arithmetic prompts. The accuracy is evaluated across prompts where the result is represented by a single token, with both operands constrained to the interval [ 0,300 ] [0,300] , where 300 300 represents the maximum value of operands in the analyzed prompts ( Section 2.1 ), chosen for efficiency. It is noteworthy that the high accuracy rates observed in division operations for the smaller-scale models (GPT-J, Pythia-6.9B) can be attributed to the non-uniform distribution of answers in integer division, i.e. - half of the legal prompts result in the token β 0 0 β.
## Appendix I Results on additional models
To demonstrate the generalizability of our findings across differently-trained LLMs, we conduct our primary experiments on Llama3-70B ( Dubey et al., 2024 ) , Pythia-6.9B ( Biderman et al., 2023 ) , and GPT-J ( Wang & Komatsuzaki, 2021 ) . We replicate the experiments for circuit discovery and evaluation ( Section 2.1 ), linear probing for answer token embeddings ( Section 2.2 ), top-k neuron faithfulness analysis ( Section 3.1 ), and heuristic analysis ( Section 4.2 ). The results obtained are similar to those of Llama3-8B across all three LLMs:
β’ The circuit comprises a sparse subset of attention heads that project operand and operator information to the final position, along with all MLP layers. Early-layer MLPs process information at the operand and operator positions, while middle- and late-layer MLPs process the combined information at the last token. These findings are illustrated for Llama3-70B ( Figure 16(a) ), Pythia-6.9B ( Figure 17(a) ), and GPT-J ( Figure 18(a) ).
β’ Linear probing results indicate that the correct answer can only be extracted with high accuracy in the final position, following the processing initiated by middle-layer MLPs. These observations are presented for Llama3-70B ( Figure 16(b) ), Pythia-6.9B ( Figure 17(b) ), and GPT-J ( Figure 18(b) ).
β’ The circuit requires a sparse subset of middle- and late-layer MLP neurons to achieve maximal faithfulness. These results are depicted for Llama3-70B ( Figure 16(c) ), Pythia-6.9B ( Figure 17(c) ), and GPT-J ( Figure 18(c) ).
β’ The heuristic neurons in the middle- and late-layer MLPs are the model components that write the correct answer to the model output. This is shown by repeating the prompt-guided knockout experiment ( Section 4.2 ). The ablation of specific heuristic neurons associated with prompts results in a more significant reduction in model accuracy compared to the ablation of a random set of neurons of equivalent size. These findings are illustrated for Llama3-70B ( Figure 8 ), Pythia-6.9B ( Figure 17(d) ), and GPT-J ( Figure 18(d) ).
The results observed in Llama3-70B exhibit the highest similarity to those reported in Llama3-8B, with a more pronounced knockout effect compared to the other two models. We hypothesize that this indicates a more sophisticated development of the bag of heuristics in Llama3-70B, potentially facilitated by its larger size and modern training methodology.
## Appendix J Additional examples for arithmetic heuristics
We present supplementary examples of heuristic neurons ( Table 4 , Table 5 , Table 6 , and Table 7 for the four arithmetic operators, respectively). Each neuron π‘ p β o β s β t l , n \mathbf{h}_{post}^{l,n} found to be important for a specific arithmetic operator ( Section 3.1 ) is categorized as one or more heuristics. For each designated operator, we randomly present four neurons. We compute and report each neuronβs activation pattern, the ten numerical tokens with the strongest embeddings in each neuronβs value vector π― o β u β t l , n \mathbf{v}_{out}^{l,n} , and provide a non-exhaustive list of the heuristics it implements.