Language Models Use Trigonometry to Do Addition — part 3/4

Cleaned-text code points [31564, 47174). Bibliography omitted here; source.md includes it.

The mere existence of the Pizza algorithm demonstrates that even one layer transformers have a complex solution space. Thus, even if the Clock algorithm is used by LLMs, it could be one method of an ensemble. We see evidence of this in Appendix D , in that helix ⁡ ( a + b ) \mathrm{helix}(a+b) is less causally implicated for Llama3.1-8B than other models, which we hypothesize is due to its use of gated MLPs. Additionally, other models must necessarily use modified algorithms for addition because of different tokenization schemes. For example, Gemma-2-9B ( Riviere et al., 2024 ) tokenizes each digit of a number separately and must use additional algorithms to collate digit tokens. Additionally, at different scales LLMs potentially learn different algorithms, providing another reason to be skeptical that the Clock algorithm is the one and only explanation for LLM addition.

## 6 Conclusion

We find that three mid-sized LLMs represent numbers as generalized helices and manipulate them using the interpretable Clock algorithm to compute addition. While LLMs could do addition linearly, we conjecture that LLMs use the Clock algorithm to improve accuracy, analogous to humans using decimal digits (which are a generalized helix with T = [ 10,100 , … ] T=[10,100,\dots] ) for addition rather than slide rules. In Appendix E , we present preliminary results that GPT-J would be considerably less accurate on “linear addition” due to noise in its linear representations. Future work could analyze if LLMs have internal error-correcting codes for addition like the grid cells presented in Zlokapa et al. (2024) .

The use of the Clock algorithm provides striking evidence that LLMs trained on general text naturally learn to implement complex mathematical algorithms. Understanding LLM algorithms is important for safe AI and can also provide valuable insight into model errors, as shown in Appendix F . We hope that this work inspires additional investigations into LLM mathematical capabilities, especially as addition is implicit to many reasoning problems.

## Acknowledgments

We thank Josh Engels for participating in extensive conversations throughout the project. We also thank Vedang Lad, Neel Nanda, Ziming Liu, David Baek, and Eric Michaud for their helpful suggestions. This work is supported by the Rothberg Family Fund for Cognitive Science and IAIFI through NSF grant PHY-2019786.

## Appendix A Performance of all models on a + b = a+b=

We test three models, GPT-J, Pythia-6.9B, and Llama3.1-8B on the task a + b = a+b= . At first, we attempted to prompt each model with just a + b = a+b= , but we achieved significantly better results by including additional instructions in the prompt. After non-exhaustive testing, we used the prompts listed in Table 2 to test each model for all 10000 addition prompts (for a , b ∈ [ 0 , 99 ] a,b\in[0,99] ). We plot a heatmap of the accuracy of the model by a a and b b in Fig. 11 . All three models are able to competently complete the task, with Llama3.1-8B achieving an impressive 98% accuracy. However, in the main paper we focus on analyzing GPT-J because it employs simple MLPs that are easier to interpret. We note that all three models struggle with problems with larger values of a a and b b .

## Appendix B Additional Results on the Structure of Numbers

Our Fourier decomposition results in Section 4.1 are sensitive to the number of a a values analyzed. In particular, we find that the T = 2 T=2 Fourier component is not identified when analyzing h 361 0 h^{0}_{361} , but is identified when analyzing h 360 0 h^{0}_{360} (Fig. 12 ). While we consider this sensitivity to sample size to be a limitation of our Fourier analysis, we note that the Fourier analysis is itself preliminary. We find that the T = 2 T=2 Fourier feature is causally relevant when fitting the residual stream in Section 4.4 . Additionally, in later sections we find that neurons often read from the helix using the T = 2 T=2 Fourier feature, indicating its use downstream (Fig. 34 ). Thus, we identify T = 2 T=2 as an important Fourier feature.

We compare the residual stream of GPT-J after layer 0 on the inputted integers a 1 , a 2 ∈ [ 0 , 99 ] a_{1},a_{2}\in[0,99] using Euclidean distance and cosine similarity in Fig. 13 . We visually note periodicity in the representations, with a striking period of 10 10 . To analyze the similarity between representations further, we calculate the Euclidean distance between a a and a + δ ​ n a+\delta n for all values of δ ​ n \delta n . In Fig. 14 , we see that representations continue to get more distant from each other for a ∈ [ 0 , 99 ] a\in[0,99] as δ ​ n \delta n grows, albeit sublinearly. This provides evidence that LLMs represent numbers with more than just periodic features. When a a is restricted to a ∈ [ 0 , 9 ] a\in[0,9] , we observe a linear relationship in δ ​ n \delta n , implying some local linearity. The first principal component of the numbers a ∈ [ 0,360 ] a\in[0,360] (shown in Fig. 15 ) is also linear with a discontinuity at a = 100 a=100 . Thus, our focus on two digit addition is justified, as three-digit integers seem to be represented in a different space.

## Appendix C Additional Helix Fitting Results

### C.1 Helix Properties

For the input of layer 0 of GPT-J, we plot the magnitude of the helical fit’s Fourier features. We do so by taking the magnitude of columns of C C in Eq. 2 . We find that these features roughly increase in magnitude as period increases, which matches the ordering in the Fourier decomposition presented in Fig. 2 .

Additionally, we visualize the cosine similarity matrix between columns of C C , which represents the similarity between helix components. In Fig. 17 , we observe that components are mostly orthogonal, as expected. A notable exception is the similarity between the T = 100 T=100 sin \sin component and the linear component.

To ensure that the helix represents a true feature manifold, we design a continuity experiment inspired by Olah & Jermyn (2024) . We first fit all a a that do not end with 3 with a T = [ 100 ] T=[100] helix. Then, we project a = 3 , 13 , … , 93 a=3,13,\dots,93 into that helical space in Fig. 18 . We find that each point is projected roughly where we expect it to be. For example, 93 93 is projected between 89 89 and 95 95 . We take this as evidence that our helices represent a true nonlinear manifold.

### C.2 Additional Causal Experiments for Helix Fits

We first replicate our helix fitting activation patching results on Pythia-6.9B and Llama3.1-8B in Fig. 19 .

To ensure the helix fits are not overfitting, we use a train-test split. We train the helix with 80% of a a values and patch using the other 20% of a a values (left of Fig. 20 ). We observe that the helix and circular fits still outperform the PCA baseline. We also randomize the order of a a and find that the randomized helix is not causally relevant (middle of Fig. 20 ), suggesting that the helix functional form is not naturally over expressive. Finally, we demonstrate that our results hold when fitting the b b token on a + b a+b with helix ⁡ ( b ) \mathrm{helix}(b) (right of Fig. 20 ). Note that when activation patching fits on the b b token, we use clean/corrupted prompt pairs of the form ( a + b ′ a+b^{\prime} , a + b a+b ), in contrast to the ( a ′ + b a^{\prime}+b , a + b a+b ) pairs we used for the a a token.

We perform an ablation experiment by ablating the columns of C † C^{\dagger} from each h a l h_{a}^{l} . In Fig. 21 , we see that ablating the helix dimensions from the residual stream like this affects performance about as much as ablating the entire layer, providing additional causal evidence that the helix is necessary for addition. However, when we attempt to fit a a with helix ⁡ ( a ) \mathrm{helix}(a) for other tasks in Fig. 22 , we find that while the fit is effective, it sometimes underperforms PCA baselines. This suggests that while the helix is sufficient for addition, additional structure is required to capture the entirety of numerical representations. For a description of the prompts used and accuracy of GPT-J on these other tasks, see Table 3 .

## Appendix D Additional Clock algorithm evidence

We show that helix ⁡ ( a + b ) \mathrm{helix}(a+b) fits last token hidden states for Pythia-6.9B and Llama3.1 8B in Fig. 23 . Notably, the results for Llama3.1-8B are less significant than those for GPT-J and Pythia-6.9B. This is surprising, since the helical fit on the a a token is causal for Llama3.1-8B in Fig. 19 , and is a sign that Llama3.1-8B potentially uses additional algorithms to compute a + b a+b . We hypothesize that this might be due to Llama3.1-8B using gated MLPs, which could lead to the emergence of algorithms not present in GPT-J and Pythia-6.9B, which use simple MLPs. Nikankin et al. (2024) ’s analysis of Llama3-8B’s top neurons in addition problems identifies neurons with activation patterns unlike those we identified in GPT-J. Due to this evidence, along with the importance of MLPs in the addition circuit, we consider it likely that Llama3.1-8B implements modified algorithms, but we do not investigate further.

### D.1 Attention Heads

In Fig. 24 , we use activation patching to show that a sparse set of attention heads are influential for addition. In Fig. 25 , we find that patching in k = 20 k=20 heads at once is sufficient to restore more than 80% of the total effect of patching all k = 448 k=448 heads. In Fig. 26 , we also find that all attention heads in GPT-J are well modeled using helix ⁡ ( a , b , a + b ) \mathrm{helix}(a,b,a+b) . We judge this by the fraction of a head’s total effect recoverable when patching in a helical fit.

We categorize heads as a , b a,b , a + b a+b , and mixed heads using a confidence score (detailed in Section 5.2 ). To make our categorization useful, we aim to categorize as few heads as mixed as possible. We find that using m = 4 m=4 mixed heads is sufficient to achieve almost 80% of the effect of patching the actual outputs of the k = 20 k=20 heads (Fig. 27 ), although using m = 0 m=0 mixed heads still achieves 70% of the effect. In Fig. 28 , we analyze the properties of each head type. a + b a+b heads tend to attend to the last token and occur in layers 19 onwards. a , b a,b heads primarily attend to the a a and b b tokens and occur prior to layer 18. Mixed heads attend to the a , b a,b , and last tokens, and occur in layers 15-18.

To understand what each head type reads and writes to, we use a modification of the path patching technique we have discussed so far. Specifically, we view mixed and a , b a,b heads as “sender” nodes, and view the total effect of each downstream component if only the direct path between the sender node and the component is patched in (not mediated by any other attention heads or MLPs). In Fig. 29 , we find that both a , b a,b and mixed heads generally impact downstream MLPs most. Similarly, we consider mixed and a + b a+b heads as “receiver” nodes, and patch in the path between all upstream components and the receiver node to determine what components each head relies on to achieve its causal effect. We find that a + b a+b heads rely predominantly on upstream MLPs, while mixed heads use both a , b a,b heads and upstream MLPs. This indicates that mixed heads may have some role in creating helix ⁡ ( a + b ) \mathrm{helix}(a+b) .

### D.2 MLPs and Neurons

In Fig. 30 , we see that patching k = 11 k=11 MLPs achieves 95% of the effect of patching all MLPs. We consider these MLPs to be circuit MLPs. Zooming in at the neuron level, we find that roughly 1% of neurons are required to achieve an 80% success rate on prompts while mean ablating all other neurons (Fig. 31 ). Note the use of accuracy over logit difference as a metric in this case. Fig. 31 shows that ablating some neurons actually helps performance as measured by logit difference, while hurting accuracy. To account for this seemingly contradictory result, we hypothesize that ablating some neurons asymmetrically boosts the answer token across prompts, such that some prompts are boosted significantly while other prompts are not affected. We do not investigate this further as it is not a major part of our argument and instead use an accuracy threshold.

When plotting the distribution of top neurons across layers in Fig. 32 , we find that almost 75% of top neurons are located in the k = 11 k=11 circuit MLPs we have identified. We then path patch each of these neurons to calculate their direct effect. In Fig. 33 , we see that roughly 700 neurons are required to achieve 80% of the direct effect of patching in all k = 4587 k=4587 top neurons. 84% of the top DE neurons occur after layer 18, which corresponds with our claim that MLPs 19-27 primarily write the correct answer to logits.

When we Fourier decompose the k = 4587 k=4587 top neurons’ preactivations with respect to the value of a + b a+b in Fig. 34 , we see spikes at periods T = [ 2 , 5 , 10 , 100 ] T=[2,5,10,100] . These are the exact periods of our helix parameterization. To leverage this intuition, we fit the neuron preactivation patterns using the helix inspired functional form detailed in Eq. 2 . We use a stochastic gradient descent optimizer with lr = 1 ​ e − 1 , epochs = 2500 \mathrm{lr}=1e-1,\mathrm{epochs}=2500 and a cosine annealing learning rate scheduler to minimize the mean squared error of the fit. In Fig. 35 , we see that more important neurons with larger total effect are fit better with this functional form, as measured by normalized root mean square error (NRMSE).

## Appendix E Why Use the Clock Algorithm at All?

We conjecture that LLMs use the Clock algorithm as a form of robust, error correcting code. If LLMs used a linear representation of numbers to do addition, that representation would have to be extremely precise to be effective.

To preliminarily test this conjecture, we take the first 50 PCA dimensions of the number representations for a ∈ [ 0 , 99 ] a\in[0,99] in GPT-J after layer 0 and fit a line ℓ \ell to it. The resulting line has an R 2 R^{2} of 0.997 0.997 , indicating a very good fit. We consider all problems a 1 + a 2 a_{1}+a_{2} . We do addition on this line by taking ℓ ⁡ ( a 1 ) + ℓ ⁡ ( a 2 ) \ell(a_{1})+\ell(a_{2}) . If ℓ ⁡ ( a 1 ) + ℓ ⁡ ( a 2 ) \ell(a_{1})+\ell(a_{2}) is closest to ℓ ⁡ ( a 1 + a 2 ) \ell(a_{1}+a_{2}) , we consider the addition problem successful.

We then take the percentage of successful addition problems where the answer a 1 + a 2 a_{1}+a_{2} is less than some threshold α \alpha , and compare the accuracy as a function of α \alpha for GPT-J and linear addition. Surprisingly, we find that for α = 100 \alpha=100 , linear addition has an accuracy of less than 20%, while GPT-J has an accuracy of more than 80% (Fig. 36 ).

Thus, even with very precise linear representations, doing linear addition leads to errors. We interpret LLMs use of modular circles for addition as a built-in redundancy to avoid errors from their imperfect representations.

## Appendix F Investigating Model Errors

Given that GPT-J implements an algorithm to compute addition rather than relying on memorization, why does it still make mistakes? For problems where GPT-J answers incorrectly with a number, we see that it is most often off by − 10 -10 (45.7%) and 10 10 (27.9%), cumulatively making up over 70% of incorrect numeric answers (Fig 37 ). We offer two hypotheses for the source of these errors: 1) GPT-J is failing to “carry” correctly when creating the a + b a+b helix or 2) reading from the a + b a+b helix to answer logits is flawed.