Language Models Use Trigonometry to Do Addition — part 4/4

Cleaned-text code points [47174, 51000). Bibliography omitted here; source.md includes it.

We test the first hypothesis by analyzing the distribution of GPT-J errors. If carrying was the problem, we would expect that when the model is off by − 10 -10 , the units digits of a a and b b add up to 10 or more. Using a Chi-squared test with a threshold of α = 0.05 \alpha=0.05 , we see that the units digit of a a and b b summing to more than 10 10 is not more likely for when the model’s error is − 10 -10 than otherwise (Fig. 38 ). This falsifies our first hypothesis. Thus, we turn to understanding how the a + b a+b helix is translated to model logits.

Since MLPs most contribute to direct effect, we begin investigating at the neuron level. We sort neurons by their direct effect, and take the k = 693 k=693 highest DE neurons required to achieve 80% of the total direct effect (Fig 33 ). Then, we use the technique of LogitLens to understand how each neuron’s contribution boosts and suppresses certain answers (see nostalgebraist (2020) for additional details). For the tokens [ 0,198 ] [0,198] (the answer space to a + b a+b ), we see that each top DE neuron typically boosts and suppresses tokens periodically (Fig. 39 ). Moreover, when we Fourier decompose the LogitLens of the max activating a + b a+b example for each neuron, we find that a neuron whose preactivation fit’s largest term is c T i , a + b c_{T_{i},a+b} in Eq. 3 often has LogitLens with dominant period of T i T_{i} as well (Fig. 40 ). We interpret this as neurons boosting and suppressing tokens with a similar periodicity that they read from the residual stream helix with.

Despite being periodic, the neuron LogitLens are complex and not well modeled by simple trigonometric functions. Instead, we turn to more broadly looking at the model’s final logits for each problem a + b a+b over the possible answer tokens [ 0,198 ] [0,198] . We note a similar distinct periodicity in Fig. 41 . When we Fourier decompose the logits for all problems a + b a+b , we find that the most common top period is 10 10 (Fig. 42 ). Thus, it is sensible that the most common error is ± 10 \pm 10 , since a + b − 10 a+b-10 , a + b + 10 a+b+10 are also strongly promoted by the model. To explain why − 10 -10 is a more common error than 10 10 , we fit a line of best fit through the model logits for all a + b a+b , and note that the best fit line almost always has negative slope (Fig. 42 ), indicating a preference for smaller answers. This bias towards smaller answers explains why GPT-J usually makes mistakes with larger a a and b b values (Fig. 11 , Appendix A ).

## Appendix G Tooling and Compute

We used the Python library nnsight \mathrm{nnsight} to perform intervention experiments on language models ( Fiotto-Kaufman et al., 2024 ) . All experiments were run on a single NVIDIA RTX A6000 GPU with 48GB of VRAM. With this configuration, all experiments can be reproduced in two days.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .