Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls — part 2/3

Cleaned-text code points [15939, 31923). Bibliography omitted here; source.md includes it.

Attention heads Att 3 2 \textsc{Att}^{2}_{3} , Att 4 2 \textsc{Att}^{2}_{4} each attend to positions ( b 0 , b 2 , c 1 ) (b_{0},b_{2},c_{1}) and ( b 3 , “ ​ # ​ ” , c 0 ) (b_{3},\text{``}\#\text{''},c_{0}) . Inspecting what was “cached” in the first layer at those timesteps reveals the necessary partial products to compute c 2 c_{2} . For example, at timestep b 0 b_{0} , Att 1 1 \textsc{Att}^{1}_{1} , Att 2 1 \textsc{Att}^{1}_{2} attend to a 2 , b 0 a_{2},b_{0} ; at timestep b 2 b_{2} Att 1 1 \textsc{Att}^{1}_{1} attends to a 1 , b 1 a_{1},b_{1} while Att 2 1 \textsc{Att}^{1}_{2} attends to a 0 , b 2 a_{0},b_{2} ; at timestep c 0 c_{0} Att 1 1 \textsc{Att}^{1}_{1} attends to a 1 , b 0 a_{1},b_{0} , Att 2 1 \textsc{Att}^{1}_{2} attends to a 0 ​ b 1 a_{0}b_{1} . Thus the model can derive partial products, a 2 ​ b 0 , a 1 ​ b 1 , a 0 ​ b 2 , a 1 ​ b 0 , a 0 ​ b 1 a_{2}b_{0},a_{1}b_{1},a_{0}b_{2},a_{1}b_{0},a_{0}b_{1} with its attention tree. 4 4 4 Note that there may be a couple of different ways that a 0 ​ b 0 a_{0}b_{0} is derived. One possibility is to re-use a 0 , b 0 a_{0},b_{0} information that was fetched at various timesteps. Another possibility is when a 0 a_{0} is slightly attended to at Att 3 2 \textsc{Att}^{2}_{3} (difficult to see in our visuals). Note that a 0 ​ b 0 a_{0}b_{0} plays a relatively minor role in computing c 2 c_{2} compared to all other partial products.

While Figure 4 shows an example of the “attention tree” for predicting c 2 c_{2} , one can similarly reconstruct the correct trees for all digits c 0 , … , c 7 c_{0},\ldots,c_{7} using the attention patterns for all digits in Figure 10 .

In summary, for each output step c k c_{k} , the ICoT model constructs a binary-tree-like graph, spread out across timesteps, to attend to the correct pairs of tokens, allowing it to compute partial products.

## 4 Feature Geometry of ICoT

In addition to the mechanisms seen in Section 3 , we also study the geometry of features in ICoT.

### 4.1 Digit-wise Multiplications as Minkowski Sums

Note from Section 3.3 that the attention patterns are sparse, often only attending to the two digits a i , b j a_{i},b_{j} being multiplied. In such a case, the outputs of the attention head form a Minkowski sum.

Namely, consider a single head Att 1 \textsc{Att}^{1} ( i , j i,j ) at the first layer, attending to two digits a i a_{i} , b j b_{j} . Let W O ∈ ℝ d × d h ​ e ​ a ​ d , W V ∈ ℝ d h ​ e ​ a ​ d × d W_{O}\in\mathbb{R}{}^{d\times d_{head}},W_{V}\in\mathbb{R}{}^{d_{head}\times d} be the output and value weights of the attention head, E [ a i ] ∈ ℝ d E[a_{i}]\in\mathbb{R}{}^{d} the token embedding for token a i a_{i} , and A i := W O W V E [ a i ] , B j := W O W V E [ b j ] , A i , B j ∈ ℝ d A_{i}:=W_{O}W_{V}E[a_{i}],B_{j}:=W_{O}W_{V}E[b_{j}],A_{i},B_{j}\in\mathbb{R}{}^{d} .

In such a case, when the model spends α % \alpha\% of its attention on digit a i a_{i} , and thus attends to digit b j b_{j} by ( 1 − α ) % (1-\alpha)\% , the set of all possible values for the output of the attention head forms a Minkowski sum:

Att 1 ​ ( i , j ) \displaystyle\textsc{Att}^{1}(i,j) = α ​ A i + ( 1 − α ) ​ B j + ϵ \displaystyle=\alpha A_{i}+(1-\alpha)B_{j}+\epsilon (3) { Att 1 ​ ( i , j ) } i , j \displaystyle\{\textsc{Att}^{1}(i,j)\}_{i,j} ⊆ ( α ​ A ) ⊕ ( ( 1 − α ) ​ B ) ⊕ ϵ \displaystyle\subseteq(\alpha A)\oplus((1-\alpha)B)\oplus\epsilon (4) (ignoring position embeddings). See Figure 5 (a) for a visualization.

Visually, 3D PCAs can reveal nested representations. Namely, we can observe clusters, each cluster corresponding to a feature (i.e., a i a_{i} ). These clusters form a “global” geometry. When zoomed in to each cluster, we observe additional clusters for a second feature (i.e., b j b_{j} ) that form a “local” geometry of the same shape as its global counterpart. See Figure 5 (b-d) for examples.

This observation can be explained by deconstructing the covariance of the attention output:

Σ Att = α 2 ​ Σ A + ( 1 − α ) 2 ​ Σ B , \displaystyle\Sigma_{\textsc{Att}}=\alpha^{2}\Sigma_{A}+(1-\alpha)^{2}\Sigma_{B}, (5) where Σ A = Cov ​ ( A i ) , Σ B = Cov ​ ( B j ) \Sigma_{A}=\text{Cov}(A_{i}),\Sigma_{B}=\text{Cov}(B_{j}) . First, note that if we ignore positional encodings, Σ A \Sigma_{A} and Σ B \Sigma_{B} share the same eigenvectors, as they each depend on the same terms ( E ⁡ [ ⋅ ] , W O , W V E[\cdot],W_{O},W_{V} ), which are picked by PCA. Further note that fixing a value for a i a_{i} leaves a local covariance, Σ l ​ o ​ c ​ a ​ l | a i = ( 1 − α ) 2 ​ Σ B \Sigma_{local|a_{i}}=(1-\alpha)^{2}\Sigma_{B} , which again share the same eigenvectors with the global Σ Att \Sigma_{\textsc{Att}} term, leading to the same local geometry when projected onto.

### 4.2 Embedding Digits on a Pentagonal Prism via Fourier Bases

Similar to Kantamneni & Tegmark (2025) , we find that our model encodes digits in Fourier space. Specifically, the model’s embeddings E E , the final hidden layer 𝐡 L \mathbf{h}^{L} , and even the weights of the last MLP can be well reconstructed from a small set of Fourier basis functions.

Figure 6 shows a 3D PCA visualization of the final hidden layer at timestep t c 2 t_{c_{2}} , for both the SFT and ICoT models. While the SFT hidden states do not reveal any obvious patterns, the ICoT hidden states reveal a striking pattern: the ten digits form vertices of a pentagonal prism .

This structure is naturally explained by Fourier modes. Consider the Fourier expansion ∑ C n ∗ e − 2 ​ π ​ i ​ k ​ n 10 , n = 0 , … , 9 . \sum C_{n}*e^{-2\pi i\tfrac{kn}{10}},\quad n=0,\ldots,9.

where C n ( ≠ c k ) C_{n}(\neq c_{k}) is some constant per digit n n . Following Kantamneni & Tegmark (2025) , we take frequencies k ∈ { 0 , 1 , 2 , 5 } k\in\{0,1,2,5\} , yielding the real Fourier basis Φ ⁡ ( n ) = [ 𝟏 ​ ( n ) cos ⁡ ( 2 ​ π ​ n 10 ) sin ⁡ ( 2 ​ π ​ n 10 ) cos ⁡ ( 2 ​ π ​ n 5 ) sin ⁡ ( 2 ​ π ​ n 5 ) 𝒑 ⁡ ( n ) ( k = 0 ) ( k = 1 ) ( k = 1 ) ( k = 2 ) ( k = 2 ) ( k = 5 ) ] , \Phi(n)=\left[\begin{array}[]{c@{\quad}c@{\quad}c@{\quad}c@{\quad}c@{\quad}c}\mathbf{1}(n)&\cos\!\left(2\pi\tfrac{n}{10}\right)&\sin\!\left(2\pi\tfrac{n}{10}\right)&\cos\!\left(2\pi\tfrac{n}{5}\right)&\sin\!\left(2\pi\tfrac{n}{5}\right)&\bm{p}(n)\\[-2.0pt] \scriptstyle(k=0)\hfil\hskip 8.19447pt&\scriptstyle(k=1)\hfil\hskip 8.19447pt&\scriptstyle(k=1)\hfil\hskip 8.19447pt&\scriptstyle(k=2)\hfil\hskip 8.19447pt&\scriptstyle(k=2)\hfil\hskip 8.19447pt&\scriptstyle(k=5)\end{array}\right], where 𝟏 ​ ( n ) ≡ 1 \mathbf{1}(n)\equiv 1 (the DC component) and 𝒑 ⁡ ( n ) ≡ ( − 1 ) n \bm{p}(n)\equiv(-1)^{n} (the Nyquist/parity vector). The sine terms for k = 0 k=0 and k = 5 k=5 vanish over n = 0 , … , 9 n=0,\dots,9 and are omitted.

The final hidden layer 𝐡 L \mathbf{h}^{L} can be reconstructed via these six terms (see Appendix B ), indicating that the final hidden state is encoded using Fourier bases.

Revisiting Figure 6 , the first principal component (PC1) aligns with the parity vector 𝒑 ⁡ ( n ) \bm{p}(n) , separating even from odd digits. Second and third principal components span the k = 2 k=2 Fourier pair ( cos , sin ⁡ ( 2 ​ π ​ n 5 ) \cos,\sin(\tfrac{2\pi n}{5}) ), so the digits lie on two regular pentagons: one each for even and odd digits. The digits within each pentagon advance by n + 4 ( mod 10 ) n+4\pmod{10} (e.g., n = → → 8 ​ … n=0\!\to\!4\!\to\!8\ldots , same for odd digits), allowing a walk around the pentagon while staying within the even/odd set. Interestingly, taking ( mod 5 ) \pmod{5} on such a sequence yields decreasing steps of 1 ( n ( mod 5 ) = → → 3 ​ … n\pmod{5}=0\!\to\!4\!\to\!3\ldots ). Lastly, the two pentagons are parallel and stacked along PC1, with corresponding vertices differing by ± 5 \pm 5 (same phase, opposite parity). Together, these yield the pentagonal-prism geometry in Figure 6 .

## 5 Pitfalls of Learning: Lack of Long-Range Dependency

Given our insights, we revisit why Transformers fail at multiplication under standard fine-tuning.

In particular, in Figure 7 (a), we inspect the gradient norms (top row) and losses (bottom row) per token c k c_{k} over the course of training. There are a few observations to make.

First, note from the loss curves that the first two digits, c 0 , c 1 c_{0},c_{1} , followed by the last digit, c 7 c_{7} , are learned first, as indicated by their immediate drop in loss to near zero. This aligns with the gradient norms observed for these tokens: within the first few steps, these tokens receive gradients, but their norms quickly drop to near zero once the loss for these tokens reach zero. Also note that the order in which tokens are learned according to gradient norms and losses is consistent.

The model then eventually learns to predict c 2 c_{2} . However, middle digits, c 3 c_{3} to c 6 c_{6} are never learned. Despite only the middle digits receiving gradients (as they are the only sources of loss remaining), their losses plateau, suggesting that the model is stuck in a local optimum that lacks the long-range dependencies to properly learn the middle digits.

Note that scaling to a larger model does not address this issue, as the same pattern can be found in a 12 layer 8 head model: see Appendix C .

## 6 Learning Multiplication Without ICoT

To further validate our understanding of why Transformers fail to learn multiplication, we demonstrate an example of a simple fix to teach Transformers multiplications without needing ICoT.

In particular, we leverage the observation from Section 3.1 in that ( i i ) multi-digit multiplication requires long-range dependencies between digit c k c_{k} and pairwise products { a i ​ b j | i + j ≤ k } \{a_{i}b_{j}|i+j\leq k\} , and ( i ​ i ii ) such dependency can be summarized by an intermediate value c ^ k \hat{c}_{k} to produce c k c_{k} .

Thus in order to guide the Transformer to learn long-range dependencies, we simply add an auxiliary loss term to predict c ^ k \hat{c}_{k} at each timestep t c k t_{c_{k}} . We attach an additional linear regression head 𝐰 h ∈ ℝ d \mathbf{w}_{h}\in\mathbb{R}^{d} to the output of H ( = 2 ) H(=2) attention heads in the second layer. These regression heads are trained to predict the correct accumulated sum c ^ k \hat{c}_{k} at each timestep t c k , c k ∈ [ 0 , … ​ 7 ] t_{c_{k}},c_{k}\in[0,\ldots 7] with a MSE loss: z i h \displaystyle z_{i}^{h} = 𝐰 h ⊤ ​ Att h 2 ​ ( ⋅ ) \displaystyle=\mathbf{w}^{\top}_{h}\textsc{Att}^{2}_{h}(\cdot) (6) ℒ a ​ u ​ x \displaystyle\mathcal{L}_{aux} = 1 H ​ ∑ h ∈ H 1 8 ​ ∑ i = 0 7 ( z i h − c ^ i ) 2 \displaystyle=\frac{1}{H}\sum_{h\in H}\frac{1}{8}\sum_{i=0}^{7}(z_{i}^{h}-\hat{c}_{i})^{2} (7) ℒ \displaystyle\mathcal{L} = ℒ L ​ M + λ ​ ℒ a ​ u ​ x \displaystyle=\mathcal{L}_{LM}+\lambda\mathcal{L}_{aux} (8)

where ℒ L ​ M \mathcal{L}_{LM} is the standard language modeling loss.

This introduces an inductive bias for the task, and allows our 2-layer model to correctly learn 4x4 multiplication with 99% accuracy. Again, note that a larger 12-layer model still fails at multiplication under standard fine-tuning.

Revisiting Figure 7 (b) demonstrates a very different learning dynamic. We observe the model learn early and last digits ( c 0 , c 1 , c 7 c_{0},c_{1},c_{7} ) and work inwards ( c 2 , c 3 , c 4 , c 6 c_{2},c_{3},c_{4},c_{6} , and finally c 5 c_{5} ).

Limitation. Obviously the suggested inductive bias pertains specifically to our task. However, our experiments demonstrate the pitfall of Transformers that require long-range dependencies, and that it is possible to overcome such a pitfall with the correct inductive biases. We speculate that there are other generalizing inductive biases that can improve performance on tasks with long-range dependencies ( Tay et al., 2020 ) , and leave this for future work.

### 6.1 Does the model with auxiliary loss learn the same mechanism as ICoT?

A natural question that arises is whether ICoT and our inductive bias leads to the same mechanisms.

Inspecting the attention patterns suggests that a similar (but not necessarily exact) mechanism is learned: see Figure 8 . Namely, the model similarly forms an “attention tree” to sparsely attend to the correct pairs of digits for each c i c_{i} in the first layer (red boxes). Interestingly, in the auxiliary-loss model we also observe an attention head (Layer 2 Head 2) that simultaneously attends to all the necessary digits, { a i ≤ k , b i ≤ k } \{a_{i\leq k},b_{i\leq k}\} , at each timestep t c k t_{c_{k}} , forming a parallelogram-like attention pattern (black box) akin to the shape seen in Figure 2 .

## 7 Related Work

##### Studying Transformers with Arithmetic Tasks.

A growing line of work study Transformers under controlled settings to better characterize their behavior ( Allen-Zhu & Li, 2023a ; Allen-Zhu & Li, 2023b ; Li et al., 2023 ; Nanda et al., 2023b ; Park et al., 2024b ; Park et al., 2024a ) . Often, arithmetics is a natural and popular domain ( Lee et al., 2023 ; Ye et al., 2024 ; Nikankin et al., 2024 ) , which has led to numerous insights. For instance, Nanda et al. (2023a) study how Transformers perform modular addition to explain grokking. Kantamneni & Tegmark (2025) find that large language models use trigonometry to do addition, encoding digits using Fourier bases, while Nikankin et al. (2024) suggest that they also rely on heuristics. Cai et al. (2025) study length generalization in Transformers using arithmetic tasks. Similarly, we study the limitations of Transformers by studying why it fails to learn multi-digit multiplication.

Process Supervision. Recent work trains models with process supervision , in which feedback is given not just on final correctness but on each intermediate reasoning step. For example, Uesato et al. (2022) demonstrate that process-supervision can yield less reasoning errors on GSM8K compared to outcome-only supervision. Similarly, Lightman et al. (2023) show that step-level human feedback on MATH leads to stronger reward models. More recently, Zhong et al. (2023) ’s Math-Shepherd automates step-wise rewards via continuation-based verification, improving performance on both GSM8K and MATH. ICoT similarly plays the role of process supervision in latent space, by slowly removing chain-of-thought tokens during training such that the model internalizes the reasoning procedure. We thus use ICoT’s success on multiplication to study why Transformers fail.

## 8 Conclusion

In this work, we study why Transformers fail on a seemingly simple task of multi-digit multiplication. We answer this question by reverse-engineering a model trained with implicit chain-of-thought, and uncover that it has learned to compute the correct long-range dependencies needed for multi-digit multiplication. Our findings point to a pitfall of the standard recipe for training language models: using gradient descent with an auto-regressive loss on Transformers does not encourage the model to learn the right long-range dependencies. While we provide a simple example of how the right inductive bias can address such a limitation, we anticipate future work to provide a generic solution to improve on tasks with long-range dependencies.

#### Acknowledgments

XB, CT acknowledge the support of NSF grants IIS-2126602. YD is supported by an NSERC Discovery Grant (RGPIN-2024-05178) and a Starter Grant from the University of Waterloo. AL, MW, and FV acknowledge support from the Superalignment Fast Grant from OpenAI, Effective Ventures Foundation, Effektiv Spenden Schweiz, and the Open Philanthropy Project. IP thanks Keya Hu for constructive feedback and AL thanks Thomas Fel for fruitful discussions regarding feature geometry.

### Reproducibility Statement

Our code to reproduce all of our experiments can be found in https://github.com/ajyl/icot . Appendix A provides details of our training setup, including data formats, sample size, and hyperparameters.

## Appendix A Training Details

Here we provide additional details regarding the training procedures of each of our models.