Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls — part 3/3

Cleaned-text code points [31923, 41094). Bibliography omitted here; source.md includes it.

### A.1 ICoT Training

Our ICoT training setup follows the practice outlined in the original ICoT paper ( Deng et al., 2024 ) .

ICoT works by initially presenting explicit chain-of-thought tokens during training, but gradually removing them across numerous “stages” (e.g., epochs). Concretely, the training examples at each epoch may have the following form: ( Epoch 1 ) \displaystyle(\text{Epoch 1}) a 0 ​ a 1 ​ a 2 ​ a 3 ∗ b 0 ​ b 1 ​ b 2 ​ b 3 % % % ​ q 0 ​ … ​ q i ​ … ​ q j ​ … ​ q k ​ … ​ q τ ​ # ​ # ​ # ​ # ​ c 0 ​ … ​ c 7 \displaystyle\quad a_{0}a_{1}a_{2}a_{3}*b_{0}b_{1}b_{2}b_{3}\%\%\%\ q_{0}\ldots q_{i}\ldots q_{j}\ldots q_{k}\ldots q_{\tau}\ \#\#\#\#\ c_{0}\ldots c_{7} ( Epoch 2 ) \displaystyle(\text{Epoch 2}) a 0 ​ a 1 ​ a 2 ​ a 3 ∗ b 0 ​ b 1 ​ b 2 ​ b 3 % % % ​ q i ​ … ​ q j ​ … ​ q k ​ … ​ q τ ​ # ​ # ​ # ​ # ​ c 0 ​ … ​ c 7 \displaystyle\quad a_{0}a_{1}a_{2}a_{3}*b_{0}b_{1}b_{2}b_{3}\%\%\%\ q_{i}\ldots q_{j}\ldots q_{k}\ldots q_{\tau}\ \#\#\#\#\ c_{0}\ldots c_{7} ( Epoch 3 ) \displaystyle(\text{Epoch 3}) a 0 ​ a 1 ​ a 2 ​ a 3 ∗ b 0 ​ b 1 ​ b 2 ​ b 3 % % % ​ q j ​ … ​ q k ​ … ​ q τ ​ # ​ # ​ # ​ # ​ c 0 ​ … ​ c 7 \displaystyle\quad a_{0}a_{1}a_{2}a_{3}*b_{0}b_{1}b_{2}b_{3}\%\%\%\ q_{j}\ldots q_{k}\ldots q_{\tau}\ \#\#\#\#\ c_{0}\ldots c_{7} … \displaystyle\ldots ( Epoch N ) \displaystyle(\text{Epoch N}) a 0 ​ a 1 ​ a 2 ​ a 3 ∗ b 0 ​ b 1 ​ b 2 ​ b 3 % % % ​ # ​ # ​ # ​ # ​ c 0 ​ … ​ c 7 \displaystyle\quad a_{0}a_{1}a_{2}a_{3}*b_{0}b_{1}b_{2}b_{3}\%\%\%\ \#\#\#\#\ c_{0}\ldots c_{7} where q i q_{i} are CoT tokens and % , # \%,\# are special delimiters. These delimiters have no special meaning beyond matching the setup of Deng et al. (2024) . Note that after each epoch, the model sees a shorter chain by truncating some tokens, and that by the end, only the operands and final answer remain.

The actual format of our ICoT data is as follows. Using an example input of 8331 × 5015 8331\times 5015 , digits are presented in least-significant digits first, resulting in the following format: 1338 ∗ 5105 | | 5614 + 013380 ( 569421 ) + 0000000 ( 5694210 ) + 0005561 % % # # # # 56997714 1338*5105||5614+013380(569421)+0000000(5694210)+0005561\%\%\#\#\#\#56997714

Unlike Deng et al. (2024) , instead of using a pre-trained 12-layer GPT model, we train a smaller 2-layer, 4-head GPT-based model from scratch, not only to remove any confounding factors from pre-trained knowledge, but also because the 2-layer 4-head architecture is the simplest form in which ICoT succeeds but standard fine-tuning fails. The training data consists of 80,800 samples, while the validation and test sets each contain 1,000 held out samples. We train with a learning rate of 5e-5 , and remove 8 chain-of-thought tokens at every “stage” (which in our case is an epoch). Both training and validation loss converge after 13 epochs, and achieves 100% accuracy on the test set.

### A.2 Standard Fine-Tuning

Similar to ICoT, for our standard fine-tuning model, we train a 2-layer, 4-head GPT-based model from scratch, on the same data as ICoT. We use a learning rate of 5e-5 , and the input format is a 0 ​ a 1 ​ a 2 ​ a 3 ∗ b 0 ​ b 1 ​ b 2 ​ b 3 % % ​ # ​ # ​ # ​ # ​ c 0 ​ … ​ c 7 a_{0}a_{1}a_{2}a_{3}*b_{0}b_{1}b_{2}b_{3}\%\%\#\#\#\#c_{0}\ldots c_{7} . All other hyperparameters match those in our ICoT setup. The model’s loss and accuracy plateaus after 13 epochs, it achieves only about 1% train and validation accuracy, while digit-level accuracy converges at approximately 81%, and remains the same even after 60 epochs.

Note that scaling the model larger to a 12-layer, 8-head model achieves the same low accuracy at 1% and digit-level accuracy of 80%.

## Appendix B Fourier structure in model’s weights, activations

Here we provide a deeper dive into the Fourier structure found in the ICoT model’s weights and hidden states. Namely, we analyze the model’s embedding weights, final MLP’s weights, and last hidden layer:

1. Embeddings ℰ ∈ ℝ 10 × d \mathcal{E}\in\mathbb{R}{}^{10\times d}

2. Final layer MLP output weights W o ​ u ​ t ∈ ℝ d m ​ l ​ p × d W_{out}\in\mathbb{R}{}^{d_{mlp}\times d} , given MLP ​ ( 𝐱 ) = σ ⁡ ( W i ​ n ​ 𝐱 ) ​ W o ​ u ​ t \text{MLP}(\mathbf{x})=\sigma(W_{in}\mathbf{x})W_{out}

3. Final hidden layer 𝐡 L t ∈ ℝ N × d \mathbf{h}^{L}_{t}\in\mathbb{R}{}^{N\times d}

where N ( = 1,000 ) N(=1,000) is the size of our validation set. For the latter two, we first project them onto the model’s embedding space: W o ​ u ​ t ^ = ( ℰ W o ​ u ​ t ) ⊤ ∈ ℝ d m ​ l ​ p × 10 \widehat{W_{out}}=(\mathcal{E}W_{out})^{\top}\in\mathbb{R}{}^{d_{mlp}\times 10} 𝐡 L ^ = ( ℰ 𝐡 L ) ⊤ ∈ ℝ N × 10 \widehat{\mathbf{h}^{L}}=(\mathcal{E}\mathbf{h}^{L})^{\top}\in\mathbb{R}{}^{N\times 10}

Each item X ∈ { ℰ , W o ​ u ​ t ^ , 𝐡 L ^ } X\in\{\mathcal{E},\widehat{W_{out}},\widehat{\mathbf{h}^{L}}\} is a collection of row vectors 𝐱 ∈ ℝ 10 \mathbf{x}\in\mathbb{R}{}^{10} whose ten entries correspond to digits n = 0 , … , 9 n=0,\ldots,9 .

We find that vectors 𝐱 \mathbf{x} are encoded in a low-dimensional trigonometric subspace.

Namely, consider the Fourier expansion ∑ C n ∗ e − 2 ​ π ​ i ​ k ​ n 10 , n = 0 , … , 9 . \sum C_{n}*e^{-2\pi i\tfrac{kn}{10}},\quad n=0,\ldots,9.

where C n ( ≠ c k ) C_{n}(\neq c_{k}) is some constant per n n . Following Kantamneni & Tegmark (2025) , we take frequencies k ∈ { 0 , 1 , 2 , 5 } k\in\{0,1,2,5\} , yielding the real Fourier basis Φ ⁡ ( n ) = [ 𝟏 ​ ( n ) cos ⁡ ( 2 ​ π ​ n 10 ) sin ⁡ ( 2 ​ π ​ n 10 ) cos ⁡ ( 2 ​ π ​ n 5 ) sin ⁡ ( 2 ​ π ​ n 5 ) 𝒑 ⁡ ( n ) ( k = 0 ) ( k = 1 ) ( k = 1 ) ( k = 2 ) ( k = 2 ) ( k = 5 ) ] , \Phi(n)=\left[\begin{array}[]{c@{\quad}c@{\quad}c@{\quad}c@{\quad}c@{\quad}c}\mathbf{1}(n)&\cos\!\left(2\pi\tfrac{n}{10}\right)&\sin\!\left(2\pi\tfrac{n}{10}\right)&\cos\!\left(2\pi\tfrac{n}{5}\right)&\sin\!\left(2\pi\tfrac{n}{5}\right)&\bm{p}(n)\\[-2.0pt] \scriptstyle(k=0)\hfil\hskip 8.19447pt&\scriptstyle(k=1)\hfil\hskip 8.19447pt&\scriptstyle(k=1)\hfil\hskip 8.19447pt&\scriptstyle(k=2)\hfil\hskip 8.19447pt&\scriptstyle(k=2)\hfil\hskip 8.19447pt&\scriptstyle(k=5)\end{array}\right], where 𝟏 ​ ( n ) ≡ 1 \mathbf{1}(n)\equiv 1 (the DC component) and 𝒑 ⁡ ( n ) ≡ ( − 1 ) n \bm{p}(n)\equiv(-1)^{n} (the Nyquist/parity vector). The sine terms for k = 0 k=0 and k = 5 k=5 vanish over n = 0 , … , 9 n=0,\dots,9 and are omitted.

Let F ∈ 10 × 6 F\in 10\times 6 be a Fourier matrix with rows indexed by n ∈ { 0 , … , 9 } n\in\{0,\ldots,9\} and columns as defined above.

For each row 𝐱 ∈ ℝ 10 \mathbf{x}\in\mathbb{R}{}^{10} we fit least squares coefficients C = arg min C ∈ ℝ 6 ∥ x − F C ∥ 2 2 C\;=\;\arg\min_{C\in\mathbb{R}{}^{6}}\;\|x-FC\|_{2}^{2} and quantify goodness-of-fit using coefficient of determination R 2 ​ ( x ) = 1 − ‖ x − F ​ C ‖ 2 2 ‖ x − x ¯ ‖ 2 2 , R^{2}(x)\;=\;1-\frac{\|x-FC\|_{2}^{2}}{\|x-\bar{x}\|_{2}^{2}}, We report the median R 2 R^{2} over the set of rows in each X X (i.e., over d mlp d_{\text{mlp}} rows for W o ​ u ​ t ^ \widehat{W_{out}} , d d rows for ℰ \mathcal{E} , and over batch examples for 𝐡 L \mathbf{h}^{L} .

In Table 1 we observe strong fits: the per-row medians lie between 0.85 0.85 and 0.99 0.99 , indicating that a six-dimensional trigonometric basis over digits captures the vast majority of variance:

We can extend the Fourier bases to include additional terms, for k = 3 , 4 k=3,4 , which forms a 8 dimensional basis (excluding sine terms for k = 0 , 5 k=0,5 ), which leads to perfect R 2 R^{2} fits.

## Appendix C Per Token Gradients and Losses: 12-Layer Model

Even with a larger 12 layer model, the model fails to learn the right long-range dependencies. Figure 9 displays the results – we see the similar patterns as the 2-layer model, in which middle digits never receive the right gradients and loss does not drop.

## Appendix D Attention Patterns of All Models

ICoT

SFT

Auxiliary Loss Model

In Section 3.3 , we illustrate how a binary tree is constructed for c 2 c_{2} in the ICoT model. In Figure 10 , we present the attention patterns for all digits across the three models, from with attention trees can be derived for the ICoT model for each solution token c i c_{i} .

## Appendix E LLM Usage

We used LLMs to proof read our draft and polish our notations.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .