A Mechanistic Analysis of Looped Reasoning Language Models โ€” part 2/4

Cleaned-text code points [15342, 31256). Bibliography omitted here; source.md includes it.

Test-time computation broadly refers to giving a model the ability to expend additional computational cycles at inference in proportion to the difficulty of the input, rather than committing to a fixed compute budget for all examples. In this paper, we focus specifically on recurrence as a mechanism for scaling computation at test time, as opposed to alternative strategies such as early-exit architectures ( Schuster et al., 2022 ) or continuous thought machines ( Darlow et al., 2025 ) . Classic approaches to adaptive test-time compute include Adaptive Computation Time ( Graves, 2016 ) and subsequent probabilistic halting frameworks such as PonderNet ( Banino et al., 2021 ) . Building on these foundations, recent work has begun to characterize when additional inference compute actually generalizes beyond training-time budgets ( Schwarzschild et al., 2021 ) , how to mitigate failures due to excessive computation (โ€œoverthinkingโ€) ( Bansal et al., 2022 ) , how to ensure stable dynamics with repeated iteration ( Bear et al., 2024 ) , and how looped Transformers can better generalize to out of distribution tasks at test time ( McLeish et al., 2024 ) .

## 4 Looped Transformers Tend Towards the Same Attention Patterns

Recent work ( Bai et al., 2019 ; Bansal et al., 2022 ; Anil et al., 2022 ) has noted that weight-tied Transformer models often tend towards consistent behavior with repeated iterations. Often 1 1 1 As originally observed by Geiping et al. (2025) , other forms of limiting behavior can occur. In App. C we show that 1) this is rare, the vast majority of tokens reach a fixed-point and 2) even with other limiting behavior, stages of inference remain constant. , this takes the form of convergence to a fixed point ๐‘ฟ โ€ฒ = S k โก ( ๐‘ฟ โ€ฒ ) {\bm{X}}^{\prime}=\stack_{k}({\bm{X}}^{\prime}) . We motivate our work by noting that if this is true for a model with cyclic recurrence, it is also true cyclically:

###### Proposition 4.1 (Cyclic recurrent blocks reach cyclic fixed points) .

Let ( l , k ) (l,k) -Recurrent block reach a fixed point ๐— โ€ฒ {\bm{X}}^{\prime} such that S k โก ( ๐— โ€ฒ ) = ๐— โ€ฒ \stack_{k}({\bm{X}}^{\prime})={\bm{X}}^{\prime} . Then any cyclic permutation of blocks 1 , โ€ฆ , k 1,\dots,k will also have reached a fixed point.

We highlight however that these fixed points are not necessarily the same: the action of each successive layer doesnโ€™t necessarily result in the same point, and the cycle of layers can instead trace out an arbitrary cycle in latent space. Indeed, in Sec. 4.1 we demonstrate that this cyclic fixed point behavior โ€“ illustrated in Fig. 1 โ€“ is observed frequently in practice. The alternative, where all layers result in the same fixed point, requires that the action of each Transformer block tends to zero.

Convergence to this cyclic behavior implies that the residual stream tends towards being similar across recurrences. Given that block weights are also shared across recurrent iterations -- and assuming that the inputs to each block are bounded 2 2 2 This is a reasonable assumption since all models considered in this work apply a norm before the attention block. โ€“ this implies that the attention patterns will converge, as shown in Proposition 4.2 .

###### Proposition 4.2 (Recurrent attention patterns change slowly under state convergence) .

Fix a layer โ„“ \ell in the recurrent block and consider its attention weight matrices to be tied across recurrences (so ๐– Q , โ„“ , ๐– K , โ„“ {\bm{W}}_{Q,\ell},{\bm{W}}_{K,\ell} are the same for all t t ). Let โˆฅ โ‹… โˆฅ \|\cdot\| denote a submultiplicative matrix norm that is invariant under transposition (e.g. Spectral or Frobenius norms). Assume the corresponding attention inputs are bounded under this norm as โ€– ๐— โ„“ , t โ€– โ‰ค B \|{\bm{X}}_{\ell,t}\|\leq B for all t t . Define ฮบ โ„“ = โ€– ๐– Q , โ„“ โ€‹ ๐– K , โ„“ โŠค โ€– \kappa_{\ell}=\|{\bm{W}}_{Q,\ell}{\bm{W}}_{K,\ell}^{\top}\| . Then, writing ๐’ฎ โ„“ โ€‹ ( ๐— ) := softmax โก ( A โ„“ โ€‹ ( ๐— ) ) \mathcal{S}_{\ell}({\bm{X}}):=\mathrm{softmax}(A_{\ell}({\bm{X}})) with A โ„“ โ€‹ ( โ‹… ) A_{\ell}(\cdot) as defined above, for any t โ‰ฅ 1 t\geq 1 , โ€– ๐’ฎ โ„“ โ€‹ ( ๐‘ฟ โ„“ , t ) โˆ’ ๐’ฎ โ„“ โ€‹ ( ๐‘ฟ โ„“ , t โˆ’ 1 ) โ€– โ‰ค L sm โ€‹ 2 โ€‹ B โ€‹ ฮบ โ„“ d โ€‹ โ€– ๐‘ฟ โ„“ , t โˆ’ ๐‘ฟ โ„“ , t โˆ’ 1 โ€– , \big\|\mathcal{S}_{\ell}({\bm{X}}_{\ell,t})-\mathcal{S}_{\ell}({\bm{X}}_{\ell,t-1})\big\|\;\leq\;L_{\mathrm{sm}}\,\frac{2B\,\kappa_{\ell}}{\sqrt{d}}\,\big\|{\bm{X}}_{\ell,t}-{\bm{X}}_{\ell,t-1}\big\|, where L sm L_{\mathrm{sm}} is a Lipschitz constant of the row-wise softmax with respect to the chosen norm.

Since these attention patterns are characteristic of the different mixing stages of inference (defining, for example, ColSum concentration), we see that mixing behavior will tend towards being constant across recurrences.

### 4.1 Empirical Validation

We focus our attention on three different pretrained looped language models: Ouro 1.4B ( Zhu et al., 2025 ) , Huginn-0125 ( Geiping et al., 2025 ) , and Llama with retrofitted recurrence ( McLeish et al., 2025 ) . Additional models can be found in App. D , with architecture and training choices summarized in Table 1 . These models cover a range of design choices; later in Sec. 4.2 we will isolate the impact of these architectural differences. Except where otherwise specified, all results are visualized on the same random subset of 256 examples from the GSM8k test set; additional results targetting non-reasoning behavior are presented in Sec. E.4 , but we observe no significant changes and the conclusions of the main text remain unchanged.

We start by plotting the norm of the residual stream difference between subsequent iterations in Fig. 3 (and the same for cosine similarities in Fig. 24 ). This validates the starting assumption of Proposition 4.2 that looped models tend towards behavior in which the layerwise residual stream does not significantly change between recurrences.

For each of these models, we plot Frobenius norm between the realized attention matrices at different layers, for 8 loops of the recurrent block in Fig. 2 . The diagonal patterns of high similarity demonstrate that the attention matrices of any given layer are most similar to those of the same layer at different recurrences โ€“ as predicted by Proposition 4.2 . We note that this convergence towards similar attention patterns occurs remarkably quickly : for looped Ouro attention patterns appear to converge after the first iteration, and both Huginn-0125 and the retrofitted Llama model demonstrate this cyclic behavior immediately following the prelude.

However, despite the tendency visible in Fig. 3 towards small changes between successive iterations we discover that it is not the case that looped models always reach a fixed point, or even consistent limiting behavior: for each model we find an โ€œapproximate fixed pointโ€ per layer by iterating 128 times, then compute the norm of the difference between the output of each layer at every recurrence and its corresponding fixed point. This is visualized in Fig. 4 . We see that while Huginn-0125 and Retrofitted Llama demonstrate fast convergence to a fixed point, Ouro does not. As discussed by Bansal et al. (2022) ; Anil et al. (2022) , this supports the suggestion that input injection encourages fixed-point convergence: we investigate this further in Sec. 4.2 .

Where a model does reach a fixed point, this implies that the action of the entire recurrent block tends towards tracing out a consistent cycle in latent space. We visualize this for retrofitted Llama in Fig. 5 .

### 4.2 Impact of Architecture Choices

Several existing works ( Bansal et al., 2022 ; Anil et al., 2022 ) have noted that input injection is important in order for a recurrent model to reach a fixed point: in this section we replicate this finding and supplement with additional insights on the impact of norm structure in reaching a fixed point. We conduct a series of experiments on randomly initialized models. These demonstrate similar cyclic behavior to their trained counterparts, suggesting that behavior observed here is likely to generalize to the cyclic behavior of trained models. We compare pre-norm (used by the retrofitted recurrent models) and the norms used by the Huginn-0125 and Ouro models, testing each both with and without input injection: in this way we test the most significant architectural differences between the pretrained Looped models tested; see Table 1 for details. Each model has 12 layers with no prelude or coda; see Fig. 31 for alternative configurations.

Our results are visualized in Fig. 6 , where we visualise the mean over 3 random model initializations for each configuration. We see that input injection results in stable fixed point behavior for all norm types other than Ouro, whereas omitting input injection means that only pre-norm reaches a stable fixed point. However, this fixed point reached by pre-norm without input injection is a โ€œdegenerateโ€ one: each layer converges to the same fixed point. This can be determined from the rightmost frame of Fig. 6 , which demonstrates that the lowest cosine similarity between the first layer and any other layerโ€™s fixed point still converges to 1.

## 5 Stages of Inference in Looped Models Mirror Feedforward Computation

The previous section shows that, empirically, a wide range of models converge to a regime in which attention patterns within individual layers change only minimally across recurrences. As a result, attention dynamics in looped Transformers are constrained in depth, since layers are cyclically weight-tied to earlier ones. This behavior contrasts with feedforward Transformers, which impose no such constraints and exhibit sharp, layer-wise changes in attention patterns across depth. Prior work has linked these sharp transitions to characteristic stages of inference ( Lad et al., 2024 ; Queipo-de-Llano et al., 2025 ) , introduced in Sec. 2.2 . In this section, we study how cyclic weight sharing alters these stages of inference in looped Transformers. In the main text we frame our analysis using ColSum concentration , a metric for identifying stages of inference introduced in Sec. 2.2 , with extensive additional results in Sec. E.3 .

We visualize ColSum concentration over the realized depth of Retrofitted Llama in Fig. 7 , revealing consistent mixing cycles that repeat with every iteration of the recurrent block. However, each individual layer (solid colorful lines) changes very little in realized depth: after an initial transitory phase they quickly converge towards constant behavior.

Instead of occurring throughout the realized depth of the looped model, we find that the familiar feedforward stages of inference occur within each looped block. Fig. 8 demonstrates that ColSum concentration within each looped block closely resembles that of feedforward models. We draw attention to two observations: 1) Ouro 1.4B, despite being trained from scratch with recurrence, mirrors Llama mixing stages and 2) the retrofitted models closely follow the stages of inference of their associated base model, but the initial and final stages are performed only once by the prelude and coda respectively, while the โ€œmiddleโ€ stages are repeated in the recurrent block. It is particularly remarkable that these stages of inference appear in each Ouro recurrent block when pretraining from initialization; we discuss further the formation of stages of inference, and attempt to isolate their formation from specific training procedures, in Sec. 5.1 .

However, Huginn-0125 does not demonstrate clear stages of inference ( Fig. 36 ). We suggest that this is likely due to the specific norm structures used by these models ( Table 1 ). Huginn-0125 and Ouro both use a โ€œsandwichโ€ norm structure, but Huginn-0125 implements this by normalizing the residual streams whereas Ouro instead normalizes the outputs of the attention and MLP units, only normalizing the residual stream at the end of each recurrent block. We demonstrate the impact of the different norms by plotting residual stream magnitudes for a range of models in Fig. 9 . This means that Huginn-0125 is unable to develop the growth in residual stream magnitude that Queipo-de-Llano et al. (2025, Section 3.3) note causes compression behavior, leading to sink formation and stages of inference.

### 5.1 Self-Organization Into Stages of Inference

An open question from our analysis of pretrained models ( Fig. 8 ) is whether stages of inference emerge naturally during pretraining, or whether they are instead induced by specific aspects of the training procedure. Several mechanisms may introduce an implicit bias toward feedforward stages of inference: retrofitted models ( McLeish et al., 2025 ) may inherit the inference stages of the underlying base model; models trained with recurrence schedulers that permit a single recurrence ( Geiping et al., 2025 ; McLeish et al., 2025 ) are partially optimized as feedforward models; training objectives that decompose into separate loss terms for each recurrence ( Zhu et al., 2025 ) partially correspond to feedforward model training.

Therefore in this section we seek to investigate whether stages of inference can arise in Looped Transformers without these training biases. To explore this we pre-train several small-scale Looped Transformers, explicitly removing the biases above: we pre-train from scratch with a constant recurrence of 4, using a standard loss that considers only the final latent state when predicting the next token. Our code and training procedure are adapted from Karpathy (2025) ; additional details can be found in App. B .

ColSum concentrations for these trained Looped Transformers with configurations ( 2 , 4 โŠ— 4 , 2 ) (2,4\otimes 4,2) , ( 2 , 8 โŠ— 4 , 2 ) (2,8\otimes 4,2) and ( 2 , 12 โŠ— 4 , 2 ) (2,12\otimes 4,2) are plotted in Fig. 10 . We compare these to a โ€œcontrolโ€ feedforward network without recurrence, of depth 12. These experiments are on a small scale and as such need to be treated with caution. However, they appear to provide initial evidence that โ€“ even without training methods that may bias towards feedforward stages of inference โ€“ looped models have a tendency to self-organize into multiple different mixing stages in recurrent depth, which resemble feedforward stages. The fact that the optimization of these models results in these mixing stages of inference suggests that they are beneficial to language modeling even when applied repeatedly in recurrent depth. We additionally test the impact of input injection and sandwich layers in Sec. E.6 .

### 5.2 Stability to Unseen Numbers of Recurrences

Sec. 4 establishes that some looped Transformer architectures converge to a fixed point (such as the retrofitted series and Huginn-0125) while others do not (such as Ouro). However, when evaluating these looped Transformers within the range of recurrences on which they were trained ( Sec. 5 ), we see extremely similar behavior: convergence to feedforward stages of inference within each recurrent block.

In this section, we demonstrate that a significant difference arises when generalizing to unseen test-time recurrence depths: models that do not reach a fixed point exhibit unstable stages of inference . Conversely, models with recurrent-block-wise stages of inference that also converge to a fixed point are guaranteed to keep enacting these stages of inference for arbitrary test time recurrences.

We first verify this stability in Fig. 11 , an extended plot of Fig. 7 . This demonstrates that for retrofitted Llama (results hold for other models using input injection), each individual layer quickly reaches a stable states and then exhibits consistent stages of inference behavior for an arbitrary number of test time recurrences. However, Ouro 1.4B does not exhibit this behavior, with individual layers changing continuously throughout later recurrences. The effect that this has on stages of inference is visualized in Fig. 12 .