A Mechanistic Analysis of Looped Reasoning Language Models — part 3/4

Cleaned-text code points [31256, 47111). Bibliography omitted here; source.md includes it.

Existing research implies that this stability correlates with out-of-domain performance. Models which exhibit “stable” stages of inference for arbitrary test time iterations also avoid performance deterioration in this extrapolation regime: whereas extrapolation beyond training recurrences harms the performance of Ouro ( Zhu et al., 2025 , Tab. 10) , Huginn-0125 performance remains constant in this extrapolation region ( Geiping et al., 2025 , Fig. 1) .

## 6 Conclusion

This paper examines the limiting behavior of Looped Transformers, exploring implications for “mixing” stages of inference observed in feedforward models. We demonstrate across a range of architectures that recurrent blocks tend to “mirror” the stages of a feedforward Transformer, and provide evidence that this may be emergent behavior learned during training, even when not explicitly encouraged by the training process. We further investigate the implications for these mixing stages when models converge to a stable fixed point, and when they do not.

#### Implications of Findings

The implications of our findings are bidirectional. On the one hand, the structure of looped architectures provides a novel lens to study stages of inference, while tracking these stages simultaneously reveals the internal mechanics of recurrent depth. In particular, since looped models decouple functional depth from parameter count, they provide an interesting new perspective on why these stages of inference form: previous work had suggested that these stages exist to mitigate the “harms” of transformer depth ( Barbero et al., 2025 ; Queipo-de-Llano et al., 2025 ) , however our work shows that looped models develop these same stages while simultaneously improving performance with greater recurrent depth. On the other hand, our findings that looped models exhibit (and self-organize into) similar stages of inference to feedforward models means that insights from the feedforward setting can be applied to looped models: predictable stages offer actionable pathways for efficient architectural design, including stage-dependent attention sparsification and the leaner parameterization of middle-stage MLPs where representations are reliably compressed and low-rank.

#### Limitations and Future Work

We focus exclusively on cyclic recurrence as this appears to be the dominant approach in the literature. However, this means our analysis does not extend to sequential recurrence with multiple separate recurrent blocks; for analysis of this setting we refer the reader to Pappone et al. (2025) . Despite empirically investigating the architectural choices that result in stable limiting behavior in looped models, we have not established analytically why this is the case, nor whether this stable limiting behavior is desirable or restrictive for reasoning tasks.

## Impact Statement

Ethical aspects and future societal consequences of this particular work are limited. The goal of our work is to advance understanding of looped Language Models, which themselves seem to demonstrate strong reasoning performance; as such, our work is in support of a field that has potential societal consequences if future models are able to undertake more advanced reasoning tasks. However, we do not within this work introduce any more powerful reasoning models, and the impact of our work is limited to understanding existing models, and potentially guiding future advancements.

## Acknowledgments

HB acknowledges funding support from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems No. EP/S024050/1. MB is partially supported by the EPSRC Turing AI World-Leading Research Fellowship No. EP/X040062/1 and EPSRC AI Hub No. EP/Y028872/1.

## Appendix Contents

## Appendix A Proofs of Propositions

#### Proof of Proposition 4.1

###### Proof sketch.

Note S k ⁡ ( 𝑿 ′ ) = B k ⁡ ( B k − 1 ⁡ ( … ​ B 1 ⁡ ( 𝑿 ′ ) ​ … ) ) = 𝑿 ′ \stack_{k}({\bm{X}}^{\prime})=\block_{k}(\block_{k-1}(\dots\block_{1}({\bm{X}}^{\prime})\dots))={\bm{X}}^{\prime} Applying B 1 \block_{1} to both sides yields to B 1 ⁡ ( B k ⁡ ( B k − 1 ⁡ ( … ​ B 1 ⁡ ( 𝑿 ′ ) ​ … ) ) ) = B 1 ⁡ ( 𝑿 ′ ) . \block_{1}(\block_{k}(\block_{k-1}(\dots\block_{1}(\\ {\bm{X}}^{\prime})\dots)))=\block_{1}({\bm{X}}^{\prime}). Defining 𝒀 ′ = B 1 ⁡ ( 𝑿 ′ ) {\bm{Y}}^{\prime}=\block_{1}({\bm{X}}^{\prime}) , we obtain B 1 ⁡ ( B k ⁡ ( B k − 1 ⁡ ( … ​ B 2 ⁡ ( 𝒀 ′ ) ​ … ) ) ) = 𝒀 ′ \block_{1}(\block_{k}(\block_{k-1}(\dots\block_{2}({\bm{Y}}^{\prime})\dots)))={\bm{Y}}^{\prime} . The general proof follows by induction, and extends trivially to input injection. ∎

###### Proof.

Assume that ( l , k ) (l,k) -Recurrent block S k S_{k} reaches a fixed point such that S k ​ ( 𝑿 ′ ) = 𝑿 ′ S_{k}({\bm{X}}^{\prime})={\bm{X}}^{\prime} . Note that S k ⁡ ( 𝑿 ′ ) = B k − 1 ⁡ ( B k − 2 ⁡ ( … ​ B 0 ⁡ ( 𝑿 ′ ) ​ … ) ) = 𝑿 ′ \stack_{k}({\bm{X}}^{\prime})=\block_{k-1}(\block_{k-2}(\dots\block_{0}({\bm{X}}^{\prime})\dots))={\bm{X}}^{\prime} define the cyclic shift function f n ​ ( i ) = ( i + n ) mod k f_{n}(i)=(i+n)\mod k . Now we aim to prove by induction ∀ n ∈ ℤ + ∪ { 0 } : B f n ​ ( k − 1 ) ⁡ ( B f n ​ ( k − 2 ) ⁡ ( … ​ B f n ​ ( 0 ) ⁡ ( 𝒁 ′ ) ​ … ) ) = 𝒁 ′ \forall n\in\mathbb{Z}_{+}\cup\{0\}:\block_{f_{n}(k-1)}(\block_{f_{n}(k-2)}(\dots\block_{f_{n}(0)}({\bm{Z}}^{\prime})\dots))={\bm{Z}}^{\prime} for some 𝒁 ′ {\bm{Z}}^{\prime} . The base case n = 0 n=0 is trivial as ∀ i < k : f 0 ​ ( i ) = i \forall i<k:f_{0}(i)=i . Now assume for some n = j n=j B f j ​ ( k − 1 ) ⁡ ( B f j ​ ( k − 2 ) ⁡ ( … ​ B f j ​ ( 0 ) ⁡ ( 𝒁 ′ ) ​ … ) ) = 𝒁 ′ \block_{f_{j}(k-1)}(\block_{f_{j}(k-2)}(\dots\block_{f_{j}(0)}({\bm{Z}}^{\prime})\dots))={\bm{Z}}^{\prime} (8) Now, let n = j + 1 n=j+1 . Note f j + 1 ​ ( i ) = ( i + j + 1 ) mod k = f j ​ ( i + 1 ) f_{j+1}(i)=(i+j+1)\mod k=f_{j}(i+1) Therefore B f j + 1 ​ ( k − 1 ) ⁡ ( B f j + 1 ​ ( k − 2 ) ⁡ ( … ​ B f j + 1 ​ ( 0 ) ⁡ ( 𝒀 ) ​ … ) ) \displaystyle\block_{f_{j+1}(k-1)}(\block_{f_{j+1}(k-2)}(\dots\block_{f_{j+1}(0)}({\bm{Y}})\dots)) = B f j ​ ( k ) ⁡ ( B f j ​ ( k − 1 ) ⁡ ( … ​ B f j ​ ( 1 ) ⁡ ( 𝒀 ) ​ … ) ) \displaystyle=\block_{f_{j}(k)}(\block_{f_{j}(k-1)}(\dots\block_{f_{j}(1)}({\bm{Y}})\dots)) (9) = B f j ​ ( 0 ) ⁡ ( B f j ​ ( k − 1 ) ⁡ ( … ​ B f j ​ ( 1 ) ⁡ ( 𝒀 ) ​ … ) ) \displaystyle=\block_{f_{j}(0)}(\block_{f_{j}(k-1)}(\dots\block_{f_{j}(1)}({\bm{Y}})\dots)) (10) Now take Eq. 8 and apply B f j ​ ( 0 ) \block_{f_{j}(0)} to both sides, defining a new fixed point 𝒁 ′′ = B f j ​ ( 0 ) ⁡ ( 𝒁 ′ ) {\bm{Z}}^{\prime\prime}=\block_{f_{j}(0)}({\bm{Z}}^{\prime}) : B f j ​ ( 0 ) ⁡ ( B f j ​ ( k − 1 ) ⁡ ( B f j ​ ( k − 2 ) ⁡ ( … ​ B f j ​ ( 0 ) ⁡ ( 𝒁 ′ ) ​ … ) ) ) \displaystyle\block_{f_{j}(0)}(\block_{f_{j}(k-1)}(\block_{f_{j}(k-2)}(\dots\block_{f_{j}(0)}({\bm{Z}}^{\prime})\dots))) = B f j ​ ( 0 ) ⁡ ( 𝒁 ′ ) \displaystyle=\block_{f_{j}(0)}({\bm{Z}}^{\prime}) (11) B f j ​ ( 0 ) ⁡ ( B f j ​ ( k − 1 ) ⁡ ( B f j ​ ( k − 2 ) ⁡ ( … ​ B f j ​ ( 1 ) ⁡ ( 𝒁 ′′ ) ​ … ) ) ) \displaystyle\block_{f_{j}(0)}(\block_{f_{j}(k-1)}(\block_{f_{j}(k-2)}(\dots\block_{f_{j}(1)}({\bm{Z}}^{\prime\prime})\dots))) = 𝒁 ′′ \displaystyle={\bm{Z}}^{\prime\prime} (12) Combining Eq. 12 and Eq. 10 , we see that there exists a fixed point 𝒁 ′′ {\bm{Z}}^{\prime\prime} such that B f j + 1 ​ ( k − 1 ) ⁡ ( B f j + 1 ​ ( k − 2 ) ⁡ ( … ​ B f j + 1 ​ ( 0 ) ⁡ ( 𝒁 ′′ ) ​ … ) ) = 𝒁 ′′ \block_{f_{j+1}(k-1)}(\block_{f_{j+1}(k-2)}(\dots\block_{f_{j+1}(0)}({\bm{Z}}^{\prime\prime})\dots))={\bm{Z}}^{\prime\prime} Therefore completing the induction step and proving the proposition. ∎

#### Proof of Proposition 4.2

###### Proof.

Define 𝒮 ℓ ​ ( 𝑿 ) := softmax ⁡ ( A ℓ ​ ( 𝑿 ) ) = softmax ⁡ ( 𝑿 ​ 𝑾 Q ​ 𝑾 K ⊤ ​ 𝑿 ⊤ d ) \mathcal{S}_{\ell}({\bm{X}}):=\mathrm{softmax}(A_{\ell}({\bm{X}}))=\mathrm{softmax}{\left(\frac{{\bm{X}}{\bm{W}}_{Q}{\bm{W}}_{K}^{\top}{\bm{X}}^{\top}}{\sqrt{d}}\right)} Let L sm L_{\mathrm{sm}} be the Lipschitz constant of the row-wise softmax such that ‖ 𝒮 ℓ ​ ( 𝑿 ℓ , t ) − 𝒮 ℓ ​ ( 𝑿 ℓ , t − 1 ) ‖ ≤ L sm ​ ‖ A ℓ ​ ( 𝑿 ℓ , t ) − A ℓ ​ ( 𝑿 ℓ , t − 1 ) ‖ \big\|\mathcal{S}_{\ell}({\bm{X}}_{\ell,t})-\mathcal{S}_{\ell}({\bm{X}}_{\ell,t-1})\big\|\leq L_{\mathrm{sm}}\big\|A_{\ell}({\bm{X}}_{\ell,t})-A_{\ell}({\bm{X}}_{\ell,t-1})\big\| (13) Recently Nair (2025) shows that L sm = 1 / 2 L_{\mathrm{sm}}=1/2 .

Define for convenience 𝑴 = 𝑾 Q ​ 𝑾 K ⊤ {\bm{M}}={\bm{W}}_{Q}{\bm{W}}_{K}^{\top} and Δ ℓ , t = 𝑿 ℓ , t − 𝑿 ℓ , t − 1 \Delta_{\ell,t}={\bm{X}}_{\ell,t}-{\bm{X}}_{\ell,t-1} . Then A ℓ ​ ( 𝑿 ℓ , t ) − A ℓ ​ ( 𝑿 ℓ , t − 1 ) \displaystyle A_{\ell}({\bm{X}}_{\ell,t})-A_{\ell}({\bm{X}}_{\ell,t-1}) = 𝑿 ℓ , t ​ 𝑾 Q ​ 𝑾 K ⊤ ​ 𝑿 ℓ , t ⊤ d − 𝑿 ℓ , t − 1 ​ 𝑾 Q ​ 𝑾 K ⊤ ​ 𝑿 ℓ , t − 1 ⊤ d \displaystyle=\frac{{\bm{X}}_{\ell,t}{\bm{W}}_{Q}{\bm{W}}_{K}^{\top}{\bm{X}}^{\top}_{\ell,t}}{\sqrt{d}}-\frac{{\bm{X}}_{\ell,t-1}{\bm{W}}_{Q}{\bm{W}}_{K}^{\top}{\bm{X}}^{\top}_{\ell,t-1}}{\sqrt{d}} = 1 d ​ ( ( Δ ℓ , t + 𝑿 ℓ , t − 1 ) ​ 𝑴 ​ ( Δ ℓ , t + 𝑿 ℓ , t − 1 ) ⊤ − 𝑿 ℓ , t − 1 ​ 𝑴 ​ 𝑿 ℓ , t − 1 ⊤ ) \displaystyle=\frac{1}{\sqrt{d}}\left(\left(\Delta_{\ell,t}+{\bm{X}}_{\ell,t-1}\right){\bm{M}}\left(\Delta_{\ell,t}+{\bm{X}}_{\ell,t-1}\right)^{\top}-{\bm{X}}_{\ell,t-1}{\bm{M}}{\bm{X}}^{\top}_{\ell,t-1}\right) = 1 d ​ ( Δ ℓ , t ​ 𝑴 ​ Δ ℓ , t ⊤ + Δ ℓ , t ​ 𝑴 ​ 𝑿 ℓ , t − 1 ⊤ + 𝑿 ℓ , t − 1 ​ 𝑴 ​ Δ ℓ , t ⊤ + 𝑿 ℓ , t − 1 ​ 𝑴 ​ 𝑿 ℓ , t − 1 ⊤ − 𝑿 ℓ , t − 1 ​ 𝑴 ​ 𝑿 ℓ , t − 1 ⊤ ) \displaystyle=\frac{1}{\sqrt{d}}\left({\Delta_{\ell,t}{\bm{M}}\Delta_{\ell,t}^{\top}}+\Delta_{\ell,t}{\bm{M}}{\bm{X}}_{\ell,t-1}^{\top}+{\bm{X}}_{\ell,t-1}{\bm{M}}\Delta_{\ell,t}^{\top}+\cancel{{\bm{X}}_{\ell,t-1}{\bm{M}}{\bm{X}}_{\ell,t-1}^{\top}}-\cancel{{\bm{X}}_{\ell,t-1}{\bm{M}}{\bm{X}}^{\top}_{\ell,t-1}}\right) = 1 d ​ ( Δ ℓ , t ​ 𝑴 ​ ( Δ ℓ , t + 𝑿 ℓ , t − 1 ) ⊤ + 𝑿 ℓ , t − 1 ​ 𝑴 ​ Δ ℓ , t ⊤ ) \displaystyle=\frac{1}{\sqrt{d}}\left(\Delta_{\ell,t}{\bm{M}}\left(\Delta_{\ell,t}+{\bm{X}}_{\ell,t-1}\right)^{\top}+{\bm{X}}_{\ell,t-1}{\bm{M}}\Delta_{\ell,t}^{\top}\right) = 1 d ​ ( Δ ℓ , t ​ 𝑴 ​ 𝑿 ℓ , t ⊤ + 𝑿 ℓ , t − 1 ​ 𝑴 ​ Δ ℓ , t ⊤ ) \displaystyle=\frac{1}{\sqrt{d}}\left(\Delta_{\ell,t}{\bm{M}}{\bm{X}}_{\ell,t}^{\top}+{\bm{X}}_{\ell,t-1}{\bm{M}}\Delta_{\ell,t}^{\top}\right) Therefore, ‖ A ℓ ​ ( 𝑿 ℓ , t ) − A ℓ ​ ( 𝑿 ℓ , t − 1 ) ‖ \displaystyle\big\|A_{\ell}({\bm{X}}_{\ell,t})-A_{\ell}({\bm{X}}_{\ell,t-1})\big\| = ‖ 1 d ​ ( Δ ℓ , t ​ 𝑴 ​ 𝑿 ℓ , t ⊤ + 𝑿 ℓ , t − 1 ​ 𝑴 ​ Δ ℓ , t ⊤ ) ‖ \displaystyle=\big\|\frac{1}{\sqrt{d}}\left(\Delta_{\ell,t}{\bm{M}}{\bm{X}}_{\ell,t}^{\top}+{\bm{X}}_{\ell,t-1}{\bm{M}}\Delta_{\ell,t}^{\top}\right)\big\| ≤ 1 d ​ ( ‖ Δ ℓ , t ​ 𝑴 ​ 𝑿 ℓ , t ⊤ ‖ + ‖ 𝑿 ℓ , t − 1 ​ 𝑴 ​ Δ ℓ , t ⊤ ‖ ) \displaystyle\leq\frac{1}{\sqrt{d}}\left(\big\|\Delta_{\ell,t}{\bm{M}}{\bm{X}}_{\ell,t}^{\top}\big\|+\big\|{\bm{X}}_{\ell,t-1}{\bm{M}}\Delta_{\ell,t}^{\top}\big\|\right) ≤ ‖ 𝑴 ‖ ​ ( ‖ 𝑿 ℓ , t ‖ + ‖ 𝑿 ℓ , t − 1 ‖ ) d ​ ‖ Δ ℓ , t ‖ \displaystyle\leq\frac{\big\|{\bm{M}}\big\|\left(\big\|{\bm{X}}_{\ell,t}\big\|+\big\|{\bm{X}}_{\ell,t-1}\big\|\right)}{\sqrt{d}}\big\|\Delta_{\ell,t}\big\| Assuming ‖ 𝑿 ℓ , t ‖ ≤ B \|{\bm{X}}_{\ell,t}\|\leq B for all t t , and defining κ ℓ = ‖ 𝑾 Q , ℓ ​ 𝑾 K , ℓ ⊤ ‖ = ‖ 𝑴 ‖ \kappa_{\ell}=\|{\bm{W}}_{Q,\ell}{\bm{W}}_{K,\ell}^{\top}\|=\big\|{\bm{M}}\big\| , we see ‖ A ℓ ​ ( 𝑿 ℓ , t ) − A ℓ ​ ( 𝑿 ℓ , t − 1 ) ‖ ≤ 2 ​ B ​ κ ℓ d ​ ‖ 𝑿 ℓ , t − 𝑿 ℓ , t − 1 ‖ \big\|A_{\ell}({\bm{X}}_{\ell,t})-A_{\ell}({\bm{X}}_{\ell,t-1})\big\|\leq\frac{2B\kappa_{\ell}}{\sqrt{d}}\big\|{\bm{X}}_{\ell,t}-{\bm{X}}_{\ell,t-1}\big\| (14) Combining Eq. 13 and Eq. 14 , we complete the proof: ‖ 𝒮 ℓ ​ ( 𝑿 ℓ , t ) − 𝒮 ℓ ​ ( 𝑿 ℓ , t − 1 ) ‖ ≤ L sm ​ 2 ​ B ​ κ ℓ d ​ ‖ 𝑿 ℓ , t − 𝑿 ℓ , t − 1 ‖ \big\|\mathcal{S}_{\ell}({\bm{X}}_{\ell,t})-\mathcal{S}_{\ell}({\bm{X}}_{\ell,t-1})\big\|\leq L_{\mathrm{sm}}\frac{2B\kappa_{\ell}}{\sqrt{d}}\big\|{\bm{X}}_{\ell,t}-{\bm{X}}_{\ell,t-1}\big\| (15) ∎

## Appendix B Additional Experimental Details

Unless stated otherwise, all of our experiments are averaged over the same subset of 256 random examples from the test split of the GSM8k ( Cobbe et al., 2021 ) dataset. A few illustrative plots (for example, latent space trajectories) are instead produced with a test sequence that we obtain from Barbero et al. (2025) : “Hello! I’ve been well. I hope that you’re doing well.” Additional results targetting non-reasoning behavior using the HellaSwag dataset (following an identical setup of running inference on the same 256 random examples from the test split) can be found in Sec. E.4 .

All pretrained models are obtained from Huggingface, model references provided in Table 2 . We use standard settings for the tokenizers of each model, and as such some models prepend a BOS token whereas others do not: we make this clear in the ‘Prepends BOS’ column of the same table.

Our small training runs in Sec. 5.1 are performed by adapting a publicly available fork of Nanochat ( Karpathy, 2025 ) , https://github.com/TrelisResearch/nanochat/tree/recursive . For all experiments we use a model dimension (residual stream) of 512 (4 heads of dimension 128) and train for 3.7B tokens. As discussed in the main text, loss is the same as that of a regular feedforward model: cross entropy loss on the final output representation (as opposed to the summed loss of Zhu et al. (2025) ). Each model is trained for a constant 4 recurrences (as opposed to the Poisson sampling of Geiping et al. (2025) ). All models use pre-norm only: this norm does not result in the stability issues reported by Geiping et al. (2025) ; Zhu et al. (2025) , but this is likely because we are operating at a far smaller scale.

## Appendix C Non-Fixed-Point Limiting Behavior

### C.1 How Frequent is Non-Fixed-Point Behavior?

In this section we investigate more closely the “orbits” and “sliders” initially observed by Geiping et al. (2025) . These are important as they appear to represent stable limiting behavior that are not fixed points .

We develop a heuristic algorithm to detect these behaviors, presented in Algorithm 1 . We use the sequence of cosine similarities from the final recurrent layer (per token), where the output residual stream after each of 128 recurrences is compared to the final residual stream (as in Fig. 22 ). This is visualized in the leftmost column of Fig. 13 . We set threshold τ = 0.05 \tau=0.05 and fixed-point fraction ρ = 0.9 \rho=0.9 .

Using this algorithm, we classify the limiting behavior over all tokens in the GSM8k test set for the Huginn-0125 and Retrofitted Llama models. We discover that the system prompt used before presenting the GSM8k question has a large impact on the limiting behavior types and as such we test the following prompts across both models:

#### Long Persona

This is the system prompt used by Geiping et al. (2025) to produce their Orbit plots:

#### Long Persona (Padded)

This is the same prompt as above but with all tokens replaced with the padding token: we include this to test whether the behavior arises purely due to the length of the input.

#### Short Math

This is the system prompt used by Geiping et al. (2025) in their GSM8k benchmark evaluation:

We present the percentage of GSM8k tokens that exhibit each classification of limiting behavior in Table 3 and the percentage of GSM8k examples that exhibit each classification of limiting behavior across any of their tokens in Table 4 .

These results reveal that these non-fixed-point limiting behaviors appear to be extremely rare in practice: without a system prompt (the setting used throughout this paper) only approximately 0.02% of tokens exhibit non-fixed-point behavior. This percentage can be significantly increased with the longer system prompt, but these behaviors remain rare at 0.14%. Curiously, the “Persona” system prompt used by Geiping et al. (2025) seems to increase the occurrence of these orbits and sliders more than a comparable prompt of padding tokens, suggesting that the effect is not purely to do with sequence length.