A Mechanistic Analysis of Looped Reasoning Language Models β€” part 1/4

Cleaned-text code points [0, 15342). Bibliography omitted here; source.md includes it.

# A Mechanistic Analysis of Looped Reasoning Language Models

###### Abstract

Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM’s layers in the latent dimension, resulting in looped reasoning language models . Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design.

###### Keywords:

\printArXivAffiliationsAndNotice

## 1 Introduction

The vast majority of current LLMs are based on the Transformer architecture ( Vaswani et al., 2017 ) , which comprises a sequence of blocks traversed in a feedforward manner to predict the next token. As the capability of these models increased, attention turned to eliciting reasoning capabilities in LLMs by increasing test-time computation, commonly through chain-of-thought (CoT) prompting ( Wei et al., 2022 ) or reinforcement-learning-based fine-tuning, first popularized in the DeepSeek-R1 architecture ( Guo et al., 2025 ) . More recently, research has explored building reasoning capabilities directly into the model architecture via recurrent looping ( Geiping et al., 2025 ; Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) , where additional test-time compute is spent by taking more recurrent steps, echoing early designs in this direction ( Graves, 2016 ) . Despite growing empirical success, the mechanisms underlying these models remain poorly understood, as well as their benefits and limitations when compared to feedforward computation.

In this paper, we compare how feedforward and looped LLMs organize computation across (effective) depth through the lens of stages of inference ( Lad et al., 2024 ; Queipo-de-Llano et al., 2025 ) , a perspective suggesting that LLM inference can be decomposed into several distinct computational stages. Building on prior observations that repeated application of a shared recurrent block can approach a fixed point or steady state ( Yang et al., 2023 ; Geiping et al., 2025 ) , we show that such behavior necessarily implies one of two possibilities: either the contribution of the component Transformer blocks vanishes asymptotically, or their sequential application traces out a constant cyclic trajectory in latent space. We further demonstrate empirically that the latter behavior arises in practice when certain architectural conditions are met, and that this appears to be emergent behavior from the Transformer architecture itself, appearing in both trained recurrent models and untrained, randomly initialized models.

## 2 Preliminaries

### 2.1 Looped Transformers

In this section we introduce Looped Transformers and define the notation that we will use throughout the paper. We represent the input sequence to our transformer of length T T and dimension D D as 𝑿 ∈ ℝ T Γ— D {\bm{X}}\in\mathbb{R}^{T\times D} . Following the notation of Yudin et al. (2025) we define Attn \attn , the dot product self-attention mechanism, as a map f : ℝ T Γ— D β†’ ℝ T Γ— D f:\mathbb{R}^{T\times D}\to\mathbb{R}^{T\times D} : Attn ⁑ ( 𝐗 ) \displaystyle\attn({\bm{X}}) = softmax ⁑ ( 𝑿 ​ 𝑾 Q ​ 𝑾 K ⊀ ​ 𝑿 ⊀ d ) ​ 𝑿 ​ 𝑾 V , \displaystyle=\mathrm{softmax}{\left(\frac{{\bm{X}}{\bm{W}}_{Q}{\bm{W}}_{K}^{\top}{\bm{X}}^{\top}}{\sqrt{d}}\right)}{\bm{X}}{\bm{W}}_{V}, (1) = softmax ⁑ ( A ⁑ ( 𝑿 ) ) ​ 𝑿 ​ 𝑾 V , \displaystyle=\mathrm{softmax}{\left(A({\bm{X}})\right)}{\bm{X}}{\bm{W}}_{V}, (2) where 𝑾 Q , 𝑾 K , 𝑾 V ∈ ℝ D Γ— d {\bm{W}}_{Q},{\bm{W}}_{K},{\bm{W}}_{V}\in\mathbb{R}^{D\times d} are projection matrices and A A is defined for convenience.

A transformer block typically comprises an attention mechanism and a position-wise MLP as follows: 𝑿 ^ \displaystyle\hat{{\bm{X}}} = n 2 ​ ( 𝑿 + Attn ⁑ ( n 1 ​ ( 𝐗 ) ) CLOSE , \displaystyle=n_{2}\left({\bm{X}}+\attn(n_{1}({\bm{X}})\right), (3) 𝑿 β€² \displaystyle{{\bm{X}}}^{\prime} = n 4 ​ ( 𝑿 ^ + MLP ​ ( n 3 ​ ( 𝑿 ^ ) ) ) , \displaystyle=n_{4}\left(\hat{{\bm{X}}}+\text{MLP}(n_{3}(\hat{{\bm{X}}}))\right), (4) where n 1 , n 2 , n 3 , n 4 n_{1},n_{2},n_{3},n_{4} are each optional norms – here we are borrowing from the notation of Geiping et al. (2025) . We denote the action of a Transformer block B : ℝ T Γ— D β†’ ℝ T Γ— D \block:\mathbb{R}^{T\times D}\to\mathbb{R}^{T\times D} as 𝑿 β€² = B ⁑ ( 𝑿 ) {\bm{X}}^{\prime}=\block({\bm{X}}) , and refer to the intermediate hidden-state matrices 𝑿 {\bm{X}} between blocks as the residual stream .

Looped Transformers are Transformers that utilize β€œrecurrence in depth” – that is, they reapply layers to repeatedly act on the latent states. Recent research has identified that an effective way ( Bae et al., 2025 ) to achieve this is via cyclic recurrence : a fixed sequence of layers is repeated in a β€œcyclic” pattern. This is the approach that we focus on in this work, and we introduce it in more detail below. For convenience, we define a k k -stacked block as a composition of Transformer blocks, S k ⁑ ( 𝑿 ) = B k ⁑ ( B k βˆ’ 1 ⁑ ( … ​ B 1 ⁑ ( 𝑿 ) ​ … ) ) \stack_{k}({\bm{X}})=\block_{k}(\block_{k-1}(\dots\block_{1}({\bm{X}})\dots)) .

In the case of Geiping et al. (2025) ; McLeish et al. (2025) this stacked block may also take an additional input 𝒁 ∈ ℝ T Γ— D {\bm{Z}}\in\mathbb{R}^{T\times D} which is typically initialized from a Normal distribution, as well as the original input to the recurrent section: this is known as input injection ( Bai et al., 2019 ; Anil et al., 2022 ) , and the two inputs are projected into common feature space ℝ D \mathbb{R}^{D} before the block is applied. In the case of input injection, a k k -stacked block therefore becomes S k ⁑ ( 𝑿 , 𝒁 ) = B k ⁑ ( B k βˆ’ 1 ⁑ ( … ​ B 1 ⁑ ( [ 𝑿 , 𝒁 ] ​ 𝑾 I ) ​ … ) ) , \stack_{k}({\bm{X}},{\bm{Z}})=\block_{k}(\block_{k-1}(\dots\block_{1}([{\bm{X}},{\bm{Z}}]{\bm{W}}_{I})\dots)), (5) where [ β‹… , β‹… ] [\cdot,\cdot] denotes concatenation in the channel dimension and 𝑾 I ∈ ℝ 2 ​ D Γ— D {\bm{W}}_{I}\in\mathbb{R}^{2D\times D} is a learned projection matrix.

This allows us to define a ( k βŠ— l ) (k\otimes l) -Recurrent block as a k k -stacked block repeated l l times: R l , k ( 𝑿 ) = S k ( S k ( … S k ( ⏞ Γ— l 𝑿 ) … ) ) , R_{l,k}({\bm{X}})=\overbrace{\stack_{k}(\stack_{k}(\dots\stack_{k}(}^{\times l}{\bm{X}})\dots)), (6)

which with input-injection becomes R l , k ​ ( 𝑿 , 𝒁 ) = OPEN S k ⁑ ( 𝑿 , S k ⁑ ( … ​ 𝑿 , S k ⁑ ( 𝑿 , 𝒁 ) ) ​ … ) ) ⏞ Γ— l . R_{l,k}({\bm{X}},{\bm{Z}})=\overbrace{\stack_{k}({\bm{X}},\stack_{k}(\dots{\bm{X}},\stack_{k}({\bm{X}},{\bm{Z}}))\dots))}^{\times l}. (7)

Note that the input 𝑿 {\bm{X}} is only β€œinjected” at the start of each stack of blocks – once per recurrence. Additionally, a complete looped Transformer may have multiple feedforward layers before the Recurrent block, and multiple feedforward layers after the Recurrent block: following the convention of Geiping et al. (2025) , we refer to these as prelude and coda layers respectively, and these are simply separate stacked blocks with non-tied layer weights. Where prelude and coda are used, we will frequently refer to this as a sandwich block structure.

We combine and adapt the nomenclature of Geiping et al. (2025) ; Saunshi et al. (2025) and refer to a looped Transformer with p p prelude layers, k k recurrent layers and c c coda layers with the tuple ( p , k , c ) (p,k,c) . Where input injection is used, we add an I I subscript ( p , k , c ) I (p,k,c)_{I} , and when referring to a recurrent layer looped a specific number of times l l , we denote this as ( p , k βŠ— l , c ) (p,k\otimes l,c) .

In summary, a ( p , k βŠ— l , c ) (p,k\otimes l,c) looped Transformer is defined as 𝑿 0 \displaystyle{\bm{X}}_{0} ← S p ⁑ ( 𝑿 ) \displaystyle\leftarrow\stack_{p}({\bm{X}}) 𝑿 i \displaystyle{\bm{X}}_{i} ← S k β€² ⁑ ( 𝑿 i βˆ’ 1 ) i ∈ { 1 , … , l } \displaystyle\leftarrow\stack_{k}^{\prime}({\bm{X}}_{i-1})\qquad i\in\{1,\dots,l\} 𝑿 \displaystyle{\bm{X}} ← S c β€²β€² ⁑ ( 𝑿 l ) , \displaystyle\leftarrow\stack_{c}^{\prime\prime}({\bm{X}}_{l}), where β€² \prime and β€²β€² \prime\prime indicates that these are different stacks between which weights are not shared. A ( p , k βŠ— l , c ) I (p,k\otimes l,c)_{I} looped Transformer (with input injection) is defined as 𝑿 \displaystyle{\bm{X}} ← S p ⁑ ( 𝑿 ) \displaystyle\leftarrow\stack_{p}({\bm{X}}) 𝒁 i \displaystyle{\bm{Z}}_{i} ← S k β€² ⁑ ( 𝑿 , 𝒁 i βˆ’ 1 ) i ∈ { 1 , … , l } \displaystyle\leftarrow\stack_{k}^{\prime}({\bm{X}},{\bm{Z}}_{i-1})\qquad i\in\{1,\dots,l\} 𝑿 \displaystyle{\bm{X}} ← S c β€²β€² ⁑ ( 𝒁 l ) , \displaystyle\leftarrow\stack_{c}^{\prime\prime}({\bm{Z}}_{l}), where 𝒁 0 {\bm{Z}}_{0} is initialized such that each column is sampled from 𝒩 ⁑ ( 𝟎 , Οƒ 2 ​ 𝕀 D ) \mathcal{N}(\bm{0},\sigma^{2}\mathbb{I}_{D}) .

We will describe our results via grouping : β€’ No grouping : The value is visualized as it evolves through sequential layers of the model, irrespective of whether these layers are repeated.

β€’ By recurrence : Separate lines are visualized for each complete pass through the recurrent block. The x x -axis is typically percentage scaled to represent relative depth within that block (including prelude/coda), allowing us to overlay and compare successive passes. Always presented with a green - yellow colorbar, with later recurrences colored more yellow.

β€’ By layer : Separate lines are visualized for each unique layer, showing how the value evolves across recurrences. Always presented with a blue - green colorbar, with later layers colored more green.

### 2.2 Stages of Inference

The behavior of layers in feedforward Transformers appears to change sharply with depth: Lad et al. (2024) originate the term β€œstages of inference” and demonstrate how several different layer mechanisms emerge at different depths. Queipo-de-Llano et al. (2025) further develop this viewpoint, focusing on behaviors that can be characterized by the mixing (or lack thereof) induced by the attention heads. We focus on this latter perspective. Mixing in this context refers to the extent to which the attention mechanism incorporates information from previous tokens at each layer. Throughout the main text of this paper we quantify our study of mixing behavior through the ColSum Concentration metric introduced in Queipo-de-Llano et al. (2025) ; we introduce and discuss additional metrics in App. E .

We first define the column sum c j = βˆ‘ i A i ​ j c_{j}=\sum_{i}A_{ij} to capture how much attention mass is received by token j j . We normalize this as c ^ j = c j / T \hat{c}_{j}=c_{j}/T to obtain a probability distribution, noting that βˆ‘ i , j A i ​ j = T \sum_{i,j}A_{ij}=T since A A is row-stochastic. From this distribution, we define the ColSum Concentration via its normalized entropy as C = 1 βˆ’ H col ∈ [ 0 , 1 ] = 1 + 1 log ⁑ T ​ βˆ‘ j c j ​ log ⁑ c j C=1-H_{\text{col}}\in[0,1]=1+\frac{1}{\log T}\sum_{j}c_{j}\log c_{j} .

Large values of C C indicate a high concentration of attention mass: few columns receive most of the mass. We note therefore that this metric captures the well-studied attention sink ( Xiao et al., 2023 ; Barbero et al., 2025 ) behavior, but generalizes to capture concentration over any token position. This property is useful for our investigation as not all models studied herein exhibit sinks on the first token; in particular OLMo-2 frequently concentrates attention mass on punctuation, echoing a result in Sandoval-Segura et al. (2025) .

## 3 Related Work

#### Looped and Recurrent Transformers

Reusing the same Transformer block for multiple iterations is an idea that has been explored in the literature. This began with the introduction of Universal Transformers ( Dehghani et al., 2018 ) , which have also resulted in sparsified and conditional-computation extensions ( Tan et al., 2023 ; CsordΓ‘s et al., 2024 ) . Other more recent recurrent style architectures with a higher focus on reasoning-style tasks have been HRM ( Wang et al., 2025 ) and TRM ( Jolicoeur-Martineau, 2025 ) . Within language modeling, we highlight Huginn-0125 ( Geiping et al., 2025 ) , Ouro ( Zhu et al., 2025 ) , and Mixture-of-Recursions ( Bae et al., 2025 ) as models that have been pretrained from random initialization, as well as recent work by ( McLeish et al., 2025 ; Koishekenov et al., 2025 ) that retrofit recurrence into pretrained LLMs.

In terms of mechanistic studies, we highlight Pappone et al. (2025) , who analyze β€œtwo-scale” latent dynamics in recurrent Transformers. However, the setting of their analysis is different from ours: they analyse a model in which each recurrent block comprises either 1 or 2 layers, and the model comprises multiple separate recurrent blocks. In this way, the two scales they refer to correspond to the outputs of each iteration of a given recurrent block, and outputs of recurrent blocks when transitioning between different recurrent blocks. We instead study a single looped block with deeper cycles (4+ layers), closer to common looped architectures, and we analyze the internals of these cyclic blocks by examining the latent states of each separate layer. We also highlight work related to looped model expressivity ( Xu and Sato, 2024 ; Saunshi et al., 2025 ) , as well as work on neural network stability and fixed-point dynamics ( Bai et al., 2019 ; Anil et al., 2022 ; Ke et al., 2024 ; Yudin et al., 2025 ) . The only work – of which we are aware – that analyses the internal states of the cyclic recurrent blocks is Lu et al. (2025) , who demonstrate cyclic behavior in logit lens prediction throughout recurrent blocks.

#### Stages of Inference and Attention Dynamics

The idea that LLMs organize their feedforward computation into several distinct stages of inference was first proposed by Lad et al. (2024) . Building on this, Queipo-de-Llano et al. (2025) explain the emergence of these stages through the behavior of attention heads, driven by massive activations ( Sun et al., 2024 ) . We also highlight a complementary line of work that analyzes LLM learning dynamics via attention patterns through the lens of mixing ( Barbero et al., 2024 ; Arroyo et al., 2026 ; VeličkoviΔ‡ et al., 2024 ; Barbero et al., 2025 ) , motivated by information propagation challenges originally studied in Graph Neural Networks (GNNs) ( Cai and Wang, 2020 ; Alon and Yahav, 2020 ; Arroyo et al., 2025 ; Hariri et al., 2025 ; Blayney et al., 2025 ) .

#### Test-time Computation