The Topological Trouble With Transformers — part 2/2

Cleaned-text code points [15261, 30071). Bibliography omitted here; source.md includes it.

Figure 5 b depicts a transformer unrolled in depth, sometimes referred to as a looped transformer . Depth recurrence—whether of individual layers or ranges of layers, and whether deterministic or adaptive—is a very popular and successful approach. Some methods are designed and trained to allow for inference time scaling ( Yang et al., 2024a ; Nowak et al., 2024 ; Raposo et al., 2024 ; Alabdulmohsin and Zhai, 2025 ; Bae et al., 2025 ; Chen et al., 2025a ; Geiping et al., 2025 ; Rodkin et al., 2025 ; Yu et al., 2025 ; Zhu et al., 2025 ; Zeng et al., 2026 ; Jeddi et al., 2026 , e.g.,) ; others incorporate recurrence via pretraining ( Sanyal, 2026 ) or fine tuning a pretrained model ( Koishekenov et al., 2025 ; McLeish et al., 2025 ) ; and surprisingly, several operate purely as an inference-time method to improve reasoning ( Li et al., 2025b ; Chen et al., 2026 ; Ng, 2026 ) .

While depth recurrence can increase the expressivity of a transformer ( Saunshi et al., 2025 ) , it does not enable indefinite state tracking; the propagation is still depth limited. To appreciate this fact, pick any layer l l in the lower or upper stack as the representation of s ⁡ ( 0 ) s(0) , the state at the first stack. Then note that any s ⁡ ( t + 1 ) s(t+1) , if it is to depend arbitrarily on s ⁡ ( t ) s(t) recursively, must be in a higher layer. The state representation still shifts upward due to the parallel propagation of activation across steps t t , regardless of how deep the transformer is made to be with recurrent depth (Figure 1 b).

Indefinite tracking of state with an arbitrary state update function requires sequential dependency that precludes parallelization across the sequence length during training . Two examples of autoregressive updates are presented in Figures 5 c,d. Figure 5 c depicts a blockwise-recurrent model ( Hutchins et al., 2022 ; Chevalier et al., 2023 ; Chen et al., 2025b ; Borazjanizadeh and McClelland, 2025 ) in which a subsequence of input steps is run in parallel (two in the Figure) followed by an autoregressive iteration; Figure 5 d shows a model in which one input step is presented per autoregressive step, and at step t t , all stacks up to t − 1 t-1 send a signal from the deep layer to the shallow layer, yielding a fully recurrent model. This model may have attractor dynamics since each layer continues to update, converging only when all previous steps have reached asymptote.

Having addressed recurrence with multiple input steps per autoregressive step (Figure 5 c) and a standard one-to-one mapping (Figure 5 d), we can also consider a setup where multiple autoregressive steps are executed for each input step (Figure 6 ). Latent-thought models have this form ( Hao et al., 2025 ; Jolicoeur-Martineau, 2025 , e.g.,) . Some of these models can track state, whereas others do not ( Galashov et al., 2025 , e.g.,) .

The examples of recurrence we have given all involve signal propagation from deeper layers to shallower layers, but within-layer propagation over inputs steps (Figure 7 ) also yields state dynamics ( Allen-Zhu, 2025 ; Gu and Dao, 2024 ; Fagnou et al., 2026 , e.g.,) .

Table 1 attempts to lay out a taxonomy of recurrent transformer architectures that includes the cases we have considered so far. We characterize architectures along two dimensions: recurrence axis and input tokens per recurrence step . The axis can be depth alone (Figures 5 b,c,d), in step alone (Figure 7 ), and in both depth and step (Figure 6 ). The ratio of input tokens to recurrence steps can be greater than one (Figure 5 b,c), equal to one (Figure 5 d, 7 ), or less than one (Figure 6 ). 3 3 3 We define a ‘recurrence step’ strictly as a sequential dependency that precludes parallelization across the sequence length during training. In cells of the taxonomy, we list some popular and representative transformer-based architectures. We omit architectures that are described as “recurrent” in the sense that processing is iterative, but that are neither recurrent in depth nor step, e.g., Transformer-XL ( Dai et al., 2019 ) . Being recurrent in depth and/or step is necessary for state tracking but is not sufficient. Essentially, full-fledged state tracking requires sequential dynamics during training; any model that can be entirely parallelized across the context has limitations in updating state.

While we have not shown all such architectures, we have not succeeded in identifying any examples of work that lies in the empty cells of the taxonomy. Some of these cells would be worth exploration. For example, architectures in the second row, third column, could include ones with within-layer attractor dynamics, where the model’s activation may iterate to convergence before advancing to the next token. Architectures in the first row, second and third columns, are interesting because they feed activation from a deep layer to a shallow layer (once for ratio = 1 \text{ratio}=1 , multiple times for ratio > 1 \text{ratio}>1 ) in a manner that—unlike the looped transformer—does allow for indefinite state propagation.

We have deliberately sidestepped the question of what a recurrent arrow signifies in terms of activation dynamics. Different models make different proposals. The arrow could indicate copying key-value cache ( Yang et al., 2024c ) , it could represent a source of keys and values for cross attention ( Fan et al., 2021 ) or self attention ( Oncescu et al., 2026 ) , or it might represent a direct connection through a linear layer, an MLP, or an adapter such as LoRA ( Hu et al., 2022 ) . The coupling of step-to-step recurrence with attention—which gives neurons direct visibility to the entire step history—prevents credit assignment bottlenecks that arise when training traditional recurrent neural networks ( Ke et al., 2018 ; Oncescu et al., 2026 ) .

## 4 Architectural limitations and workarounds

Liu et al. (2026) point to the weakness of modern massively parallel architectures on problems that are inherently sequential, problems where combinatorics make it impractical to parallelize, such as state tracking, multihop inference, and planning. Formal analyses point to bounds on serial capacity for transformers ( Merrill and Sabharwal, 2023 ) . State-space models (SSMs) are often touted as a means of state propagation, but SSMs with linear updates are no more expressive than an ordinary transformer ( Merrill et al., 2025 ) . In contrast, chain of thought does enhance model expressivity ( Li et al., 2024 ; Merrill and Sabharwal, 2024 ) , as one would intuit: Allowing a model to talk to itself, whether in natural language or latent space, sends signals from deep in the transformer to shallow layers, thereby propagating state forward. It is no wonder that frontier models have become increasingly reliant on internal ‘thinking’. However, the reliance on intermediate outputs to track micro-state may perform wasteful computation steps and unnecessarily consume the context window. Implicit activation dynamics—albeit recurrent—might be adequate to efficiently update mundane state information of the sort that people can process unconsciously and automatically. In human terms, it is fine to talk to yourself if you are reasoning through a calculus problem, but it is a bit weird to have to continually remind yourself of the relationship between two characters in a book you are reading.

Given their limitations in state tracking, why are transformers as successful as they are? The short answer is that by being able to reexamine their entire input history, they can often turn a state-tracking problem into a working memory problem, i.e., into retrieval from the context window. For example, consider the latch problem from the 1990s that required a sequence-processing model latch on to a bit of information early in a sequence and to retain it over a long time gap ( Mozer, 1991 ; Bengio et al., 1994 ) . This problem in large part motivated LSTM but is entirely trivial for a transformer that can re-index into its input history to retrieve the early information. Transformers learn many clever strategies including this sort of lookback ( Prakash et al., 2026 ) , associatve scans ( Li et al., 2025a ) , more specialized algorithms for formal language understanding ( Allen-Zhu and Li, 2025 ; Piotrowski et al., 2025 ) , including belief updating that reflects state uncertainty ( Shai et al., 2024 ) . A simple illustration is the computation of parity from an input sequence of 0 0 ’s and 1 1 ’s. Instead of maintaining a parity state, the model can compute pairwise parity in the first layer of a transformer, then combine the pairs into four-bit parity in the second layer, and so forth, achieving a computation from scratch in log 2 ⁡ n \log_{2}n feedforward layers for a sequence of maximum length n n . Such alternate and even shortcut solutions are common for transformers to develop in practice ( Liu et al., 2022 ) .

Another factor in the success of transformers is their support of state compositionality . State need not be—as we have depicted it in Figure 1 —a monolithic representation. The representation of state can be split across embeddings and updated asynchronously for each component. For example, if the model needs to track the changing locations of two entities, those state variables can be updated independently for each entity.

## 5 Promising directions

In addition to the native mechanisms that transformers have to estimate state, several emerging research directions seem particularly promising for promoting state maintenance and updating. These directions balance the need for expressivity with computational feasibility.

### 5.1 Enhanced State-Space Models

While most linear SSMs do not exceed the expressivity of standard transformers, Delta Net ( Schlag et al., 2021 ) , when its eigenvalue range is extended to include negative values ( Grazzi et al., 2025 ) , maintains the highly desirable property of being trainable in parallel while simultaneously achieving greater expressivity than a standard transformer. The delta rule underlying Delta Net has inspired a range of further developments, including RWKV-7 ( Peng et al., 2025 ) and PaTH attention ( Yang et al., 2025b ) , that also achieve state tracking beyond the capability of ordinary transformers while demonstrating competitive language modeling at scale. Similarly, other new forms of attention are being developed with sequential dependencies that promise to be more powerful in stateful tasks ( Beltagy et al., 2020 ; Fagnou et al., 2024 ; Lin et al., 2025 ; Leviathan et al., 2025 ; Fagnou et al., 2026 ) , including the notion of gated linear attention ( Yang et al., 2024b ) and gated Delta Net ( Yang et al., 2025a ) that, when mixed with standard transformer blocks is more powerful than either, both in theory and practice ( Merrill et al., 2026 ) .

### 5.2 Approximating state tracking in feedforward transformers

Rather than incorporating recurrence, feedforward transformers can be steered to better approximate state tracking through specialized training objectives and structural priors ( Hu et al., 2025 ; Teoh et al., 2025b ; Huang et al., 2026 ) . The promise is that such biases will encourage models to bolster their native lookback abilities. However, we hope that future research will take into account structured, compositional state representations.

### 5.3 Coarse recurrence

Introducing recurrence at a coarser granularity than individual tokens can mitigate the computational burden of token-by-token state updates. In the past, block-recurrent models have operated on this principle by compressing and passing memory forward in fixed- length chunks. A promising approach is to consider linguistic structure in identifying the chunks: Borazjanizadeh and McClelland (2025) have modeled language as a sequence of discrete ‘thoughts’ by chunking at the sentence level.

### 5.4 Leveraging representational alignment

Variable-depth models dynamically select the number of iterations of a layer or the number of times a range of layers repeats. The fact that this approach succeeds with only fine tuning—and sometimes with no training whatsoever—suggests that the model is well predisposed to communication of representations across layers, due to the alignment resulting from the residual connections. Similarly, canon layers ( Allen-Zhu and Li, 2025 ) leverage alignment of representations from one input step to the next. We speculate that there are additional means of leveraging this alignment.

### 5.5 Efficient training of recurrence

Any architecture which is capable of indefinite, arbitrary state propagation requires autoregressive processing. Autoregressive pretraining is computationally inefficient and limits parallelization ( Chevalier et al., 2023 ) . One potential solution is a multi-stage training scheme in which initial pretraining relies entirely on standard, parallelizable feedforward transformer architectures. Recurrent mechanisms are then introduced only at a later training stage. To ensure training efficiency during these subsequent recurrent stages, optimization techniques such as truncated gradient methods and—for training attractor dynamics—recurrent backpropagation ( Almeida, 1987 ; Pineda, 1987 ; Liao et al., 2018 ) . Methods have also been proposed to increase arithmetic intensity to obtain scaling that is near linear time in context length versus quadratic for a naive implementation ( Oncescu et al., 2026 ) .

## 6 Conclusions

Although the transformer’s feedforward design has expanded the limits of context-based retrieval, its topological structure remains fundamentally at odds with the iterative nature of state tracking. As we have argued, the current reliance on explicit natural-language-like “thought” to bypass depth limitations is an inefficient workaround for a structural deficiency. By transitioning toward implicit, recurrent activation dynamics, we can move beyond these depth-limited constraints to attain robust long-term coherence and multihop inference.

The taxonomy and research directions proposed in this article provide a roadmap for improving sequential inference dependencies without sacrificing the foundational strengths of modern models. Ultimately, bridging the gap between the transformer’s parallel efficiency and the brain’s inherent dynamical nature is essential. The next generation of foundation models must do more than simply re-scan the past; they must maintain a fluid, evolving representation of reality that persists across the many time scales required for temporally extended cognition.

## Acknowledgments and Disclosure of Funding

Many thanks to Sunny Sanyal, Kazuki Irie, Kevin Murphy, Jay McClelland, and Chris Williams for helpful feedback on earlier drafts of the manuscript.