The Topological Trouble With Transformers — part 1/2
Cleaned-text code points [0, 15261). Bibliography omitted here; source.md includes it.
# The Topological Trouble With Transformers
###### Abstract
Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking—the iterative updating of latent variables reflecting an evolving environment—involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model’s depth. While this depth limit can be bypassed by dynamic depth models and by explicit or latent thinking that externalizes state representations, these solutions are computationally and memory inefficient. In this article, we argue that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures. We introduce a taxonomy of recurrent and continuous-thought transformer architectures, categorizing them by their recurrence axis (depth versus step) and their ratio of input tokens to recurrence steps. Finally, we outline promising research directions, including enhanced state-space models and coarse-grained recurrence, to better integrate state tracking into modern foundation models.
## 1 Introduction
Progress in understanding human cognition has resulted from conceptualizing the brain as a dynamical system. In terms of its hardware, the physical brain is composed of billions of interacting neurons whose collective behavior is inherently dynamical. In terms of its function, the emergent mind can be usefully modeled as a dynamical process with a high-dimensional state, s s , that evolves over time, modulated by external stimuli, x x . These levels can be bridged by formalizing the state progression as s t = f ( s t − 1 , x t ) s_{t}=f(s_{t-1},x_{t}) , assuming discrete time t t .
From this perspective, an ideal architecture for modeling temporally extended cognition would be a recurrent neural network (RNN), which explicitly performs such a state-update operation. In principle, gradient-based training procedures might discover the function f f from data such that the important input signals would be integrated into the state representation and held until later required. The appeal of RNNs was somewhat dampened in the 1990s by the inherent limitations of gradient-based training ( Mozer, 1992 ; Hochreiter, 1998 ; Hochreiter et al., 2001 ) .
Until the transformer ( Vaswani et al., 2017 ) came along, feedforward nets did not seem like a viable approach to replicating human thought and reasoning. The transformer, with an audaciously long context window, retains all information in its history, often postponing the selection of relevant data until required for inference ( Meng et al., 2022 ) . In contrast, RNNs filter information as it arrives into a bottlenecked state representation ( Hochreiter and Schmidhuber, 1997 ) . This article is about what can go wrong with the transformer’s strategy and approaches that can address its limitations.
## 2 State tracking
Tracking the evolving world state is an essential ingredient for language understanding and reasoning, regardless of how tracking is achieved. The transformer’s strategy often leverages its capacity to retrieve static, previously observed information from a context window. The attention mechanism of transformers is highly effective at retrieving past tokens ( Olsson et al., 2022 ) . However, this lookup mechanism is conceptually distinct from the explicit maintenance of a dynamic state—the iterative, inherently sequential updating of latent variables that reflect a changing environment. The term belief state is often used to refer to this compact, sufficient summary of the knowledge an AI agent has about its environment ( Chrisman, 1992 ; Kaelbling et al., 1998 ) . Belief state can be a set of facts or it can be a probability distribution over possible worlds.
To illustrate, consider the game of twenty questions. Each answer to a question narrows the hypothesis space, and each subsequent question should be designed to shrink the hypothesis space further. In a game where one is asked to guess a number, if one is told that the number is larger than 50, it would make no sense to follow up it with a guess of 25. And on the flip side, if one is asked to think of a number between 1 and 100 and respond ‘higher’ or ‘lower’ to guesses, maintaining the state---the hidden number---is critical to preventing inconsistent answers. Yet, here is a trace from Gemini 3 (Fast) revealing a failure mode of models: 1 1 1 This example was originally suggested by Gamal ElSayed circa 2020, still causing some modern foundation models to fail. Other models may have been trained to address this particular failure mode. Baldelli et al. (2026) formally address the failure of standard models to reliably maintain a consistent hidden state over sequential interactions.
Even though the model cannot generate a random number internally, it can play the game without actually having a number in mind simply by choosing responses consistent with its previous responses. However, consistency requires tracking the valid range. Gemini 3 Thinking does generate an explicit target, but then it fails to make use of the generated target even though it is also a part of the input token stream in this case:
Beyond games, state tracking is essential to understanding the ever-changing world, the structure of arguments, and social interactions. State tracking failures in foundation models lead to loss of coherence in multi-turn conversations ( Laban et al., 2025 ) , inefficiency in information gathering ( Sawyer et al., 2025 ) , and breakdowns in communication and cooperation in multi-agent settings ( Davidson et al., 2025 ; Khatua et al., 2026 ) .
Without proper state tracking, models flip-flop in their interpretations and fail to detect their inconsistencies, e.g., the meaning of a polysemous word ( Lepori et al., 2025 ) : 2 2 2 The example that follows was produced by Gemini 2.5 Flash in 2025. Although the model sometimes responded correctly, and newer and more powerful models are much less susceptible to this error, the example reveals a fundamental limitation of the core architecture.
In this example, the model jumps from one interpretation of bank to the other without a human-like acknowledgment of the reversal (e.g., “Oh wait, I misinterpreted. You must have been talking about a financial institution.”). We consider this flip-flop a failure to track the world situation, the model’s earlier responses, and the listener’s expectations.
While tracking Fred’s location in the above example should not be challenging, in the most general case, it is untenable for models—and people—to maintain and track probabilistic belief states over all environmental possibilities because the distributions explode in dimensionality. People adopt heuristics such as sampling ( Vul et al., 2014 , e.g.,) , collapsing distributions into prototypical cases ( Tversky and Kahneman, 1971 ) , or forming concrete mental models most consistent with premises ( Johnson-Laird, 1983 ) , like a MAP estimate. Nonetheless, even finite-memory, deterministic state tracking can be unreliable in a transformer decoder.
We will use a schematic to explain the challenge of state tracking. In Figure 1 a, we depict the transformer with input steps shown along the horizontal axis and blocks (or layers) of the transformer along the vertical axis. Activation propagates from bottom to top (shallow to deep layers). For three selected blocks, we use color to indicate the functional connectivity of a causal transformer: activation in a block is influenced by all blocks immediately below and below to the left. In Figure 1 b, we depict the flow of state information, indicated by the green rectangles. The integration of the state representation and a new input (purple arrows) leads to a new state representation, s t = f ( s t − 1 , x t ) s_{t}=f(s_{t-1},x_{t}) . Because the architecture is feedforward, s t s_{t} must lie deeper in the stack than s t − 1 s_{t-1} , eventually topping out of the model. Figure 2 shows examples from several recent articles matching this upward activation flow. This flow can make the integration of information over the sequence unreliable ( Biran et al., 2024 ; Grant et al., 2025 ; Lepori et al., 2025 ; Sawyer et al., 2025 ; Venhoff et al., 2025 ; Baldelli et al., 2026 ) .
Not every state-tracking problem requires depth linear in the number of layers. Merrill and Sabharwal (2025) prove the necessity and sufficiency of log n \log n layers to recognize regular language strings of up to length n n and graph-connectivity problems with n n vertices. However, this proof addresses only the constructability of solutions, not their learnability. Fagnou et al. (2024) prove log n + 1 \log n+1 layers are required for an entity tracking task with n n state changes and obtain empirical results consistent with this theory.
In practice, many researchers have identified clever solutions obtained by training depth-limited models on specific finite sequence-length problems ( Li et al., 2025a ; Piotrowski et al., 2025 ; Prakash et al., 2026 ; Shai et al., 2024 ) . Essentially, when the state update function, f f is of a certain form, the sequence of state updates may be composed into a simpler one-step function, e.g., there exists a function g g computable by a transformer layer such that s t = f ( … f ( f ( s 0 , x 1 ) , x 2 ) , … , x t ) = g ( s 0 , x 1 , … , x t ) . s_{t}=f(\dots f(f(s_{0},x_{1}),x_{2}),\dots,x_{t})=g(s_{0},x_{1},\dots,x_{t}). Training losses have been proposed that aim to steer models toward such solutions, to the extent they exist exactly or approximately ( Hu et al., 2025 ; Teoh et al., 2025b ; Huang et al., 2026 ) . However, when belief-state cascades push deeper and deeper into a network, computational limitations arise because the resulting representations are unavailable to shallower layers. To illustrate, Figure 3 depicts the processing of the bank dialog. Using a technique called Patchscopes ( Ghandeharioun et al., 2024 ) , Lepori et al. (2025) observed that the embedding of the polysemous word ‘bank’ was ambiguous (i.e., a mixture of money bank vs. river bank) at shallow layers of the network, but deep in the network, the model selected the river bank interpretation. The previous sentences provided context (e.g., ‘took the day off work’, ’fishing pole’) to support one interpretation over the other. In the Figure, we depict this contextualization as occurring at the sixth block of the transformer stack, which means that when subsequent tokens are processed, the disambiguation is not available in blocks 1-5. Lepori et al. (2025) show that this delayed disambiguation leads to downstream errors whenever the response generation outpaces the model’s internal semantic convergence, such as when the model forms its yes/no response to the question about an ATM at the bank.
If reasonable models make errors when the inference cascade is just two steps ( Lepori et al. studied Gemma2-9B), it should be no surprise that models produce more severe failures in comprehending extended multi-agent conversations.
One solution to the depth dilemma is chain-of-thought style “thinking” where the model recasts a deep representation as one or more output tokens, which are then available to the model on its input ( Wei et al., 2022 ) . For ordinary, step-by-step microcognition, this solution is a cop out. Inferences that people make automatically and unconsciously and then utilize consistently, such as the selection of a polysemous word’s meaning, should not require elaborated, extended cognition. Even those who disagree with this desideratum should agree that if cognition in a transformer can be shifted from explicit thought traces to implicit activation dynamics, the resulting model will be more powerful.
## 3 Recurrent architectures
As we previously mentioned, the state tracking ability of a feedforward model is limited by model depth ( Merrill and Sabharwal, 2023 ; Strobl et al., 2024 ; Merrill and Sabharwal, 2025 , e.g.,) and by the fact that effectively utilizing the state representation becomes more challenging as it shifts upwards to deeper layers ( Biran et al., 2024 ; Lepori et al., 2025 ; Sawyer et al., 2025 ; Venhoff et al., 2025 ) .
The alternative is a recurrent model, which is necessary to express arbitrary state dynamics, i.e., s t = f ( s t − 1 , x t ) s_{t}=f(s_{t-1},x_{t}) . In this section, we explore how to combine recurrence with transformers. Given the subtle yet critical differences among varieties of recurrence, they are easily conflated and mistakenly treated as equivalent. This equivalence is problematic because recurrence is not in itself sufficient for state tracking; the most popular form of recurrence is unable to track state ( Merrill et al., 2025 ) .
Figure 4 , adapted from Rumelhart et al. (1986) , shows a simple recurrent net (left) and the net unrolled t t steps to form a weight-constrained feedforward net (right).
Unrolling the net requires making a copy of all neurons in the network for each step of recurrence. At each step, activation flows through each connection.
Now consider a transformer with some recurrent connections, as depicted in Figure 5 a. As before, each box denotes a transformer block and the horizontal and vertical axes correspond to input tokens and layers, respectively. The additional arrows are meant to illustrate one possible type of recurrence involving activation flow from a deep layer to a shallow layer at every input step. The nature of the activation flow is not important for our purposes; for example, the shallow layer may generate queries that allow it to cross attend to keys and values from the deep layer ( Fan et al., 2021 , e.g.,) .
Unlike the simple RNN in Figure 4 , which can be unrolled in only one way, unrolling the recurrent transformer in Figure 5 a results in ambiguity due to the fact that that the transformer architecture incorporates three distinct ordered dimensions: (1) the layers of the architecture, bottom to top (shallow to deep) in the Figure; (2) the input steps of the architecture, left to right in the Figure; and (3) autoregressive steps performed at execution. During model pretraining, an ordinary (feedforward) transformer has only a single autoregressive step because all input steps are run in parallel; and during inference, autoregressive steps are typically confounded with input steps, although multi-token prediction allows for multiple input/output steps per autoregressive step ( Gloeckle et al., 2024 ) . However, recurrence allows for further decoupling. In particular, many forms of recurrence require that even when a model is trained via teacher forcing, it must still be unrolled autoregressively ( Teoh et al., 2025a ) . This necessary sequentiality is what we mean by autoregressive unrolling , not the sequentiality that arises from token-by-token generation in a pure feedforward model.