Recirculation — part 1/5

Cleaned-text code points [0, 15187). Bibliography omitted here; source.md includes it.

# Recirculation

###### Abstract

We describe an inference-time architectural enhancement for off-the-shelf foundation models that markedly reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase. Motivated by the fundamental limitation that state updates in feedforward transformers are bounded by model depth, our technique, recirculation , introduces a specific form of recurrence that allows the model to act as a dynamical system and track belief states. We distinguish this technique from chain-of-thought computation—which is better reserved for complex inferences rather than basic state tracking—as well as from popular depth-recurrence techniques (looping) and the costly training of recurrent transformers. We also propose and evaluate an adaptive variant of recirculation which requires only light tuning of hyperparameters while freezing the original model weights. Relative to the off-the-shelf baseline, adaptive recirculation achieves remarkable gains on the Gemma3 family, including a systematic reduction in perplexity on a suite of datasets, a 21% increase in accuracy on GSM8k, and reliable improvements in accuracy on other downstream tasks. Our training-free approach succeeds by leveraging the model itself to inform architectural modifications, suggesting a route to architectural evolution guided by a trained network’s properties rather than forced, arbitrary design choices.

## 1 State tracking

The ability to track a fluid, evolving state of affairs is essential for comprehending language, reasoning about situations, and modeling the world around us. Traditional approaches to state tracking, such as recurrent neural nets and Kalman filters, involve iterative, sequential updating of latent variables to capture dynamics. The parallel operation of transformers, during both training and prefill, precludes tracking state in the traditional manner ( Mozer et al., 2026 ) . Transformers learn a variety of clever solutions that can make them effective at state tracking for finite sequence lengths ( Li et al., 2025a ; Piotrowski et al., 2025 ; Prakash et al., 2026 ; Shai et al., 2024 ) . However, transformers can also fail to track state information needed to understand the structure of arguments and social interactions. State tracking failures in foundation models lead to loss of coherence in multi-turn conversations ( Laban et al., 2025 ) , inefficiency in information gathering ( Sawyer et al., 2025 ) , and breakdowns in communication and cooperation in multi-agent settings ( Davidson et al., 2025 ; Khatua et al., 2026 ) . Tracking deficits can lead to a range of reasoning failures arising from ineffective working memory, unstable theory-of-mind representations, and absence of a robust internal world model ( Song et al., 2026 ) .

Without proper state tracking, models can flip-flop in their interpretations and fail to detect their inconsistencies, e.g., the meaning of a polysemous word in Figure 1 ( Lepori et al., 2025 ) , known as a contextualization error. While it is perfectly sensible for a model to maintain a belief state that admits both interpretations of the word bank , after committing to the edge-of-a-river interpretation ( river bank ), a socially adept communicator should acknowledge the reversal of meaning.

In general, it becomes untenable for models—and people—to maintain and track belief states because the distributions explode in dimensionality. People adopt heuristics such as sampling ( Vul et al., 2014 , e.g.,) , collapsing distributions into prototypical cases ( Tversky and Kahneman, 1971 ) , or forming concrete mental models that are most consistent with premises ( Johnson-Laird, 1983 ) , kind of like a MAP estimate. All of these approaches nonetheless require state tracking, albeit without explicit representation of uncertainty.

Figure 2 a illustrates the challenge of state tracking in a transformer using the bank dialog of Figure 1 . Input steps are presented along the horizontal axis and blocks (or layers) of the transformer along the vertical axis. Each column represents the processing of an input token, where activation flows from bottom to top. The processing of the text day off work and fishing pole is depicted in columns on the left. Using a technique called Patchscopes ( Ghandeharioun et al., 2024 ) , Lepori et al. (2025) show that when the polysemous word bank is processed, shallow layers reflect the word’s ambiguity: the embedding is a mixture of its meanings—river edge and financial institution. In deeper layers of the stack (highlighted in yellow), the embedding is contextualized by the preceding text (depicted by the blue arrows), and the context-appropriate interpretation of the polysemous word is selected by nudging the embedding toward the river-edge representation. Because the architecture is feedforward, shallow layers of the transformer cannot access this interpretation. Thus, when the model is asked to formulate a response to the ATM question, the initial processing stages see only the ambiguous representation of bank . If the model commits to a response (highlighted yellow) using the ambiguous representation (blue arrows), it will choose the wrong response. Despite having previously interpreted bank as the river edge, it fails to access this state and responds yes based on a strong association between the ambiguous bank and ATM. Lepori et al. (2025) characterize this delayed disambiguation as a race in which the model’s response generation can outpace the model’s internal semantic convergence.

Lepori et al. (2025) obtained evidence in support of this account via an intervention experiment, illustrated in Figure 2 b. They processed the input sequence in the ordinary way up to the step at which the bank token is presented. At this step, they copied activation from a deep layer---after the ambiguity was resolved---down to a shallow layer and then continued processing the input sequence. This intervention, depicted in the Figure with an arrow, reduced contextualization errors by 60%. 1 1 1 The full experiment was based on a large set of generated questions containing a critical token that needs to be properly contextualized in order to answer correctly, with activation patching on only the critical token. Nikankin et al. (2025) have also shown that patching visual token activations back to a shallower layer in a vision-language model improves performances.

In this experiment, Lepori et al. (2025) replaced the embedding vector in a shallow layer with the corresponding vector from a deep layer, but only for one specific token pre-identified as critical in the given context. Suppose that instead of targeting this manipulation to a specific token, we did so in an undifferentiated fashion at every token position? Likely the outcome would be disastrous because the model was not trained to accommodate the resulting amplification of feedback. But what if instead of substituting one embedding vector for another, we merely leaked a small bit of activation from the deep layer to the shallow layer? The leakage might conceivably be sufficient to enrich the representation without pushing representations out of distribution. Consequently, we might observe benefits of this manipulation at inference without modifying the weights of a fully trained model .

This scheme might work even without fine tuning because a transformer’s residual stream acts like a shared blackboard onto which all layers can write, which encourages alignment of representations across layers ( Elhage et al., 2021 ) . By alignment, we mean that for a deep layer to communicate with a shallow layer, a 1:1 correspondence of features can be assumed and we do not require an arbitrary adapter such as a full-rank affine transformation or cross attention.

Why should there be alignment? Consider a particular feature of the embedding vector in the residual stream, and suppose that it is associated with a semantically meaningful concept such as moisture. Whether the feature is activated early or late in the stack, the direct effect on the output distribution is identical due to commutativity of addition. Some input tokens, e.g., pool or tears , may activate the moisture feature in the input embedding directly. An ambiguous token such as bank may not yield much intrinsic activation for moisture due to polysemy. However, if we feed back the disambiguated bank from a deep layer, the moisture feature will be amplified, providing useful information for subsequent processing. 2 2 2 This argument does not deny that the layer of origin of a feature may change the feature’s role, and in fact, the argument for leaking activation downwards is to leverage the changing roles. Layer normalization is one way that roles might depend on depth. However, while layer normalization may change a feature’s magnitude and even polarity, it does not reorient the feature. One might also be concerned about feature erasing: a feature may be ‘uncomputed’ once it is no longer needed. This concern is alleviated by recent work showing that transformers learn just as well, if not better, when each layer is forced to output orthogonal directions from the earlier layers, preventing feature erasing ( Oh et al., 2026 ; Zhang et al., 2026 ) .

## 2 Proposed method: Recirculation

We now formalize our method, which we refer to as recirculation , that involves running an LLM step-by-step and after each step, leaking a bit of the activation from a deep layer down to a shallow layer. Figure 3 a gives the general picture, where the recurrent arrow specifies one possible source- and destination-layer pair. Because this Figure does not specify how the recurrence is orchestrated with regard to input steps, the figure is ambiguous and could correspond to several distinct ideas ( Mozer et al., 2026 ) , one of which is recirculation and another of which is a very popular idea in the literature, the looped transformer ( Dehghani et al., 2019 ; Giannou et al., 2023 ) . For didactic purposes, we first discuss looped transformers and then characterize their relationship to recirculation.

A looped transformer is a parameter-efficient variant of the standard architecture. Whereas a standard transformer stacks a deep sequence of unique transformer blocks, a looped transformer applies a set of shared blocks multiple times. Figure 3 b depicts a looped transformer by unrolling the architecture of Figure 3 a vertically in depth and unrolled horizontally in input steps. At step 1, the first input token is presented, the activation stack is computed, and at layer 6 in the Figure (the loop source ), activation is passed to layer 3 (the loop destination ) of a second copy of the architecture, which then propagates activity to the output. The colored rectangles denote the current input token, the dark outlined rectangles are blocks whose activation is computed at a given step, the shaded blocks are frozen (or are replaced by KV cache), and the faint outlined rectangles are placeholders that are irrelevant at the current step. As each input step is processed, a single stack operates and the earlier stacks are frozen (or in KV cache). The Figure shows step-by-step operation of the model, as would be used with autoregressive decoding. However, when an input sequence is fixed—as during pretraining or in the prefill stage—the entire sequence can be computed in parallel.

Recurrence in a looped transformer is solely in depth: the arrows in Figure 3 b are directed within a stack. In contrast, recurrence in recirculation is in both depth and step , as depicted in Figure 3 c. Figure 3 c can be viewed as collapsing together the operation of Figure 3 b’s top stack at step i i and the bottom stack of step i + 1 i+1 . In recirculation, two input stacks are run in parallel at each recurrence step (except for the very first step, which serves as a warm up). The difference between Figures 3 b and 3 c looks to be a minor reorganization, but it fundamentally changes the nature of the computation when it comes to arbitrary state tracking. Figure 4 shows the same unrolled models but superimposes colored rectangles to indicate state propagation. To implement an arbitrary state updating function over time t t , z ⁡ ( t + 1 ) = f ⁡ ( z ⁡ ( t ) , x ⁡ ( t ) ) z(t+1)=f(z(t),x(t)) , where z z is state and x x is input, z ⁡ ( t + 1 ) z(t+1) must be one layer deeper in the looped transformer, which is just a deeper feedforward transformer with weight-sharing constraints. However, in recirculation, where there is projection in both depth and step, the same layer can hold both z ⁡ ( t ) z(t) and z ⁡ ( t + 1 ) z(t+1) . The cost of state tracking is that sequential passes must be made through the architecture, and because of this sequential updating, recirculation cannot be parallelized, even when an entire input sequence is provided, such as during prefill.

In Figures 3 and 4 , the model is unrolled to execute the transformer stack twice at each step, i.e., one more iteration than a standard transformer. This number of iterations can be increased, both for looping and recirculation. We depict two-iteration recirculation in Figure A.1 . However, all experiments reported in this article are with the one-additional-iteration variant. If the number of iterations is unbounded, recirculation behaves as a true recurrent neural net.

We note one other key difference between looping and recirculation. In looping, the activation that gets looped is the entire residual stream and it acts as the direct replacement for the input that would normally come from a preceding layer. In recirculation, activation is mixed from the source and destination layers. Initially, we formalize one-iteration recirculation as a mixture: 𝒛 t + 1 , t , d = α ​ f ​ ( 𝒛 t , t , s | d , t ) + β ​ 𝒛 t , t , d , {\boldsymbol{z}}_{t+1,t,d}=\alpha f({\boldsymbol{z}}_{t,t,s}|d,t)+\beta{\boldsymbol{z}}_{t,t,d}, (1) where t t is an index over the order of updates, s s and d d are the source and destination layer indices, respectively, α \alpha and β \beta are mixture coefficients, f ( . ) f(.) is a renormalization function, and 𝒛 𝒊 , 𝒋 , 𝒍 \boldsymbol{z_{i,j,l}} is the residual stream output after incorporating the computation of layer l l at unrolling step i i and input step j j . These three indices correspond to the three sequence dimensions depicted in Figure 3 c: l l and j j are the row and column indices of the transformer layer grid, respectively, and i i is an index over copies of architecture. The motivation for the renormalization function f f is to accommodate the possibility that embedding magnitudes grow as the layer outputs are combined. We always use a convex mixture with β ≡ 1 − α \beta\equiv 1-\alpha and rescale source to have the same L 2 L_{2} norm as the destination unless mentioned otherwise: f ⁡ ( 𝒛 | d , t ) = ‖ 𝒛 t , t , d ‖ 2 ‖ 𝒛 ‖ 2 ​ 𝒛 . f({\boldsymbol{z}}|d,t)=\frac{||{\boldsymbol{z}}_{t,t,d}||_{2}}{||{\boldsymbol{z}}||_{2}}~{\boldsymbol{z}}. (2)

## 3 Related research