Recirculation — part 2/5
Cleaned-text code points [15187, 30095). Bibliography omitted here; source.md includes it.
Looped transformers. Looped transformers run single layers or a range of layers multiple times, obtaining a deeper architecture with no addition in free parameters ( Dehghani et al., 2019 ; Giannou et al., 2023 ) . Looping, which can be deterministic or adaptive, is a very popular and successful approach. Looping can increase the expressivity of a transformer ( Saunshi et al., 2025 ) , but it does not guarantee indefinite state tracking. Some methods are designed and trained to allow for inference time scaling ( Yang et al., 2024 ; Nowak et al., 2024 ; Raposo et al., 2024 ; Alabdulmohsin and Zhai, 2025 ; Bae et al., 2025 ; Chen et al., 2025a ; Geiping et al., 2025 ; Rodkin et al., 2025 ; Yu et al., 2025 ; Zhu et al., 2025 ; Zeng et al., 2026 ; Jeddi et al., 2026 , e.g.,) ; others incorporate recurrence via pretraining ( Sanyal, 2026 ) or fine tuning a pretrained model ( Koishekenov et al., 2025 ; McLeish et al., 2025 ) ; and surprisingly, several operate purely as an inference-time method to improve reasoning ( Li et al., 2025b ; Chen et al., 2026 ; Ng, 2026 ) . The inference-time methods have the greatest similarity to recirculation, although the notion of looping (greater depth) is conceptually distinct from the recurrence that occurs in recirculation (see Figures 3 and 4 ).
Training objectives. Training losses have been proposed that aim to make embeddings in a given layer of the transformer more stateful, i.e., interpretable in terms of state updating functions. Particular losses have been proposed to steer models toward such solutions, to the extent they exist exactly or approximately ( Hu et al., 2025 ; Teoh et al., 2025 ; Huang et al., 2026 ) .
State tracking. Liu et al. (2026) point to the weakness of modern massively parallel architectures on problems that are inherently sequential, problems where combinatorics make it impractical to parallelize, such as state tracking, multihop inference, and planning. Transformers are bounded in their serial capacity based on model depth ( Merrill and Sabharwal, 2023 ; Strobl et al., 2024 ; Merrill and Sabharwal, 2025 , e.g.,) and also by the fact that effectively utilizing the state representation becomes more challenging as it shifts upwards to deeper layers ( Biran et al., 2024 ; Lepori et al., 2025 ; Mozer et al., 2026 ; Sawyer et al., 2025 ; Venhoff et al., 2025 ) . Merrill and Sabharwal (2025) prove the necessity and sufficiency of log n \log n layers to recognize regular language strings of up to length n n and graph-connectivity problems with n n vertices. However, this proof addresses only the constructability of solutions, not their learnability. In practice, many researchers have identified clever solutions obtained by training depth-limited models on specific finite sequence-length problems ( Li et al., 2025a ; Piotrowski et al., 2025 ; Prakash et al., 2026 ; Shai et al., 2024 ) .
Recurrent transformers. Models with sequential recurrent updates can express arbitrary state dynamics, z t = f ( z t − 1 , x t ) z_{t}=f(z_{t-1},x_{t}) . Some of these models operate with token-by-token recurrence ( Fan et al., 2021 , e.g.,) , but most operate with blockwise recurrence ( Bulatov et al., 2022 ; Hutchins et al., 2022 ; Chevalier et al., 2023 ; Jabri et al., 2023 ; Chen et al., 2025b ; Borazjanizadeh and McClelland, 2026 ) . State-space models (SSMs) ( Katharopoulos et al., 2020 ; Schlag et al., 2021 ; Gu and Dao, 2024 ; Allen-Zhu, 2025 ; Yang et al., 2025 ; Peng et al., 2025 ; Sun et al., 2025 ; Siems et al., 2025 , e.g.,) are often touted as a means of state propagation, but many SSMs are no more expressive than an ordinary transformer ( Merrill et al., 2025 ) and none are as expressive as a generic recurrent net. Most closely related to our work is T 2 MLR \text{T}^{2}\text{MLR} ( Cai et al., 2026 ) , which also recognizes the value of feeding intermediate representations from a deeper layer of the previous token position directly into a shallower layer of the current token position. However, the two approaches diverge substantially in their training paradigm and architectural mechanisms. While T 2 MLR \text{T}^{2}\text{MLR} requires BPTT training or fine tuning, our formulation of recirculation is an inference-time, training-free intervention. Further, T 2 MLR \text{T}^{2}\text{MLR} parameterizes the recurrence pathway with a dedicated, learnable gated fusion module, whereas recirculation relies directly on the intrinsic representation alignment of the Transformer’s residual stream.
Thinking models. One solution to the depth dilemma is chain-of-thought style “thinking” where a model can talk to itself by sequentially sending signals from deep in the transformer to shallow layers, thereby propagating state forward. This form of recurrence enhances model expressivity ( Li et al., 2024 ; Merrill and Sabharwal, 2024 ) . Thinking can be performed in natural language tokens or in latent space ( Hao et al., 2025 ; Jolicoeur-Martineau, 2025 , e.g.,) . As with other recurrent transformers, training thinking models can be costly because it restricts parallelism.
Activation steering. Recent work in activation steering demonstrates that a language model’s behavior can be predictably modulated by intervening on its latent representations ( Dathathri et al., 2020 ; Turner et al., 2023 ; Zou et al., 2023 ; Gao et al., 2025 ) . This representation space encodes complex behavioral directions, including those governing truthfulness and refusal ( Marks and Tegmark, 2023 ; Arditi et al., 2024 ) . In this context, recirculation can be viewed as an inference-time mechanism for self-steering. Rather than modifying the residual stream with a static, externally derived steering vector ( Rimsky et al., 2024 ) , recirculation leverages the model’s own contextualized deep-layer activations to guide representation trajectories in shallower layers.
## 4 Experiments
### 4.1 Hyperparameter sweeps
To explore the feasibility of recirculation, we begin by sweeping over the three hyperparameters of recirculation: the mixture coefficient α \alpha (with β = 1 − α \beta=1-\alpha unless mentioned otherwise), the source layer s s , and the destination layer d d . We incorporate recirculation into the Gemma3 1B PT (pretrained) model and compute perplexity for documents from the arXiv dataset. (Details of the simulation can be found in Appendix B.1 .) The four heatmaps of Figure 5 correspond to α ∈ { 0.04 , 0.07 , 0.10 , 0.16 } \alpha\in\{0.04,0.07,0.10,0.16\} ; the vertical and horizontal axes indicate the source and destination layers. We examine all source-destination pairs that are no more than 12 layers apart. The heatmap is coded blue-to-red to indicate perplexity lower-to-higher than a baseline no-recirculation Gemma3 1B model, which has perplexity 16.6 on this dataset. Notably, these heatmaps are fairly smooth and reveal systematic patterns. Increasing α \alpha amplifies the effect of recirculation but results in more source-destination pairs that harm perplexity. Layer 4 is desirable as a destination with a source 5-7 layers higher. For the moment, focus on the pattern of results and not on the magnitude of reduction in perplexity.
The top row of Figure 6 , shows perplexity for three datasets—arXiv, PG19, and C4—sweeping over source and destination layers and fixing α = 0.10 \alpha=0.10 . We transform absolute perplexity estimates into percentage change relative to the baseline model and average across the three datasets to get the boxed sweep showing percentage change. Blue indicates a reduction in perplexity, with the best source-destination pair obtaining an average reduction of 4.72%. Similarly to the 1B model, we identify the source and destination layers for the 4B and 12B models that yield the largest mean percentage perplexity reduction on the roughly 1.5M tokens in our tuning set. For the 1B, 4B, and 12B Gemma3 models, we found the optimal source and destination pairs based on our tuning set to be: {11, 4}, {18, 9}, and {35, 16}, respectively.
### 4.2 Perplexity evaluation
Having chosen source and destination layer based on our grid search results, we evaluate perplexity reduction on ten language modeling datasets with α = 0.15 \alpha=0.15 . Our evaluations include the three datasets used to select hyperparameters—arXiv, C4, and PG19—but the evaluation split is distinct from ‘training’ split used for hyperparameter tuning. Appendix B.2 presents details of the evaluation procedure, which included the Gemma3 1B, 4B, and 12B PT models.
Table 1 presents results for the three model sizes and ten data sets. For each model size, columns indicate perplexity of the baseline and recirculation models, and the percentage reduction in perplexity by incorporating recirculation. The 1B and 4B models obtain reductions up to 16% and the 12B model up to 35% 3 3 3 Note that the 12B model is relatively weak in terms of language modeling, but strong in downstream tasks post instruction-tuning as highlighted by its performance on standard benchmarks ( Gemma Team et al., 2024 ) . . For nine of the ten data sets, we see robust improvements across model scales. The lambada set is an anomaly due to the presence of very short sequences and tokenization artifacts. We show later that recirculation has greater benefits for longer sequences.
### 4.3 Normalization and ramping
Because the magnitude of residual-stream embeddings tends to grow over layers, we have found that renormalizing the source embedding before feeding it to the destination (Equations 1 and 2 ) better conditions the model to obtain a consistent and reliable pattern of perplexity reduction over the hyperparameter sweep (Figures 5 and 6 ). Appendix B.3 presents a range of normalization schemes we considered. The normalization scheme does not have much impact on the maximum perplexity reduction, only on the robustness of improvements over the source-destination landscape, which increases our confidence in being able to identify the optimal source-destination pair. The Appendix shows sweeps for multiple renormalization schemes, including an identity mapping (no renormalization). The Appendix also addresses the Gemma3 4B and 12B models, which turn out to critically require a non-convex mixture with β = 1 \beta=1 instead of β = 1 − α \beta=1-\alpha .
In Section 9 , we report results indicating that recirculating tokens at the beginning of the Gemma3 1B context window can be harmful. This finding does not seem surprising given there is little state information to propagate at the start of the window; and without a benefit from recirculation, the cost of potentially pushing the internal representations out of distribution may overwhelm. However, we find no harm of early-token recirculation in the 4B and 12B models. Nonetheless, we introduce ramping in the 1B model to attenuate α \alpha for the first tokens (see Appendix B.3 ), which yields a small reduction in perplexity.
### 4.4 Analysis of recirculation
#### 4.4.1 Robustness across model families
In Figure 7 , we observe that four other model families—Ministral3, Pythia, Qwen3, and Phi2—all show a robust region in the source-destination heatmap where reductions in perplexity are observed, analogous to what we observe with Gemma3 1B. The models are roughly the same size and all benefit most from recirculation in the middle region of the architecture.
The range of layers in this heatmap spans the full range of layers, which explains the difference in heatmap shape 4 4 4 Note that this Gemma3 1B heatmap is a full independent replication of Figure 5 with a different implementation, written in PyTorch and HuggingFace eco-system instead of JAX, but shares the data loader to ensure matched input. . The plot also includes recirculation to the output of layer 0 (leftmost column), whose representations apparently have not been contextualized to the point that deeper layers can interpret the recirculated signal.
The percentage reduction in perplexity is significantly larger for the Gemma model family than for the other families—about 5% versus less than 0.5%. As we explain in Appendix B.2 , the difference is due to an artifact specific to the Gemma family. When this artifact is eliminated, the Gemma family shows roughly the same sized perplexity reduction as other model families, which we will highlight in a revised version of this manuscript. Nonetheless, the basic purpose of the perplexity evaluation is to identify training-free routes to incorporating recurrence; the magnitude of the effect—averaged over all tokens—is not nearly as important as the qualitative pattern observed in the sweeps.
#### 4.4.2 Recirculation versus temperature tuning
Because perplexity is affected by the sharpness of a softmax distribution, we wanted to rule out the possibility that recirculation is merely sharpening or smoothing the distribution in an undifferentiated manner. We evaluated perplexity across a range of softmax temperatures and indeed found that with the base Gemma3 1B model, a temperature of 1.2 (versus the default of 1.0) reduced perplexity by 8.48% (see left panel of Figure C.2 ). Recirculation alone reduces perplexity by 14.21%, and therefore recirculation must not be reducible to temperature adjustment. When we combine recirculation with temperature tuning, we obtain a reduction in perplexity of 19.55%, also with temperature 1.2. The fact that the two effects are nearly additive allows us to rule out the artifactual explanation for recirculation in terms of token-independent temperature tuning.
#### 4.4.3 Recirculation versus looping
To verify that recirculation is influencing model dynamics in a structurally different manner than looped transformers ( Dehghani et al., 2019 ; Giannou et al., 2023 ; Alabdulmohsin and Zhai, 2025 ) , we compare recirculation and looped transformers. As we did with recirculation, we sweep over all ( ℓ 1 , ℓ 2 ) (\ell_{1},\ell_{2}) layer pairs with ℓ 2 > ℓ 1 \ell_{2}>\ell_{1} . To implement looping, we inserted into the transformer stack a copy of the layers from ℓ 1 + 1 \ell_{1}+1 up to and including ℓ 2 \ell_{2} immediately following the original ℓ 2 \ell_{2} in the stack. To be comparable to recirculation, we perform training-free evaluation. Although the literature indicates that models have been improved with training-free looping ( Li et al., 2025b ; Chen et al., 2026 ; Ng, 2026 ) , the heatmaps in Figure 8 indicate that, for the Gemma3 family at least, looping does not produce robust benefits. Furthermore, while recirculation shows benefits across model scales, looping a pretrained model shows benefits only at larger model scales, as suggested by our plots and observed by Ng (2026) . These qualitative differences in the heatmaps confirm that looping and recirculation are operating on different principles.
#### 4.4.4 Which tokens benefit from recirculation?