Recirculation — part 4/5

Cleaned-text code points [46034, 61757). Bibliography omitted here; source.md includes it.

Although there is a robust literature on architectural improvements to the transformer, this literature focuses almost exclusively on training models from scratch or incorporating modifications mid-training. The notable exception is the notion of training-free looping ( Li et al., 2025b ; Chen et al., 2026 ; Ng, 2026 ) , a related but distinct architectural modification. Both looping and recirculation exploit a key property of the transformer’s residual pathway: the alignment of representations across layers ( Elhage et al., 2021 ) . Recirculation is complementary to and can be combined with other methods in common practice, including looping, variable computation time, and the coarser-grain recurrence that occurs with both latent thought and chain of thought.

### 5.1 Model-design affordances

We view recirculation not as a ‘shovel ready’ technique to be incorporated into state-of-the-art models, but rather as a methodological or philosophical contribution. By focusing on training-free architectural modifications, we are essentially asking the model to reveal to us its ingrained paths for improvement. Our search over where to place recurrence in the architecture and how to mix and normalize activations is informed by the model itself. The alternative—the typical course of research in machine learning—is to propose an arbitrary modification to an architecture and then evaluate the potential of this modification via costly training. Our key hypothesis is that if we identify how the pretrained model wants us to tweak it without modifying its weights, we also identify an inductive bias that will facilitate and simplify training, whether from scratch or with fine tuning. Our adaptive recirculation experiments support this hypothesis.

In product design, practitioners talk about design affordances ( Norman, 1999 ) , which are properties of an object that inform us about how it should be used. For example, a door handle affords grasping and pulling; a door plate affords pressing. Analogously, our research explores model-design affordances —the natural properties of a foundation model that we can exploit and amplify to improve the model’s basic operation. By improving this basic substrate, we improve all capabilities that build on it. Most approaches to incorporating recurrence within the internal layers of a foundation model do so via adapters or cross attention. Our observations suggest that the simpler strategy of mixing activation vectors may suffice.

### 5.2 Limitations and future directions

We close by discussing limitations of the present work and promising future directions that will address these limitations.

• Determination of optimal hyperparameters. The optimal model hyperparameters—the source and destination layers and the recirculation coefficients α \alpha and β \beta —appear to be domain or problem dependent. However, hyperparameters selected based on one dataset or one criterion often generalize to another. For example, hyperparameters based on perplexity evaluations were used for downstream tasks; hyperparameters based on one specific dataset (MMLU) were used for a range of other single-token-response tasks; and small sets of examples are often sufficient to select hyperparameters. However, unless we can identify task-universal hyperparameters or an automatic mapping from task to hyperparameters, the practical applicability of recirculation will be limited.

• Dependence on model family. While recirculation shows the same qualitative pattern across model families, it is possible that Gemma’s Peri-LN architecture and training optimization process yield larger benefits than other model families (Appendix C.1 ). Generalizability to other architectures requires further investigation.

• Normalization. We find that normalization of source activation vectors makes recirculation behave more robustly. For the Gemma3 family, we find that of the many candidate normalization schemes we investigated, one in particular seems to work well across model sizes (Section B.3 ). Nonetheless, further experiments are needed to determine whether normalization is necessary and how to normalize for other model families.

• Blockwise recurrence. Recirculation is computationally cheap during autoregressive generation because running two stacks in parallel is nearly as efficient with modern hardware as one stack. However, during prefill, token-by-token processing is also required. For state-of-the-art foundation models, which may include a large amount of contextually relevant information, sequential processing of the prefill may be infeasible. One way to alleviate the computational burden would be to perform blockwise recirculation, i.e., to process a set of K K tokens in parallel, and then recirculate those K K tokens simultaneously with the next K K as they are processed for the first pass. We have yet to explore the trade off between K K and recirculation performance.

• Multiple recirculation iterations. A true recurrent network would recirculate the activation in stacks 1 to t − 1 t-1 at step t t . In contrast, the variant of recirculation we explored recirculates only stack t − 1 t-1 , such that each stack is recirculated exactly one time. We have yet to study an intermediate approach in which each stack is recirculated exactly r r times. Figure A.1 shows an unrolled architecture with r = 2 r=2 iterations per stack. We wished to start with the simplest instantiation, but given the low cost of recirculating r r stacks in parallel, increasing the number of iterations is worthwhile.

• Multiple recirculation paths. We picked a single {source, destination} pair for recirculation, but one could recirculate along multiple paths simultaneously. Indeed, for some architectures (e.g., see Gemma3 4B and 12B recirculation sweeps in Figure 8 ), there appeared to be distinct, disjoint regions of the hyperparameter sweep where recirculation was effective. One hypothesis is that these distinct regions are complementary, e.g., they may convey state information at different levels of abstraction.

• Adaptation. The present work has only scratched the surface of possible approaches to adapting models to enhance recirculation effects. Our results with adaptive recirculation are extremely promising considering that the tuning set we used was quite small and that tuning of model weights was not required to obtain robust performance improvements. Before turning to model-weight fine tuning, many avenues for improvement still exist, e.g., token conditioned recirculation pathways and normalization methods.

By listening to the model’s own internal dynamics rather than forcing costly architectural overhauls, recirculation offers a powerful, computationally inexpensive path forward. Unlocking these intrinsic affordances may solidify the model’s basic contextual understanding, which in turn provides the vital scaffolding required for extended, multi-turn reasoning.

## 6 Acknowledgements

Our thanks to Lei Bao, Mukul Bhutani, and Lan Le who contributed to the shaping of this research. We are also grateful to Michael Riegler and Klas Pettersen (SimulaMet, Norway), Patrick Haller (Humbolt U., Berlin), and Jonathan Yanay for their independent replications and constructive discussions of our results.

## Appendix A Unrolled recirculation architecture

Figure A.1 shows an unrolled transformer with two iterations of recirculation. In general, with k k iterations of recirculation, it will be necessary to run k + 1 k+1 stacks for each input step.

## Appendix B Recirculation implementation details

### B.1 Hyperparameter sweeps

For the hyperparameter sweeps (Figures 5 and 6 ), we used a context window of 1024 tokens and pulled roughly 500 windows from training documents in three datasets: arXiv, C4, and PG19. We used at most two windows from each document, requiring that the windows had no filler tokens (i.e., the document extended at least to the end of the window). This requirement yielded 484 windows for arXiv (495132 predicted tokens), 488 windows for C4 (499224 tokens), and 500 windows for PG19 (511000 tokens).

### B.2 Perplexity evaluation

For our perplexity evaluation (Table 1 ), we used the entire evaluation set from nine data sets (arXiv, billsum, booksum/books, C4/webtextlike, gov report, lambada, newsroom, PG19, and pubmed) and the first 10000 documents from a tenth data set (big patent). The evaluation split was labeled ‘validation’ for C4 and ‘test’ for all other datasets. We partitioned each document into chunks of 1024 tokens and excluded partially filled windows (i.e., less than 1024 tokens), except for three data sets whose documents were typically too short (c4/webtextlike, lambada, and newsroom). This procedure resulted in the evaluation of the number of tokens listed in the second column of Table 1 .

Based on hyperparameter sweeps for the 1B, 4B, and 12B models, we used the source and destination pairs shown in Table B.1 . These hyperparameters were used for all results presented in the article unless otherwise mentioned.

### B.3 Normalization and ramping

Figures B.1 - B.3 show hyperparameter sweeps for multiple renormalization techniques, i.e., scaling of the source vector when recirculating to the destination layer. Scaling helps to ensure that the source activations do not overwhelm the destination activations, due to the fact that embedding norms increase over transformer layers ( Liu et al., 2020 ; Xiong et al., 2020 ) . The heatmaps depict percentage reduction (negative) or increase (red) in perplexity over our three tuning datasets (arXiv, C4, PG-19).

The normalization schemes are summarized in the Figures and in Table B.2 . We simplify the notation used in the main paper, where 𝒛 i , j , l \boldsymbol{z}_{i,j,l} referred to the embedding at unrolling step i i for input step j j and layer l l . Instead, we denote: d ≡ \displaystyle d\equiv~ 𝒛 t , t , d , \displaystyle\boldsymbol{z}_{t,t,d}, s ≡ \displaystyle s\equiv~ 𝒛 t , t , s ​ , and \displaystyle\boldsymbol{z}_{t,t,s}\text{, and} d ′ ≡ \displaystyle d^{\prime}\equiv~ 𝒛 t + 1 , t , d . \displaystyle\boldsymbol{z}_{t+1,t,d}. The normalization schemes in Table B.2 include the simple scheme in which no normalization is applied. The no-normalization heatmaps (top row, middle panel in Figures B.1 - B.3 ) attain reasonable outcomes but clearly the schemes with L 2 L_{2} normalization (top row, right panel and second row, left panel) are better behaved in the sense that there are fewer hyperparameters that result in poorer performance. Oddly, a convex combination of source and destination vectors (top row, right panel) is superior to a nonconvex combination (second row, left panel) for the Gemma3 1B model, but the nonconvex combination is superior for Gemma3 4B and 12B. Other candidate schemes either had pathologies (non-smooth heatmaps, large red regions) or mimicked the simple norm-ratio schemes. Based on the Gemma family, we recommend evaluating at least the convex and non-convex mixtures with norm-ratio adjustment of the source.

Ramping. As we mentioned in the main article, for the Gemma3 1B model, we found a small additional reduction in perplexity if we ramped up the recirculation coefficient over the first 10 steps. Specifically, we defined the α \alpha coefficient at step t ≥ 0 t\geq 0 to be α t ≡ min ⁡ ( t / 10 , 1 ) ​ α \alpha_{t}\equiv\min(t/10,1)\alpha .

## Appendix C Basic results

### C.1 Robustness across architectures

In Figure 7 , we contrasted recirculation hyperparameter sweeps for Gemma3 1B to four other model families: Ministral3, Qwen3, Pythia, and Phi2. The Figure compares architectures using only the arXiv training set. Due to the fact that all of these models were relatively small, we utilized the same hyperparameters as we did for Gemma3 1B PT: scaling of the source layer norm to match the target layer norm, α = .07 \alpha=.07 and β = 1 − α \beta=1-\alpha . Using β = 1 \beta=1 —the choice for Gemma3 4B and 12B—produced qualitatively similar results.

In addition to the comparison across different model families, we also compared results from Gemma3 with those from other generations of the Gemma model family, including the older Gemma2 and the newer Gemma4 (Figure C.1 ). The arXiv train set is again used in this Figure. The gains for Gemma2 and Gemma4 are as pronounced as those for Gemma3 in the main text, despite the fact that there are regions of the hyperparameter space where recirculation is quite harmful. Note that smaller variants of Gemma4 (i.e., E2B and E4B) uses cross-layer KV cache sharing as well as per-layer embedding, which can explain the instability in these plots ( Team et al., 2026 ) .

We hypothesize that this favorability of Gemma architectures can be attributed to two distinct explanations: • Gemma architectures use a Peri-LN architecture ( Kim et al., 2025 ) which adds layer-norm to both the input of the layer (i.e., self-attention or MLP), as well as the output of the layer. In contrast, other architectures only use input normalization ( Xiong et al., 2020 ) . The use of output normalization might improve the model’s compatibility with recirculation by avoiding diminishing contribution from later layers ( Sun et al., 2026 ) .

• The compatibility with recirculation is strongly dependent on the optimization process used for training the model. Hence, the scheme used to train Gemma models is particularly favorable for recirculation.

### C.2 Recirculation versus temperature tuning

Gemma3 1B with the PG-19 evaluation (‘test’) set was used for temperature tuning experiments. The documents were split into 1024 token chunks per context window; partially filled sequences were excluded. We swept over a wide enough range of softmax temperatures to identify the range in which adjusting model temperature improved perplexity. Figure C.2 shows the sweep without recirculation in the left panel and with recirculation in the right panel. Whether with or without recirculation, the optimal temperature was about 1.2. Although we performed a more granular sweep for the combined experiment, we report in the main paper the optimum over only the range of temperatures used for the temperature-alone experiment.

### C.3 Recirculation versus looping

The dataset is comprised of 250 documents from the arxiv train set, with 2 subsequences taken from each document starting from the beginning of the document, each of size 1024 tokens. This is identical to our setting for the grid search results in Figure 5 . Results for recirculation are shown with α = 0.07 \alpha=0.07 , with convex combination ( β = 0.93 \beta=0.93 ) for the 1B model and a non-convex combination ( β = 1.0 \beta=1.0 ) for the 4B and 12B models, as discussed in the main text.

### C.4 Which tokens benefit from recirculation?

Experiments in the main paper are based on Gemma3 1B PT with the arXiv data set, training split, with a context window of 1024 tokens extracted from a randomly selected position within each document. Part-of-speech tags are extracted with nltk.pos_tag . We processed 24960 documents and for each, we recirculated tokens 0-767 individually and examined downstream effects at lags 1-256.

We also ran an experiment in which we recirculated all and only tokens tagged with a given part of speech (Figure C.3 ). For this experiment, 3120 documents from the arXiv train set were used, and from each, a sequence of 1024 tokens was taken from a random starting position within the document. Each of these sequences was tested with recirculation of only PoS-selected tokens.

## Appendix D Downstream evaluations

### D.1 Instruction following

We generated 800 queries of the following form: ⬇