Recirculation — part 3/5

Cleaned-text code points [30095, 46034). Bibliography omitted here; source.md includes it.

In all experiments reported to this point, tokens in every position of the context window are recirculated. Now we ask whether we can identify which tokens most contribute to recirculation performance improvements. Using a 1024-token context window and the Gemma3 1B PT model, we recirculate only the token in position t t and examine the benefit at lag k k , i.e., to the token in position t + k t+k . The increase in target log likelihood (equivalently, reduction in perplexity) relative to the no-recirculation model is a power function of k k , with large increases at short lags but a residual tail even at long lags. Figure 9 a shows the recirculation benefit averaged over token position t ∈ [ 0,767 ] t\in[0,767] for lag k ∈ [ 1,256 ] k\in[1,256] , relative to the baseline condition in which no recirculation occurs. Figure 9 b teases apart the effect of the recirculated token position, t t , shown along the horizontal axis of the Figure, and the heatmap indicates the benefit magnitude over lags k ∈ [ 1,256 ] k\in[1,256] . At the earliest positions, roughly t < 10 t<10 , recirculation reduces log likelihood, 5 5 5 In contrast to the Gemma3 1B model, we did not observe harmful effects of recirculating the early token positions with the larger Gemma3 4B and 12B models. but at all later position, recirculation yields increases, particularly at short lags. Roughly, positions 20 to 200 in the window appear to have the most persistent effects, with a measurable benefit even at lag 256.

The effectiveness of recirculation depends not only on token position, but also token content. We tag each token with a grammatical part-of-speech (PoS) and then determine the mean recirculation benefit for each PoS regardless of token position. Figure 9 c indicates that adverbs, adjectives, and verbs show the biggest effects, whereas fixed classes such as numerals, determiners, and pronouns show the smallest effects. We obtain supporting evidence in experiments where we recirculate all and only tokens of a given PoS (Appendix C.4 ) compared to a count matched random set. Interestingly, for nouns we find that plural forms yield robust benefits whereas singular forms do not. The distinction in how models process singular and plural nouns was also noted by Galashov et al. (2025) .

In contrast to the above experiment, in which we recirculated only a single token, we also conducted an experiment in which we recirculated all-but-one token. We obtain results complementary to Figures 9 a,b, hinting that the benefit of recirculating individual tokens is additive in log likelihood. In all, the position and content effects seem to further rule out artifactual explanations for the phenomenon, and to support the story that recirculation helps construct persistent state representations.

### 4.5 Generative tasks

We evaluated a range of downstream tasks that require models to generate responses, both single-token choices and thinking responses. Broadly, we observe modest to significant improvements in model accuracy.

#### 4.5.1 Instruction following

In a simple instruction-following task, we prompted models with text that included: ⬇

Exact prompts, which included two-shot examples, and additional details are presented in Appendix D.1 . This task is appealing as a test of executive function in children and neurological patients. The baseline Gemma3 1B IT (instruction tuned) model performs at chance, which is why we focus on the larger Gemma3 4B and 12B models. Larger models, despite being better, still have room for improvement as evident in Figure 10 . Using the hyperparameters we previously selected to minimize perplexity on a tuning set, recirculation reduces the error rate by about 25% for the 4B model and by 75% for the 12B model. To get an upper bound on recirculation’s potential, we overfit by performing a hyperparameter sweep specifically to maximize accuracy on this instruction-following task. Although this sweep constrains just two parameters (layer indices), recirculation improves significantly, as shown by the task-specific bars of Figure 10 .

#### 4.5.2 Contextualization

Lepori et al. (2025) curated a dataset to explore model contextualization failures of the sort that occur in the bank dialog (Figure 2 a). Each dataset instance consists of a context , a cue , and a question , includes zero or more embedded distractor sentences , and demands a yes/no answer, e.g., ⬇

Examples were counterbalanced to ensure that biases (e.g., to prefer one sense of the word bank) did not influence results. Specifically, for model responses to count as correct, the model must respond to two variants of the question, e.g., Is it a financial institution? and Is it a geographical feature? Thus, chance accuracy is 25%.

In addition to questions with polysemous words like the example above, there were factual and gender-related questions, e.g., ⬇

Figure 11 shows results for the Gemma3 1B, 4B, and 12B IT models. The abscissa of each graph is the number of distractor sentences; performance tends to drop with more distractors. Shown separately in each graph are the three question types and results for the baseline (solid) and recirculation (dashed) models. The chance rate of 25% is depicted as a dotted horizontal line.

For 1B, two of the three question types show a clear improvement with recirculation; the third is at chance for both baseline and recirculation. For 4B, two of the three question types again show a reliable improvement with recirculation; the third is about the same. For 12B, two of the three question types are at a disadvantage with recirculation; the third is at ceiling for both. On balance, recirculation is a win, though its failure on 12B is disappointing. We used source and destination hyperparameters based on perplexity evaluations in Section 4.2 for a pretrained not instruction tuned model; sweeping hyperparameters with the instruction tuned model did result in larger gains (see Appendix D.2 ).

#### 4.5.3 Multiple-choice and single-token response tasks

We turn now to standard benchmark datasets, including those from Open LLM Leaderboard (v1) ( Beeching et al., 2023 ) , tested using the eval-harness package ( Gao et al., 2024 ) . The tasks we evaluate are: ARC easy ( Clark et al., 2018 ) , ARC challenge ( Clark et al., 2018 ) , MMLU ( Hendrycks et al., 2021 ) , Winogrande ( Sakaguchi et al., 2021 ) , BoolQ ( Clark et al., 2019 ) , PiQA ( Bisk et al., 2020 ) , HellaSwag ( Zellers et al., 2019 ) , and Lambada ( Paperno et al., 2016 ) . Lambada is a fill-in-the-blank single token response task. All the rest are multiple-choice tasks. Accuracy is measured as the percent correct responses. We use the MMLU development set to search over hyperparameters, as detailed in Appendix D.3 . All results are based on zero-shot evaluation.

Table 2 shows accuracy for the eight datasets in the first column for the baseline (second column) and recirculation (third column) models. Recirculation improves performance on six of the eight datasets, although the accuracy differences—both positive and negative—are modest. Disregard the last four columns of Table 2 for the moment; we discuss them in Section 4.6 .

#### 4.5.4 GSM8k

In contrast to the single-token-response datasets examined in the previous section, we turn to problems that are solved with chain-of-thought reasoning ( Kojima et al., 2022 ; Wei et al., 2023 ) , specifically the grade-school math problems in GSM8k ( Cobbe et al., 2021 ) . With single-token responses, any benefit of recirculation must depend on inferences made in processing the problem statement; with chain of thought, recirculation has the additional opportunity to support extended reasoning.

GSM8k also contrasts with the previous datasets in that it allows for a large set of candidate responses. When the number of candidates is large, improvements in model performance can be characterized in terms of either capability sharpening or capability expansion ( Yue et al., 2025 ) . The distinction is made with pass@k metrics ( Yue et al., 2025 ) : improved model performance with pass@1 (greedy decoding) indicates sharpening—the correct response beating out other responses in the set of candidates; improved model performance with pass@128 (accepting any of 128 samples if they are correct) indicates expansion of the set of possibilities. We therefore examine model performance with pass@1 (greedy decoding) and pass@128 (using a higher temperature and nucleus sampling, as recommended by Gemma Team et al. , 2024 ). Presented results are based on zero-shot evaluation. Following Kojima et al. (2022) , we use the zero-shot chain-of-thought prompt, which is important without few-shot examples as we are only evaluating base models in our case.

Figure 12 compares baseline and recirculated Gemma3 4B models (brown and blue bars, respectively) for pass@1 and pass@128. We turn to the green bar in Section 4.6 . Recirculation improves both pass@1 and pass@128 performance, indicating its support for both capability sharpening and capability expansion. Recirculation appears to benefit extended generative responses more than single-token responses (Section 4.5.3 ), thus revealing its promise for advanced problem solving and complex reasoning.

### 4.6 Adaptive recirculation

Our strategy to this point has been to determine how far we could push recirculation as a pure inference-time technique for an off-the-shelf model. Recirculation is surprising and intriguing specifically because it works out-of-the-box. The blindingly obvious next step is to improve the robustness and gains from recirculation via model adaptation. Because training a recurrent architecture comes at a high cost and there is a risk of overfitting if the entire model is fine tuned, we begin by taking small steps away from the training-free setting, focusing on modulating the recirculation hyperparameters α \alpha and β \beta (Equation 1 ). Our goal is to make changes as minimal as possible that show benefits over the training-free recirculation setting.

For training experiments, we use 250 documents from each of arXiv, C4, and PG19. Experiments are conducted with Gemma3 1B PT. We fix the source and destination layer indices that produced earlier recirculation results (Table 1 ) and explore methods that learn representation-mixing terms α \alpha and β \beta to minimize prediction loss.

We score six different adaptation methods on nine datasets by the reduction in evaluation-set perplexity they achieve relative to the baseline (no recirculation) model. Experimental details are presented in Appendix D.5 . The six alternative methods, shown from left to right in Figure 13 , are as follows. • Fixed. We use the fixed, previously determined coefficients ( α = 0.15 \alpha=0.15 , β = 0.85 \beta=0.85 ), thereby replicating earlier results.

• Learned constant scalars . We perform gradient descent in the scalars α \alpha and β \beta to best fit the training data.

• Learned conditional scalars . We train an MLP that maps the token-specific source and destination embeddings to scalar α \alpha and β \beta values, allowing the model to self-determine the strength of recirculation for each token.

• Learned constant vectors. Instead of scaling the source and destination vectors by constants when recirculating, we scale by learned vector-valued 𝜶 \boldsymbol{\alpha} and 𝜷 \boldsymbol{\beta} : 𝒛 t + 1 , t , d = 𝜶 ∘ f ⁡ ( 𝒛 t , t , s ) + 𝜷 ∘ 𝒛 t , t , d , {\boldsymbol{z}}_{t+1,t,d}=\boldsymbol{\alpha}\circ f({\boldsymbol{z}}_{t,t,s})+\boldsymbol{\beta}\circ{\boldsymbol{z}}_{t,t,d}, (3) where ∘ \circ is the Hadamard product.

• Learned conditional vectors. We train an MLP that maps the token-specific source and destination embeddings to vector-valued 𝜶 \boldsymbol{\alpha} and 𝜷 \boldsymbol{\beta} .

• Model fine tuning. We fine tune the Gemma3 1B architecture augmented with the recurrent connections of recirculation. We fix the mixture coefficients at α = 0.15 \alpha=0.15 and β = 0.85 \beta=0.85 .

Of the methods that learn coefficients α \alpha and β \beta , the two that learn vector valued coefficients (fourth and fifth sets) perform better than those that learn scalar coefficients (second and third sets). And the two methods that learn token-conditional coefficients (third and fifth set) outperform the two methods that learn static coefficients (second and fourth set). Of the four methods, the best (fifth set: learned conditional vector 𝜶 , 𝜷 \boldsymbol{\alpha},\boldsymbol{\beta} ) trains an MLP to generate token-specific mixture vectors. The other methods (second, third, and fourth sets) can be considered ablations demonstrating that both vector-valued coefficients and conditioning on the current token is essential. Henceforth, we refer to the superior method as adaptive recirculation .

Adaptive recirculation obtains a mean 23.0% reduction in perplexity, relative to the 8.5% reduction for basic recirculation. Adaptive recirculation beats recirculation for each of the nine datasets (see Figure D.6 ). Adaptive recirculation performs at least as well as full Gemma3 1B fine tuning (23.0% versus 21.6%). The advantage of adaptive recirculation is that the Gemma3 model itself is untouched and therefore there is less risk of overfitting the model via fine tuning.

On downstream tasks, adaptive recirculation also shows promise, with the caveat that the dataset used for training the MLP is critical. Returning to the eight single-token-response datasets, the last four columns of Table 2 present accuracy for adaptive recirculation on these evaluation datasets for four different training datasets. Adapting to a small fraction of the MMLU test split (3600 of about 14000 examples) yields robust across-the-board improvements, and adapting to MMLU (auxiliary train split) yields on balance improvements. For comparability, we use the full test set of MMLU for evaluation even when training on the MMLU test split, which includes the examples used for training; performance on other datasets highlights generalization. Note that the auxiliary train set for MMLU is in general much lower quality than the well-curated MMLU test set, which may be the reason for its inferior performance. Finally, the MMLU train set is based on a combination of existing datasets, with potential contamination ( Hendrycks et al., 2021 ) . Training directly on other datasets such as ARC Easy or ARC Challenge produces significant drops in performance, although these datasets have a much smaller number of examples. Details of these experiments are provided in Appendix D.5 .

Although neither recirculation nor adaptive recirculation yield robust accuracy gains on single-token response tasks, the gains for GSM8k offer a promising signal for extended generative response tasks. The green bar in each panel of Figure 12 shows GSM8k performance with adaptive recirculation. Adaptive recirculation greatly benefits GSM8k, yielding 8.8% and 20.9% reductions in error rate with pass@1 and pass@128, respectively—a staggering result considering the model itself is untouched.

## 5 Discussion

Our work explores the consequences of recirculation, a training-free, inference-time architectural modification to incorporate recurrence into a transformer for improved state tracking. We have shown that for the Gemma3 pretrained and instruction-tuned architectures, recirculation yields notable reductions in model perplexity and enhances models’ ability to follow instructions, answer questions, and problem solve. When generating responses, recirculation incurs almost no additional computation cost; although recirculation requires running two transformer stacks in parallel instead of a single stack, modern AI hardware parallelizes efficiently in this case. However, there is an additional cost of processing the prefill context autoregressively, which for large-context problems, can be quite slow.