Recirculation — part 5/5

Cleaned-text code points [61757, 69380). Bibliography omitted here; source.md includes it.

On half the trials, the model was asked instead to identify the position of the animal (“If the first word is an animal, you say first…”). We formed 400 trials by combining twenty different fruit names with twenty different animal names. With the respond-to-fruit and respond-to-animal variants, this yields 800 trials total. The fruits are: apple, avocado, banana, blueberry, cantaloupe, cherry, grape, honeydew, kiwi, lemon, lime, mango, orange, peach, pear, pineapple, plum, raspberry, strawberry, watermelon. The animals are: bear, bird, cat, deer, dog, dolphin, elephant, fox, giraffe, lion, lizard, monkey, penguin, shark, snake, spider, tiger, whale, wolf, zebra.

We selected the most likely response among eight candidate tokens, which consisted of the words first and second, both in upper- and lower-case form and with and without a leading space.

For the Gemma3 4B and 12B IT, we used the source and destination layers determined by our previous perplexity hyperparameter sweep, and α = 0.07 \alpha=0.07 . For the task-specific result, we conducted a sweep using the instruction-following dataset to determine an upper bound on performance. These sweeps are shown in Figure D.1 . The 1B sweep is included as well, revealing that the model is essentially performing at chance. For the 4B model, source layer 18, destination layer 8 was best; for the 12B model, source layer 29, destination layer 16 was best.

### D.2 Contextualization

For Figure 11 of the main paper, we used hyperparameters chosen based on minimizing perplexity of the pretrained model. We also swept hyperparameters of the Gemma3 1B IT model using a particular condition of the Lepori et al. (2025) dataset: both the 5-distractor condition of the gender questions and the polysemy questions. These sweeps appear in the upper right of Figures D.2 - D.4 , corresponding to the 1B, 4B, and 12B models. In the lower row of the Figure, left to right we show recirculation accuracy with hyperparameters chosen based on pretrained model perplexity, instruction tuned model perplexity, accuracy for gender 5-distractor condition, and accuracy for polysemy 5-distractor condition. Note that the 1B model is not much above chance except for no-distractor polysemy questions.

### D.3 Multiple-choice and single-token response tasks

We tested the Gemma3 4B PT model. To determine the optimal hyperparameters we conducted a (source, destination) sweep using the 1531 MMLU development-set problems. These problems are distinct from the examples used for evaluation. Figure D.5 a shows the sweep, fixing α = 0.07 \alpha=0.07 . Figure D.5 b shows a scan over α \alpha , fixing source and destination layers to be the pair that yields the lowest perplexity in the sweep of Figure D.5 a. The resulting hyperparameters that were used in the various single-token response datasets were source 16, destination 5, α = 0.09 \alpha=0.09 .

### D.4 Standard benchmark datasets: GSM8k

We trained Gemma3 4B model with identical hyperparameters as the perplexity experiments (see Section 4.6 ), except that we masked the prompt part of the question, and only trained the model to predict the ground-truth response present in the dataset. We only report results using our best-performing conditional 𝜶 , 𝜷 \boldsymbol{\alpha},\boldsymbol{\beta} scheme as highlighted in Figure 13 .

### D.5 Adaptive recirculation

For all training experiments, we use 250 documents from each of the PG19, C4, and arXiv training sets. Documents are partitioned into windows of 1024 tokens and only completely full windows were included in the training set. The transformer with recirculation and the MLP module is trained with Back Propagation Through Time (BPTT). For the perplexity studies of Figure 13 , we use the Gemma3 1B PT model with the previously selected source and destination layers (Table B.1 ).

Our architecture is adapted from the next-latent prediction MLP in Teoh et al. (2025) , where we use a 2 hidden layer GELU-based MLP with the same hidden size as the model dimension, with layer-norm at the input of the MLP. The input is twice the model hidden dimension as we concatenate the source and the destination embeddings to be fed to the MLP. Output size is dependent on the formulation used; e.g., it would be twice the model hidden dimension for learned conditional vectors scheme, which predicts two scalars ( α \alpha , β \beta ) per dimension.

We use sigmoid activation at the output to ensure that these coefficients lie in [0,1]. Further, we initialize the parameters of the network such that it starts with α = 0.1 \alpha=0.1 and β = 0.9 \beta=0.9 at initialization, based on the range of values we found to be suitable from our grid-search results.

We train the model for 100 steps with a batch size of 32 using AdamW ( Loshchilov and Hutter, 2019 ) , a learning rate of 3e-4, and a weight decay of 1e-4. For all simulations other than LLM fine-tuning, we freeze the parameters of Gemma3. For LLM fine-tuning, we disable weight decay and use a small learning rate of 1e-5. Similarly, for the unconditional prediction schemes (same for all inputs), we disable weight decay and increase the learning rate to 1e-1.

Evaluation was performed with the validation or test set of nine datasets: ArXiv, PubMed, PG19, BookSum, Lambada, Gov Report, BillSum, OpenWebText, and Big Patent. Note that nonoverlapping subsets of ArXiv and PG19 were used for training and evaluation.

Figure D.6 b compares recirculation with fixed coefficients ( α = 0.15 , β = 0.85 \alpha=0.15,\beta=0.85 ) and the scheme in which an MLP is trained to produce vector-valued 𝜶 \boldsymbol{\alpha} and 𝜷 \boldsymbol{\beta} coefficients for each token (Figure D.6 a). We refer to this scheme as adaptive recirculation for short, as well as conditional 𝛂 , 𝛃 \boldsymbol{\alpha},\boldsymbol{\beta} in the text. The comparison indicates that adaptive recirculation increases the percentage reduction in perplexity for every dataset, and by a factor of three or more for many of the datasets.

When training our conditional vector 𝜶 , 𝜷 \boldsymbol{\alpha},\boldsymbol{\beta} scheme on downstream tasks with Gemma3 4B (Table 2 and Figure 12 ), we use the same hyperparameters as our perplexity experiments. However, we mask the prompt and only train on the response, which refers to only a single token in the single-token benchmarks (Table 2 ). Because the ARC datasets have fewer examples than the MMLU datasets (MMLU train set: 99842, MMLU test set: 14042, ARC easy: 2251, ARC challenge: 1119), we used multi-epoch training for the ARC datasets as we train on a total of 3200 examples (100 steps with a batch-size of 32).

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .