Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers — part 6/6

Cleaned-text code points [77710, 81052). Bibliography omitted here; source.md includes it.

The default proposed TRM uses 16 16 deep-supervision steps (outer loops) and variable L- and H-steps (inner loops). The L-steps outnumber the H-steps, typically by about 2 × 2\times . However, other configurations for the number of loops spent for deep supervision vs. inner loops are possible. We test the performance of other configurations with experiments shown in Figure 13 . We fix the L-to-H ratio at 2 2 and vary deep-supervision steps (segments) against per-segment recurrence depth (inner loops, shown with numbers next to the black markers in Figure 13 ). We measure test accuracy as a function of the number of deep-supervision steps, with fixed inference budget at approximately 1040 1040 steps. This budget also matches the max. number of effective layers reached by the baseline FPRM on this task. This isolates how a fixed inference budget is best allocated: toward more outer refinement steps or deeper inner recurrence. Figure 13 shows that the budget is best spent on outer, deep-supervision steps. This matches the finding that, in TRM’s post-norm Transformer, the gains from added effective layer depth get smaller compared to FPRM ( Figure 6 ). Fewer effective layers per segment is therefore the better strategy at a fixed compute budget. We adopt it for all experiments where we scale TRM compute (effective layers).

## Appendix G Additional Experimental Details

##### Weight initialization.

It seems that initializing the weights using a truncated normal distribution (LeCun initialization) is common practice in looped architectures. In our experiments, it accelerates the convergence but there is very little material difference in sequence accuracy after convergence.

##### Grokking.

There is some evidence for grokking in looped architectures, but on the maze task we observe convergence on the training data. And training the models for a longer period (up to 7 days) did not yield better performance.

##### Hyperparameters, device specification

We provide the values for some of the most important hyperparameters in the paper, per each model and dataset.

† The Looped Transformer row sweeps the {post-norm, pre-norm, pre-norm + residual-scaling} variants from Figure 2(a) ; the residual-scaling variant uses α 1 = 0.75 , α 2 = 0.5 \alpha_{1}{=}0.75,\alpha_{2}{=}0.5 and k train ∈ { 8 , 16 , 32 , 64 } k_{\text{train}}\in\{8,16,32,64\} .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .