Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers — part 3/6

Cleaned-text code points [31231, 47009). Bibliography omitted here; source.md includes it.

The task consists of difficult 30 × 30 30\times 30 shortest-path maze puzzles with unique solutions, introduced by Wang et al. (2025) . The train data and the test data each consist of 1000 1000 unique samples. Following Wang et al. (2025) , we do not use augmentation for this problem. The evaluation metric is exact (sequence) accuracy.

##### ARC-1 and ARC-2.

The Abstraction and Reasoning Corpus-1 (ARC-1) and -2 (ARC-2), introduced by Chollet et al. (2024) , aim to assess the ability of the model to solve novel problems from minimal examples. ARC-1 consists of 2-3 2D grid-based input-output demonstration pairs with variable size (up to 30 × 30 30{\times}30 ), through which the model is supposed to learn an underlying transformation rule and apply it to a held-out sample. The train data and the test data each contain ~ 400 400 samples. ARC-2 has similar characteristics to ARC-1, but with much more challenging problems involving several complex transformations, which are less susceptible to brute-force solutions. The benchmark contains ~ 1000 1000 training samples and ~ 360 360 test samples. The evaluation metric is exact (sequence) pass@2 (top-2 predictions) accuracy. Importantly, pretrained reasoning LLMs with CoT often struggle with these tasks. For example, DeepSeek-R1 ( 671 671 B model) achieves 15.8 % 15.8\% on ARC-1 and 1.3 % 1.3\% on ARC-2, Claude 3.7 Sonnet 16K achieves 28.6 % 28.6\% on ARC-1 and 0.7 % 0.7\% on ARC-2.

##### State tracking.

The A 5 A_{5} and S 5 S_{5} state-tracking tasks, introduced by Merrill et al. (2024) , are algorithmic benchmarks based on permutation composition. Each sample consists of an initial state and a sequence of k k update permutations; the model must apply the updates in order and predict the resulting final state. A 5 A_{5} uses the alternating group on five elements, i.e., the subgroup of even permutations, while S 5 S_{5} uses the full symmetric group on five elements. These tasks are useful proxies for stateful reasoning problems such as entity tracking, code execution, and game-state tracking, since solving them requires learning a composable update rule rather than memorizing computations at fixed lengths. We train on sequences containing up to 32 updates and evaluate out-of-distribution length generalization on sequences containing up to 128 updates. The evaluation metric is exact final-state accuracy.

### 4.2 Puzzle tasks

We first evaluate FPRM on puzzle problems, namely the Sudoku-Extreme, Maze-Hard ( Wang et al., 2025 ) , ARC-1, and ARC-2 ( Chollet et al., 2024 ) benchmarks. These benchmarks were designed to test whether latent recurrent reasoning models can solve symbolic search problems from limited supervision. In the following, we discuss the experimental results. For more information about the baselines, we refer the reader to Section 2 .

Table 1 demonstrates the effectiveness of FPRM as the best performing model with 7M parameters on Sudoku-Extreme, Maze-Hard, and ARC-1, with performance on par with TRM on ARC-2. On Sudoku-Extreme, FPRM improves upon even larger models. It is worth noting that these results are achieved without the breadth-search fixed-point method proposed by Huang et al. (2026) , which is orthogonal to FPRM’s modifications. On Maze-Hard, FPRM underperforms compared to the larger Attractor model with ~ × 4 \times\!4 more parameters. However, we note that we were not able to reproduce the results reported in the paper ( Fein-Ashley and Rashidinejad, 2026 ) .

On ARC-2, FPRM performs similarly to the publicly available TRM checkpoints , but underperforms compared to the TRM results reported in ( Jolicoeur-Martineau, 2025 ) and the URM baseline, which has roughly twice as many parameters. However, considering the recent trend, we emphasize that the ARC benchmarks appear to be much more sensitive to parameter count compared to the other puzzle tasks considered in this section ( Shu et al., 2026 ; Hu et al., 2025 ) . Therefore, we note that the comparison may not be fair in this case.

Given that FPRM performs on par with or better than the hierarchical baselines that use post-norm on the tasks from Table 1 , we hypothesize that the hierarchy might be alleviating signal propagation issues in these models. However, if this hypothesis holds, hierarchy’s benefit appears limited: in Figure 6 , both models improve as effective layer grows, but the performance gap widens in favor of FPRM, and TRM’s performance stays below FPRM at matched effective layers.

Moreover, in Table 3 , we attempt to investigate the effect of introducing our architectural modifications to the Transformer layers used in TRM on its performance on Sudoku-Extreme. We find the proposed modifications to have a detrimental impact on the performance of the models, which we attribute to, among other things, hyperparameter optimization and a careful redesigning of the looping.

### 4.3 Adaptivity

In this section, we demonstrate the adaptivity of FPRM, and compare it to TRM. We use three benchmarks: Sudoku-Extreme from Section 4.2 and state-tracking benchmarks A 5 A_{5} and S 5 S_{5} introduced in Merrill et al. (2024) . Each benchmark provides an intuitive measure of difficulty. For Sudoku-Extreme, this is the number of empty cells, an established proxy for difficulty ( Prates and Lamb, 2018 ) . For state-tracking, it is the sequence length.

##### State tracking.

As shown in Figure 4 , plain TRM fails to extrapolate: at length 128, TRM and TRM with ACT enabled both obtain 45.8 % ± 3.9 % 45.8\%\pm 3.9\% on A 5 A_{5} and 39.4 % ± 1.9 % 39.4\%\pm 1.9\% on S 5 S_{5} . Adding a causal 1D convolution layer substantially improves length generalization, reaching 91.4 % ± 2.3 % 91.4\%\pm 2.3\% on A 5 A_{5} and 97.2 % ± 2.5 % 97.2\%\pm 2.5\% on S 5 S_{5} . This modification is not part of the original TRM, but matches the task structure: group composition is a local left-to-right scan, s i = s i − 1 ⋅ g i s_{i}=s_{i-1}\cdot g_{i} , and causal convolution provides a shared translation-equivariant primitive for this operation.

However, 1D convolution does not make ACT reliable. On A 5 A_{5} , TRM+conv+ACT scales compute with sequence length but drops to 65.3 % ± 2.2 % 65.3\%\pm 2.2\% , well below TRM+conv without ACT. On S 5 S_{5} , it reaches 96.2 % ± 3.2 % 96.2\%\pm 3.2\% , but uses substantially more compute than FPRM; moreover, only a few seeds learn to adapt, while the others exhaust the full compute budget. In contrast, FPRM scales compute smoothly with sequence length while maintaining high accuracy, reaching 98.1 % ± 2.2 % 98.1\%\pm 2.2\% on A 5 A_{5} and 98.8 % ± 0.9 % 98.8\%\pm 0.9\% on S 5 S_{5} at length 128.

##### Sudoku-Extreme.

We compare the performance and efficiency of the halting mechanisms in FPRM and TRM on Sudoku puzzles of varying difficulty, measured by the number of empty cells ( Figure 6 ). The sample counts per difficulty level are shown in Figure 12 . We observe that halting mechanisms in both TRM and FPRM adapt the number of loops, i.e., inference compute, to the sample difficulty. In contrast, the default behavior of TRM during inference (deactivated ACT) does not adapt the compute to the task difficulty. Even when we enable ACT with TRM, FPRM proves to be more efficient and accurate.

##### Compute–accuracy trade-off in fixed-point halting.

The previous experiments show that FPRM adapts its compute to task difficulty. We now examine how this adaptation can be controlled. In Figure 8 , we demonstrate the effect of the step-size decay rate ( γ \gamma ) and maximum patience ( P P ) from Algorithm 1 on the halting time and performance at test-time. Max. inference-time budget was set to be the same (70k effective layers) for all experiments. We observe that larger decay rates improve the performance. Moreover, we observe that maximum patience has minimal impact on the performance, with its impact completely disappearing for larger decay rates. On the other hand, the increased performance comes at the cost of reaching much deeper effective layers before halting. The largest decay rate that achieves the best performance also exhausts the max. inference budget.

These trends follow from how γ \gamma and P P control the step size η \eta from Algorithm 1 . A decay rate closer to 1 reduces η \eta only slightly once the residual stops improving. The state therefore keeps evolving, halting is deferred, and the model reaches deeper effective layers. The accuracy gain follows directly from these additional iterations. Patience P P sets only when a decay occurs, not its magnitude, so its effect is minor; as γ \gamma approaches 1 1 each decay becomes negligible and the two patience curves coincide. The decay rate thus controls the compute-accuracy trade-off directly: a larger γ \gamma spends more compute and yields higher accuracy, while P P offers only secondary control that vanishes as compute saturates.

### 4.4 Depth-induced signal propagation issues

Here, we aim to investigate the role of pre-norm with residual scaling in mitigating the depth-induced signal propagation issues in looped models. For a comprehensive investigation, we approach the problem from three angles: (1) trainability , where we show that pre-norm with residual scaling keeps activations bounded and enables stable training at large depth (Section 4.4.1 ); (2) depth utilization , where we show that boundedness alone is not sufficient, and that pre-norm with residual scaling additionally improves depth utilization in FPRM ( Sun et al., 2025 ) (Section 4.4.2 ); and (3) residual scaling , where we analyze the training dynamics of the scaling parameters (Section 4.4.3 ).

#### 4.4.1 Boundedness of activation norms and trainability

As noted in Section 3.1 , the normalization scheme governs a trade-off between activation stability and signal propagation, which sharpens with depth. Post-norm keeps activations bounded but suffers from signal propagation issues. Pre-norm enables better signal propagation but lets activations grow exponentially. To isolate this trade-off, we adopt the Looped Transformer framework ( Saunshi et al., 2025 ; Dehghani et al., 2019 ; Kaiser and Sutskever, 2016 ) with a fixed number of effective layers. This controls for dynamic halting, which can reduce the effective layers, as introduced in Section 3 .

In Figure 2(b) we show the norm of the final activations of the residual branch of Looped Transformer at initialization. From the figure, we conclude that the magnitude of the activations grows exponentially in the case of pre-norm Looped Transformer. In contrast, when using post-norm or pre-norm with residual scaling, the architecture benefits from bounded activations. In Figure 2(a) we give evidence that boundedness is a prerequisite for reaching deeper effective layers. Pre-norm architecture with its unbounded activations diverges and can’t reach deep effective layers, while bounded architectures ensure trainability. Therefore, a Looped Transformer with pre-norm and without residual scaling cannot achieve deep effective layers, highlighting one aspect of the importance of residual scaling in pre-norm.

#### 4.4.2 Pre-norm with residual scaling enables higher depth utilization

An important measure for evaluating signal propagation issues in deep neural networks is depth utilization ( Sun et al., 2025 ) . Depth utilization measures whether all layers contribute meaningfully to the model. It is commonly probed by removing layers from a trained model and measuring the effect on performance at test-time. In models with signal propagation problems, removing deeper layers (closer to the output) usually does not impact the performance significantly, highlighting that these models fail to utilize the compute due to signal propagation issues. However, in a looped model there is no fixed stack of layers to remove, since the same block is applied to the previous latent representations. Consequently, we cannot perform the same experiment here. We therefore probe the same property from the opposite direction: instead of removing computation and measuring degradation, we add computation and measure improvement. We do so in two complementary regimes. In the first, using the state-tracking task, we ask whether the model can be trained to use the depth that a harder task requires. In the second, using the Sudoku-Extreme task, we ask whether a trained model can convert depth beyond its training regime into further gains at test time.

##### State-tracking.

In Section 4.4.1 , we discussed the necessity of using normalization layers that ensure the boundedness of the activations, which is required for the stable training. However, bounded activations are not a sufficient condition to ensure effective utilization of large depth, once it is reached. We demonstrate this in Figure 2(a) using a state-tracking task, where increasing difficulty, corresponding to longer sequences, requires training at greater depths ( Movahedi et al., 2025 ) . We train a Looped Transformer model with its number of loops (effective layer) tied to the train sequence length. We choose the sequence length from the set { 8 , 16 , 32 , 64 , 128 , 256 , 512 } \{8,16,32,64,128,256,512\} . We plot the maximum sequence length solved with > 90 % >90\% accuracy against effective layer. Because we also at test-time match effective layer to the training sequence length, a model with no signal-propagation bottleneck should solve exactly the length its depth permits, tracing the identity line y = x y=x (Figure 3 in Movahedi et al. (2025) ). We observe that this behavior is only present in the model equipped with pre-norm and residual scaling. We interpret this observation as strong evidence for improved trainability in Looped Transformers with pre-norm and residual scaling.

##### Sudoku-Extreme.

Compared to the previous experiment on the state-tracking task, here we run each model far beyond its trained depth, trying to detect the point where more compute no longer translates into improvements at test-time. We expect the performance of a model with fewer signal propagation issues to saturate later and with more effective layers, indicating that the model is capable of reaching deeper effective layer. On the other hand, the performance of a model bottlenecked by signal propagation problems is expected to saturate early, indicating that it cannot convert the extra compute into better predictions. For this experiment, we focus on two variants of the FPRM model, one with pre-norm and residual scaling, and the other with post-norm, both trained on the Sudoku-Extreme task.

In Figure 7(a) , we demonstrate that scaling test-time compute improves performance for both types of normalization. Furthermore, considering Figure 7(b) , we also observe that the majority of the improvement comes before at least half of the samples halt. However, there are clear differences between the effective layer of the two normalization methods in Figure 7(a) , as the pre-norm model’s performance saturates at almost twice as much compute, indicating improved signal propagation through depth.

The advantage of pre-norm with residual scaling also extends to the cross-model comparison in Figure 6 , between FPRM (pre-norm with residual scaling) and TRM (post-norm). In this inference-time scaling experiment, all samples are run to a varied maximum looping compute budget. For TRM, compute is scaled through deep supervision steps, which we find optimal relative to scaling L- and H-steps (see Appendix F ). FPRM outperforms TRM across a range of effective-layer depths reached, with the gap widening at higher compute budgets, consistent with FPRM making better use of its depth.

#### 4.4.3 The training dynamics of residual scales