Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers β part 1/6
Cleaned-text code points [0, 15570). Bibliography omitted here; source.md includes it.
# Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
###### Abstract
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The depth of effective layers reached by looping determines the quality of the solution these models find. Similar to deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address the signal propagation issue by using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM : a Transformer-based F ixed- P oint R easoning M odel that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to the difficulty of the task. FPRM proves effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking and ARC-AGI. The implementation can be found here .
## 1 Introduction
Reasoning in neural networks has increasingly been framed as a problem of scaling test-time compute: a model should be able to spend more computation on inputs it finds harder ( OpenAI, 2024 ; Snell et al., 2024 ) . However, doing so requires two ingredients. (1) flexibility: the possibility of spending a variable amount of compute on the problem. Once the model is flexible, the next step is (2) adaptivity: a way to scale the compute spent on the problem; i.e., when to halt the computation.
The standard way to achieve both is through a Chain-of-Thought (CoT) mechanism ( Wei et al., 2022 ) . With CoT, the model scales compute through verbalization, and makes halting decisions based on predicting a specialized halting token. However, this emerging behavior requires a special training regime and hand-crafted reasoning traces ( Guo et al., 2025 ) . This makes the method complex and undermines the desirable property of end-to-end training.
An alternative approach to end-to-end training for reasoning models is emerging in the form of looped architectures ( Dehghani et al., 2019 ; Bansal et al., 2022 ; Saunshi et al., 2025 ; Hao et al., 2024 ) . Whereas in CoT, the compute scales along the sequence dimension, in looped architectures it scales along the depth dimension ( i i ): π³ i + 1 = f ΞΈ β ( π³ i , π± ) , \mathbf{z}_{i+1}=f_{\theta}(\mathbf{z}_{i};\mathbf{x}), (1) where f ΞΈ β ( π³ , π± ) f_{\theta}(\mathbf{z};\mathbf{x}) is a neural network, π± \mathbf{x} is the input, and π³ i \mathbf{z}_{i} is the i th i^{\text{th}} latent representation. Thus, compute can be increased at test-time by looping in depth, naturally introducing flexibility from the architecture. Looped models have been shown to have an inductive bias towards learning algorithms ( Yang et al., 2024 ; Fan et al., 2025 ) and have achieved remarkable success on reasoning benchmarks. For example, Hierarchical Reasoning Model (HRM) ( Wang et al., 2025 ) and Tiny Reasoning Model (TRM) ( Jolicoeur-Martineau, 2025 ) incorporate a hierarchical structure into the looping process, proving effective in solving puzzle tasks such as sudoku, maze, and ARC-AGI ( Chollet et al., 2024 ) .
The halting decision (i.e., deciding when to stop iterating), however, is no longer trivial in looped models. Most approaches either fix or randomly sample the number of loops ( Jeddi et al., 2026 ; Zhu et al., 2025 ; Geiping et al., 2026 ; Saunshi et al., 2025 ; Botev et al., 2024 ; Bansal et al., 2022 ) , which eliminates adaptivity, or use external modules trained to make halting decisions ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Dehghani et al., 2019 ) . The latter is associated with separate Adaptive Computation Time (ACT) networks ( Graves, 2016 ) , which introduce optimization challenges, as they require a continuous relaxation of a discrete objective. As a result, ACT can fail to provide adaptivity, which we demonstrate in the case of HRM and TRM (Figures 4 , 6 ). We mitigate this issue by introducing a different halting mechanism: let the model loop until its hidden state converges to a fixed-point, and use the convergence itself as the halting signal. Unlike ACT, the fixed-point halting mechanism requires no external module and lets the model spend as much compute as a given input demands.
A further challenge arises as the number of loops in the model increases. This is desirable for harder tasks, but unrolling the recursions yields a deep effective layer. As a result, looped architectures suffer from the same signal propagation issue as deep non-looped networks. To this end, we adapt architectural techniques originally developed to optimize traditional transformers at deep effective layers, carefully modifying them for the Looped Transformer setting. Our key intuition is that Looped Transformers are, in part, very deep transformers .
Viewed this way, one design choice is surprising: looped models commonly use post-norm ( Dehghani et al., 2019 ; Geiping et al., 2026 ; Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) . Deep non-looped architectures, by contrast, prefer pre-norm, since post-norm causes unstable training ( Xiong et al., 2020 ; Noci et al., 2022 ) . Yet in looped models, post-norm serves a specific purpose. It keeps the magnitude of activations bounded as the model iterates, preventing the hidden states from diverging ( Labovich, 2026 ) (see Figure 3 ). This raises the question: can we switch the post-norm to pre-norm, while ensuring the activations stay bounded in a different way? If so, this could make the training of looped models stable at large depths. In this paper, we use residual scaling parameters to give an affirmative answer to this question. In Figure 1 , we observe that fixed-point halting, together with better signal propagation, allows FPRM, a non -hierarchical reasoning model, to outperform TRM at a lower cost on the Sudoku-Extreme benchmark. We provide a description of how the figure was made in Appendix D .
We summarize our contributions as follows:
1. Successfully training a looped pre-norm Transformer. We modify a Transformer layer to be trainable over deeper effective layers by switching post-norm to pre-norm and adding residual scaling parameters.
2. Reaching a fixed-point as a halting mechanism. To enable stable training, we propose a theoretically motivated modification specific to fixed-point models to limit the oscillation around the fixed-point.
3. Proposing the framework FPRM: Fixed-Point Reasoning Model. We show that it outperforms the baselines on Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and state-tracking benchmarks A 5 A_{5} and S 5 S_{5} among 7M parameter models. Notably, we achieve our results without the hierarchical structure of HRM and TRM. To the best of our knowledge, FPRM is the first Transformer-based reasoning model that exhibits adaptivity of compute to task difficulty (Figures 1 , 4 ).
## 2 Background and Related Works
##### Looped models.
Looped architectures address the flexibility requirement of reasoning models by decoupling effective layer from parameter count, providing variable computation per input without scaling the number of parameters. This makes looping a natural inductive bias for tasks that can be solved by repeatedly applying local or compositional subroutines. Early examples include Neural GPUs ( Kaiser and Sutskever, 2016 ) and Universal Transformers ( Dehghani et al., 2019 ) ; more recent works show that Looped Transformers can improve length generalization, learn algorithmic structure, and approximate much deeper untied models on reasoning tasks ( Yang et al., 2024 ; Fan et al., 2025 ; Saunshi et al., 2025 ; Kapl et al., 2026 ; Geiping et al., 2026 ; Giannou et al., 2023 ; Kohli et al., 2026 ) . Recent recursive reasoning models such as HRM ( Wang et al., 2025 ) and TRM ( Jolicoeur-Martineau, 2025 ) instantiate this principle in compact architectures. Using small networks, these models have been able to outperform prominent LLM-based reasoning models. At the core of these methods is the idea of the need for a hierarchical looping mechanism, wherein the compute is distributed between a fast-looping component and a slow-looping component. URM ( Gao et al., 2025 ) follows up on these works and shows that by using short-conv, performance can be further improved. However, as we describe in this paper, previous works leave two central design axes open to improvement: First, how should a looped model decide how many iterations to run on each input at test-time? FPRM addresses this through fixed-point halting. Second, how to make the model utilize its deep effective layer? FPRM does so by switching post-norm to pre-norm and adding residual scaling. Importantly, FPRM achieves better performance than TRM and HRM without using a hierarchical structure for the loops.
##### Adaptive computation.
Classical ACT methods answer the adaptivity question by learning an explicit halting rule, such as halting probabilities, per-token stopping decisions, or learned distributions over computation depth ( Dehghani et al., 2019 ; Banino et al., 2021 ) . More recent works extend this idea by picking an adaptive depth for each token ( Bae et al., 2025 ; Song et al., 2026 ) . Such mechanisms can, in principle, allocate more computation to harder instances, but they all add a separate learned decision process on top of the recurrent computation itself, and this process is hard to optimize because the halting decision is discrete rather than continuous. Moreover, analyses of HRM find that ACT does not always scale the inference compute with the actual difficulty of the input ( Ren and Liu, 2026 ; Ge et al., 2025 ) , which can more generally also be the case with CoT ( Palod et al., 2025 ) , a limitation we also observe and address by instead halting when the iteration reaches a fixed-point.
##### Deep equilibrium and fixed-point models.
An alternative is to use the convergence of the latent dynamics as the halting criterion. In this view, computation stops when consecutive iterates become sufficiently close β π³ i + 1 β π³ i β β€ Ο΅ \|\mathbf{z}_{i+1}-\mathbf{z}_{i}\|\leq\epsilon , which corresponds to convergence toward a fixed-point π³ β = f ΞΈ β ( π³ β , π± ) \mathbf{z}^{\star}=f_{\theta}(\mathbf{z}^{\star};\mathbf{x}) . This perspective is also supported by recent mechanistic analyses of looped language models, which observe that recurrent trajectories often converge to fixed-points, suggesting that weight-tied looped models can naturally implement an implicit fixed-point computation ( Blayney et al., 2026 ) . This view is most extensively developed in Deep Equilibrium Models ( Bai et al., 2019 ) , which replace a finite stack of layers by the equilibrium point of a weight-tied transformation. However, DEQ models usually find the fixed-point via a quasi-Newton approach such as Broydenβs method ( Broyden, 1965 ; Bai et al., 2019 ) or Anderson acceleration ( Anderson, 1965 ; Geng and Kolter, 2023 ) rather than fixed-point iterations, which makes them different from looped models. Moreover, DEQ models are difficult to optimize ( Bai et al., 2021 ; Anil et al., 2022 ; Jolicoeur-Martineau, 2025 ) , which we address by our proposed architectural changes. Concurrent to our work, Attractor Models ( Fein-Ashley and Rashidinejad, 2026 ) frame the iterations of TRM as a root-finding problem similar to DEQ, which they solve with Anderson acceleration. Additionally, they propose to optionally include a larger separate network to βguessβ the initial latent. Another concurrent work is Equilibrium Reasoners (EqR) ( Huang et al., 2026 ) , which also adopts a fixed-point perspective and shows that it is possible to scale the compute not only depth-wise, but also by making multiple initial guesses at training and inference time, favoring attractors with a wide basin. Our contributions are largely orthogonal: we target the signal-propagation issues that limit trainable depth, replacing post-norm with pre-norm and residual scaling, and damping the iteration to suppress oscillation around the fixed-point. Moreover, in contrast to these works we do not use the hierarchical looping of TRM.
##### Score-based methods.
Diffusion Models ( Ho et al., 2020 ; Song et al., 2021 ) and Energy-Based Models (EBMs) ( LeCun et al., 2006 ) provide another way of iterative computation. An EBM learns a scalar energy function, the core idea being that the energy represents the negative log of the unnormalized density, and solutions correspond to low energy states. EBMs generate a prediction either by descending the energy landscape ( Belanger and McCallum, 2016 ; Belanger et al., 2017 ) or by sampling with MCMC methods ( Du and Mordatch, 2019 ) . In comparison, diffusion models learn the score of the underlying distribution, which can be viewed as the gradient of a time-dependent energy function, thus the two are closely connected ( Salimans and Ho, 2021 ; Du et al., 2023 ) . While diffusion models typically integrate an SDE over a fixed time interval and in this sense are not adaptive, energy-based reasoning models ( Du et al., 2022 ; Du et al., 2024 ; Gladstone et al., 2025 ) have an adaptive halting mechanism. In these models, the halting criterion is convergence to a local minimum of the energy landscape. Compared to these methods, we model the generative process via fixed-point iterations of a learned operator, as opposed to descending through a learned energy function. The setting of learning the operator directly is important for the theoretical analysis of our architectural modifications in Section 3 .
##### Signal propagation in looped models.
As unrolled looped architectures can be viewed as deep networks, they are exposed to the same signal-propagation difficulties that arise in very deep Transformers. In deep sequence models, increasing depth can make optimization harder and can prevent later layers from being effectively used, a phenomenon often discussed as the curse of depth ( Dong et al., 2021 ; Noci et al., 2022 ; Sun et al., 2025 ) . Pre-norm is the standard remedy for these issues, but in a looped setting it removes a property the architecture relies on: bounded activations ( Figure 3 ). Bounded activations, important for stable training, are typically enforced through post-norm ( Geiping et al., 2026 ; Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) . Indeed, in our own experiments a pre-norm looped model without further modification diverges in activation norm as the iteration count grows, and fails to train at the large loop counts where this growth is most severe ( Figure 3 ). Therefore, our FPRM combines pre-norm with residual scaling to recover bounded and stable dynamics while still allowing gradients and representations to propagate through many iterations.
## 3 Method
Looped architectures rely on bounded activations for stability: as the model loops, an unbounded layer can cause the hidden states to grow without limit ( Figure 3 ). Current methods mostly employ post-norm to satisfy the boundedness condition ( Jolicoeur-Martineau, 2025 ; Wang et al., 2025 ; Geiping et al., 2026 ) . However, in fixed-depth models post-norm introduces a signal propagation issue, often associated with unstable training and restricted effective layer, as reported by Dong et al. (2021) ; Noci et al. (2022) ; Sun et al. (2025) . In Appendix B , we provide evidence for trainability issues in a toy looped model with post-norm, despite stability induced by bounded activations.