Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers — part 4/6

Cleaned-text code points [47009, 61995). Bibliography omitted here; source.md includes it.

While our original motivation for residual scaling was to prevent unbounded activations in pre-norm, we note a parallel relationship between our solution and common solutions to signal propagation problems ( Sun et al., 2025 ; Noci et al., 2022 ) . Specifically, it has been known that scaling down the output of the sub-layers when introducing them to the residual stream is beneficial to increasing effective layer, which is equivalent to increasing α 1 \alpha_{1} in Equation 2 . Moreover, in Theorem 2 , we show that the looping will become more stable for smaller α 2 \alpha_{2} in Equation 3 . In order to investigate the impact of these two parameters, we perform a coarse-grained ablation on the initial value of α 1 , α 2 \alpha_{1},\alpha_{2} on the Sudoku-Extreme task.

Section 4.4.2 demonstrates that the best initialization places α 1 \alpha_{1} at a high value and α 2 \alpha_{2} at a low value. Interestingly, the α 1 \alpha_{1} preference matches the common solutions to signal propagation problems: keeping the residual stream dominant. Moreover, the ablation also highlights the importance of having a more contractive mapping at initialization, as our convergence analysis ( Theorem 2 ) requires a sufficiently small α 2 \alpha_{2} for the loop to reach a fixed-point. On the other hand, comparing the row with the smallest α 2 \alpha_{2} choice ( α 2 = 0.25 \alpha_{2}=0.25 ) with the column with the largest α 1 \alpha_{1} ( α 1 = 0.75 \alpha_{1}=0.75 ), we observe that increasing α 1 \alpha_{1} has a slightly higher positive impact on the accuracy than a decreasing α 2 \alpha_{2} . We hypothesize that this is because it is easier for the model to recover from a bad choice of α 2 \alpha_{2} than α 1 \alpha_{1} , as the gradients for α 1 \alpha_{1} come from two different sources (the MHA and MLP sub-layers), and thus can be noisier.

In Figure 9 , we provide the distribution statistics of the residual scales after training. Interestingly, we observe that while the median of the α 1 , α 2 \alpha_{1},\alpha_{2} values over channels does not deviate significantly from the initial point, the spread of the distributions widens significantly. In the case of α 1 \alpha_{1} , the widening happens at a much larger scale, covering both very small and very large values. On the other hand, the α 2 \alpha_{2} distribution remains more concentrated, with the majority of the values actually becoming smaller than the initial point. This can be interpreted as the model learning to become more contractive during training, which is in line with the observations in ( Bansal et al., 2022 ) . Furthermore, this observation also supports our hypothesis that it might be easier for the model to learn the optimal α 2 \alpha_{2} values than the α 1 \alpha_{1} values.

## 5 Discussion

Our experiments support three broader observations about FPRM and looped reasoning models in general, which we discuss in turn.

##### Looped fixed-point models are adaptive.

FPRM adapts to the difficulty of the problem more effectively compared to TRM (Figures 4 , 6 , and Section 4.3 ), using fewer effective layers (compute), while achieving better performance. This is a consequence of FPRM halting closer to the saturation point of accuracy ( Figure 12 ). In contrast, TRM with its ACT halting mechanism either halts too early, resulting in lower performance, or too late, using excessive compute.

##### Enabling ACT at inference time.

The original proposed TRM does not use its trained ACT head at inference time, leading to the non-adaptive behavior. However, we find this to be an engineering challenge rather than a fundamental limitation. Therefore, in Figures 1 , 4 , 6 , we record the number of effective layers reached at the moment when the probability of halting exceeds 0.5 0.5 . Similarly, in the case of FPRM, we do so when the residual drops below the set threshold ( 0.1 0.1 in this case). However, note that the halted samples remain in the batch until the last sample in the batch halts. We leave the efficient implementation for future work.

##### The role of hierarchy in HRM and TRM.

While originally hierarchical reasoning was biologically motivated ( Wang et al., 2025 ) , later explanations involved likening the lower level of the hierarchy to a scratch pad, the latent representation of which is used by the higher level for prediction ( Jolicoeur-Martineau, 2025 ) . However, the role of hierarchy as the driving force behind the success of hierarchical reasoning models has been brought into question recently, with similar architectures without the hierarchy performing as well as hierarchical models ( Ge et al., 2025 ; ARC Prize Foundation, 2025 ) . In Section 4.4 , we observe that a Transformer model with post-norm, which is the building block of TRM and HRM, suffers from a signal propagation issue. On the other hand, in Section 4.2 we were able to show that by improving signal propagation, FPRM improves upon these models without requiring the hierarchy. In Figure 13 , we observe that reallocating the compute from the H- and L-steps to the additional deep supervision steps improves TRM’s performance. Since more H- and L-steps increase the effective layer of TRM within each supervision step, the signal propagation issue induced by post-norm is amplified. The improvement is therefore consistent with TRM being limited by the same signal-propagation issue we identify in Section 4.4 . In light of these results, we hypothesize that there might be a simpler explanation for the success of hierarchical models: the hierarchy improves signal propagation . We identify the theoretical explanation of the role of hierarchy through the lens of optimization and signal propagation as an interesting direction for future work.

##### Scaling behavior of FPRM.

The results of Section 4 combine into a coherent picture of how FPRM scales its computation. First, with better signal propagation FPRM is able to utilize compute more efficiently ( Figure 6 ). Second, as more difficult problems require more compute ( Merrill et al., 2024 ; Movahedi et al., 2025 ) , better test-time scaling of FPRM is mostly visible in harder tasks (Figures 4 , 5(a) ). Finally, because in FPRM halting is governed by the fixed-point optimizer rather than a learned module, a natural controlling mechanism for the compute-performance trade-off appears in the form of the decay rate γ \gamma and maximum patience P P , allowing practitioners to select a desired point on the Pareto front. However, the optimality of the algorithms learned by looped models is not guaranteed, with great variation not only possible but likely. For example, for solving A 5 A_{5} , CoT would require a super-logarithmic number of iterations ( Merrill and Sabharwal, 2024 ) , while an optimal algorithm could solve it in logarithmic time. This suboptimal scaling has also been observed in recurrent-in-depth state-space models ( Movahedi et al., 2025 ) , a behavior that we also observe in Figure 4 . Therefore, we propose it as an open challenge to find a latent reasoning architecture that achieves a solution with logarithmic complexity to state-tracking while remaining Turing-complete ( Dehghani et al., 2019 ) .

##### Limitations.

In a similar spirit to previous literature on end-to-end reasoning ( Kaiser and Sutskever, 2016 ; Fan et al., 2025 ; Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Du et al., 2022 ; Du et al., 2024 ) , we test our model only on algorithmic tasks and not on natural language. It is an open challenge to demonstrate that the compositional reasoning behavior that latent models exhibit on algorithmic tasks translates to other domains. In addition, even though the base architecture of FPRM could adopt any model (e.g. CNN, MLP, state-space models), we limit our experiments to Transformers.

## 6 Conclusion

We present architectural modifications for looped fixed-point Transformers that enable the use of pre-norm, improving the model’s ability to exploit deeper effective layer provided by looping. These modifications allow FPRM to outperform hierarchical baselines of similar size, such as HRM and TRM, on common symbolic reasoning benchmarks. We show that on state tracking and Sudoku-Extreme, FPRM is able to adapt its compute to the difficulty of the task. This capability stems from dynamically scaling depth through fixed-point iterations and improving signal propagation. We hope these architectural modifications and the accompanying insights will support further progress on latent reasoning models.

## Acknowledgments and Disclosure of Funding

We thank Felix Sarnthein, Albert Catalan-Tatjer, Jonas Geiping, Philipp Nazari, Carl Richardson, and Nouha Dziri for the helpful discussions and comments. Alexander Theus, Vera Milovanović, and Shlomo Libo Feigin are supported by the Max Planck ETH Center for Learning Systems. Vera Milovanović and Antonio Orvieto are supported by the AI2050 program at Schmidt Sciences. Antonio Orvieto, T. Konstantin Rusch, and Sajad Movahedi acknowledge the financial support of the Hector Foundation.

## Appendix A Proofs

### A.1 Fixed-point iterations bound

###### Proof.

We simplify our notation by omitting the pre-norm layer in Equation 2 . We start by unrolling the computation across the L L layers within a single fixed-point iteration. By recursively applying the layer update, we obtain 𝐳 i 2 ​ L = \displaystyle\mathbf{z}_{i}^{2L}= α 1 ⋅ 𝐳 i 2 ​ L − 1 + β 1 ⋅ f θ 2 ​ L 2 ​ L ​ ( 𝐳 i 2 ​ L − 1 ) \displaystyle\ \alpha_{1}\cdot\mathbf{z}_{i}^{2L-1}+\beta_{1}\cdot f^{2L}_{\theta^{2L}}\left(\mathbf{z}_{i}^{2L-1}\right) = \displaystyle= α 1 2 ⋅ 𝐳 i 2 ​ L − 2 + α 1 ​ β 1 ⋅ f θ 2 ​ L − 1 2 ​ L − 1 ​ ( 𝐳 i 2 ​ L − 2 ) + β 1 ⋅ f θ 2 ​ L 2 ​ L ​ ( 𝐳 i 2 ​ L − 1 ) \displaystyle\ \alpha_{1}^{2}\cdot\mathbf{z}_{i}^{2L-2}+\alpha_{1}\beta_{1}\cdot f^{2L-1}_{\theta^{2L-1}}\left(\mathbf{z}_{i}^{2L-2}\right)+\beta_{1}\cdot f^{2L}_{\theta^{2L}}\left(\mathbf{z}_{i}^{2L-1}\right) = \displaystyle= α 1 3 ⋅ 𝐳 i 2 ​ L − 3 + α 1 2 ​ β 1 ⋅ f θ 2 ​ L − 2 2 ​ L − 2 ​ ( 𝐳 i 2 ​ L − 3 ) + α 1 ​ β 1 ⋅ f θ 2 ​ L − 1 2 ​ L − 1 ​ ( 𝐳 i 2 ​ L − 2 ) + β 1 ⋅ f θ 2 ​ L 2 ​ L ​ ( 𝐳 i 2 ​ L − 1 ) \displaystyle\ \alpha_{1}^{3}\cdot\mathbf{z}_{i}^{2L-3}+\alpha_{1}^{2}\beta_{1}\cdot f^{2L-2}_{\theta^{2L-2}}\left(\mathbf{z}_{i}^{2L-3}\right)+\alpha_{1}\beta_{1}\cdot f^{2L-1}_{\theta^{2L-1}}\left(\mathbf{z}_{i}^{2L-2}\right)+\beta_{1}\cdot f^{2L}_{\theta^{2L}}\left(\mathbf{z}_{i}^{2L-1}\right) = \displaystyle= α 1 2 ​ L ⋅ 𝐳 i 0 + β 1 ⋅ ( ∑ j = 0 2 ​ L − 1 α 1 j ⋅ f θ 2 ​ L − j 2 ​ L − j ​ ( 𝐳 i 2 ​ L − j − 1 ) ) . \displaystyle\ \alpha_{1}^{2L}\cdot\mathbf{z}_{i}^{0}+\beta_{1}\cdot\left(\sum_{j=0}^{2L-1}\alpha_{1}^{j}\cdot f^{2L-j}_{\theta^{2L-j}}\left(\mathbf{z}_{i}^{2L-j-1}\right)\right). Now, substituting this expression into the fixed-point update gives 𝐳 i + 1 0 = \displaystyle\mathbf{z}_{i+1}^{0}= α 2 ⋅ 𝐳 i 2 ​ L + β 2 ⋅ 𝐱 \displaystyle\ \alpha_{2}\cdot\mathbf{z}_{i}^{2L}+\beta_{2}\cdot\mathbf{x} = \displaystyle= α 2 ⋅ ( α 1 2 ​ L ⋅ 𝐳 i 0 + β 1 ⋅ ∑ j = 0 2 ​ L − 1 α 1 j ⋅ f θ 2 ​ L − j 2 ​ L − j ​ ( 𝐳 i 2 ​ L − j − 1 ) ) + β 2 ⋅ 𝐱 \displaystyle\ \alpha_{2}\cdot\left(\alpha_{1}^{2L}\cdot\mathbf{z}_{i}^{0}+\beta_{1}\cdot\sum_{j=0}^{2L-1}\alpha_{1}^{j}\cdot f^{2L-j}_{\theta^{2L-j}}\left(\mathbf{z}_{i}^{2L-j-1}\right)\right)+\beta_{2}\cdot\mathbf{x} = \displaystyle= α 2 ​ α 1 2 ​ L ⋅ 𝐳 i 0 + α 2 ​ β 1 ⋅ ( ∑ j = 0 2 ​ L − 1 α 1 j ⋅ f θ 2 ​ L − j 2 ​ L − j ​ ( 𝐳 i 2 ​ L − j − 1 ) ) + β 2 ⋅ 𝐱 . \displaystyle\ \alpha_{2}\alpha_{1}^{2L}\cdot\mathbf{z}_{i}^{0}+\alpha_{2}\beta_{1}\cdot\left(\sum_{j=0}^{2L-1}\alpha_{1}^{j}\cdot f^{2L-j}_{\theta^{2L-j}}\left(\mathbf{z}_{i}^{2L-j-1}\right)\right)+\beta_{2}\cdot\mathbf{x}. For compactness, define ρ = α 2 ​ α 1 2 ​ L \rho=\alpha_{2}\alpha_{1}^{2L} and 𝐬 i = ∑ j = 0 2 ​ L − 1 α 1 j ⋅ f θ 2 ​ L − j 2 ​ L − j ​ ( 𝐳 i 2 ​ L − j − 1 ) . \mathbf{s}_{i}=\sum_{j=0}^{2L-1}\alpha_{1}^{j}\cdot f^{2L-j}_{\theta^{2L-j}}\left(\mathbf{z}_{i}^{2L-j-1}\right). Then the fixed-point iteration can be written as 𝐳 i + 1 0 = ρ ⋅ 𝐳 i 0 + α 2 ​ β 1 ⋅ 𝐬 i + β 2 ⋅ 𝐱 . \mathbf{z}_{i+1}^{0}=\rho\cdot\mathbf{z}_{i}^{0}+\alpha_{2}\beta_{1}\cdot\mathbf{s}_{i}+\beta_{2}\cdot\mathbf{x}. Unrolling this recursion over fixed-point iterations gives 𝐳 i + 1 0 = \displaystyle\mathbf{z}_{i+1}^{0}= ρ i + 1 ⋅ 𝐳 0 0 + β 2 ​ ( ∑ k = 0 i ρ k ) ⋅ 𝐱 + α 2 ​ β 1 ​ ( ∑ k = 0 i ρ k ⋅ 𝐬 i − k ) . \displaystyle\ \rho^{i+1}\cdot\mathbf{z}_{0}^{0}+\beta_{2}\left(\sum_{k=0}^{i}\rho^{k}\right)\cdot\mathbf{x}+\alpha_{2}\beta_{1}\left(\sum_{k=0}^{i}\rho^{k}\cdot\mathbf{s}_{i-k}\right). Since 0 ≤ α 1 , α 2 < 1 0\leq\alpha_{1},\alpha_{2}<1 , we have 0 ≤ ρ < 1 0\leq\rho<1 . Therefore, the geometric series is convergent. Taking norms and using the boundedness of each layer map, we get ‖ 𝐳 i + 1 0 ‖ ≤ \displaystyle\left\|\mathbf{z}_{i+1}^{0}\right\|\leq ρ i + 1 ​ ‖ 𝐳 0 0 ‖ + β 2 ​ ( ∑ k = 0 i ρ k ) ​ ‖ 𝐱 ‖ + α 2 ​ β 1 ​ ( ∑ k = 0 i ρ k ) ​ ( ∑ j = 0 L − 1 α 1 j ) ​ c f . \displaystyle\ \rho^{i+1}\left\|\mathbf{z}_{0}^{0}\right\|+\beta_{2}\left(\sum_{k=0}^{i}\rho^{k}\right)\left\|\mathbf{x}\right\|+\alpha_{2}\beta_{1}\left(\sum_{k=0}^{i}\rho^{k}\right)\left(\sum_{j=0}^{L-1}\alpha_{1}^{j}\right)c_{f}. Letting i → ∞ i\to\infty , the first term vanishes and the two geometric sums converge, which gives lim sup i → ∞ ‖ 𝐳 i 0 ‖ ≤ \displaystyle\limsup_{i\to\infty}\left\|\mathbf{z}_{i}^{0}\right\|\leq β 2 1 − ρ ​ ‖ 𝐱 ‖ + α 2 ​ β 1 1 − ρ ​ ( 1 − α 1 2 ​ L 1 − α 1 ) ​ c f . \displaystyle\ \frac{\beta_{2}}{1-\rho}\left\|\mathbf{x}\right\|+\frac{\alpha_{2}\beta_{1}}{1-\rho}\left(\frac{1-\alpha_{1}^{2L}}{1-\alpha_{1}}\right)c_{f}. Substituting back ρ = α 2 ​ α 1 2 ​ L \rho=\alpha_{2}\alpha_{1}^{2L} , we obtain lim sup i → ∞ ‖ 𝐳 i 0 ‖ ≤ \displaystyle\limsup_{i\to\infty}\left\|\mathbf{z}_{i}^{0}\right\|\leq β 2 1 − α 2 ​ α 1 2 ​ L ​ ‖ 𝐱 ‖ + α 2 ​ β 1 ​ ( 1 − α 1 2 ​ L ) ( 1 − α 2 ​ α 1 2 ​ L ) ​ ( 1 − α 1 ) ​ c f . \displaystyle\ \frac{\beta_{2}}{1-\alpha_{2}\alpha_{1}^{2L}}\left\|\mathbf{x}\right\|+\frac{\alpha_{2}\beta_{1}\left(1-\alpha_{1}^{2L}\right)}{\left(1-\alpha_{2}\alpha_{1}^{2L}\right)\left(1-\alpha_{1}\right)}c_{f}. We now set β 2 = 1 − α 2 ​ α 1 2 ​ L . \beta_{2}=1-\alpha_{2}\alpha_{1}^{2L}. This makes the coefficient of ‖ 𝐱 ‖ \left\|\mathbf{x}\right\| equal to 1 1 . Furthermore, setting β 1 = β 2 ​ ( 1 − α 1 ) ( 1 − α 1 2 ​ L ) \beta_{1}=\frac{\beta_{2}\left(1-\alpha_{1}\right)}{\left(1-\alpha_{1}^{2L}\right)} makes the coefficient of c f c_{f} equal to α 2 \alpha_{2} . Therefore, lim sup i → ∞ ‖ 𝐳 i 0 ‖ ≤ ‖ 𝐱 ‖ + α 2 ⋅ c f . \displaystyle\limsup_{i\to\infty}\left\|\mathbf{z}_{i}^{0}\right\|\leq\left\|\mathbf{x}\right\|+\alpha_{2}\cdot c_{f}. In particular, if the fixed-point iteration converges to 𝐳 ∞ 0 \mathbf{z}_{\infty}^{0} , then ‖ 𝐳 ∞ 0 ‖ ≤ ‖ 𝐱 ‖ + α 2 ⋅ c f . \displaystyle\left\|\mathbf{z}_{\infty}^{0}\right\|\leq\left\|\mathbf{x}\right\|+\alpha_{2}\cdot c_{f}. This completes the proof. ∎

### A.2 Contractive mapping

###### Proof.