Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers — part 5/6
Cleaned-text code points [61995, 77710). Bibliography omitted here; source.md includes it.
We first show that f θ ( . ; 𝐱 ) f_{\theta}(.;\mathbf{x}) is a contraction with respect to 𝐳 \mathbf{z} . For any 𝐳 , 𝐳 ′ \mathbf{z},\mathbf{z}^{\prime} , using the Lipschitzness of the L L -layer model, we have ‖ f θ ( 𝐳 , 𝐱 ) − f θ ( 𝐳 ′ , 𝐱 ) ‖ ≤ \displaystyle\left\|f_{\theta}(\mathbf{z};\mathbf{x})-f_{\theta}(\mathbf{z}^{\prime};\mathbf{x})\right\|\leq λ f ‖ ( α 2 ⋅ 𝐳 + β 2 ⋅ 𝐱 ) − ( α 2 ⋅ 𝐳 ′ + β 2 ⋅ 𝐱 ) ‖ \displaystyle\ \lambda_{f}\left\|\left(\alpha_{2}\cdot\mathbf{z}+\beta_{2}\cdot\mathbf{x}\right)-\left(\alpha_{2}\cdot\mathbf{z}^{\prime}+\beta_{2}\cdot\mathbf{x}\right)\right\| = \displaystyle= α 2 λ f ‖ 𝐳 − 𝐳 ′ ‖ . \displaystyle\ \alpha_{2}\lambda_{f}\left\|\mathbf{z}-\mathbf{z}^{\prime}\right\|. Therefore, if 0 ≤ α 2 λ f < 1 0\leq\alpha_{2}\lambda_{f}<1 , the map f θ ( . ; 𝐱 ) f_{\theta}(.;\mathbf{x}) is strictly contractive. By the Banach fixed-point theorem, it has a unique fixed-point 𝐳 ⋆ \mathbf{z}^{\star} , and the iteration 𝐳 i + 1 = f θ ( 𝐳 i , 𝐱 ) \mathbf{z}_{i+1}=f_{\theta}(\mathbf{z}_{i};\mathbf{x}) converges to 𝐳 ⋆ \mathbf{z}^{\star} .
We now prove the residual bound. Since 𝐳 i + 1 = f θ ( 𝐳 i , 𝐱 ) \mathbf{z}_{i+1}=f_{\theta}(\mathbf{z}_{i};\mathbf{x}) , the residual at iteration i i can be written as ‖ f θ ( 𝐳 i , 𝐱 ) − 𝐳 i ‖ = ‖ 𝐳 i + 1 − 𝐳 i ‖ . \displaystyle\left\|f_{\theta}(\mathbf{z}_{i};\mathbf{x})-\mathbf{z}_{i}\right\|=\left\|\mathbf{z}_{i+1}-\mathbf{z}_{i}\right\|. Using the contraction property of f θ ( . ; 𝐱 ) f_{\theta}(.;\mathbf{x}) , we get ‖ 𝐳 i + 1 − 𝐳 i ‖ = \displaystyle\left\|\mathbf{z}_{i+1}-\mathbf{z}_{i}\right\|= ‖ f θ ( 𝐳 i , 𝐱 ) − f θ ( 𝐳 i − 1 , 𝐱 ) ‖ \displaystyle\ \left\|f_{\theta}(\mathbf{z}_{i};\mathbf{x})-f_{\theta}(\mathbf{z}_{i-1};\mathbf{x})\right\| ≤ \displaystyle\leq α 2 λ f ‖ 𝐳 i − 𝐳 i − 1 ‖ . \displaystyle\ \alpha_{2}\lambda_{f}\left\|\mathbf{z}_{i}-\mathbf{z}_{i-1}\right\|. Applying this inequality recursively gives ‖ 𝐳 i + 1 − 𝐳 i ‖ ≤ ( α 2 λ f ) i ‖ 𝐳 1 − 𝐳 0 ‖ . \displaystyle\left\|\mathbf{z}_{i+1}-\mathbf{z}_{i}\right\|\leq\left(\alpha_{2}\lambda_{f}\right)^{i}\left\|\mathbf{z}_{1}-\mathbf{z}_{0}\right\|. Since 𝐳 1 = f θ ( 𝐳 0 , 𝐱 ) \mathbf{z}_{1}=f_{\theta}(\mathbf{z}_{0};\mathbf{x}) , we obtain ‖ f θ ( 𝐳 i , 𝐱 ) − 𝐳 i ‖ ≤ ( α 2 λ f ) i ‖ f θ ( 𝐳 0 , 𝐱 ) − 𝐳 0 ‖ . \displaystyle\left\|f_{\theta}(\mathbf{z}_{i};\mathbf{x})-\mathbf{z}_{i}\right\|\leq\left(\alpha_{2}\lambda_{f}\right)^{i}\left\|f_{\theta}(\mathbf{z}_{0};\mathbf{x})-\mathbf{z}_{0}\right\|. This completes the proof. ∎
### A.3 Mitigating oscillation through damping
###### Proof.
We first show that the fixed-points of g η , θ ( ⋅ , 𝐱 ) g_{\eta,\theta}(\,\cdot\,;\mathbf{x}) and f θ ( ⋅ , 𝐱 ) f_{\theta}(\,\cdot\,;\mathbf{x}) coincide. Since g η , θ ( 𝐳 , 𝐱 ) − 𝐳 = η ( f θ ( 𝐳 , 𝐱 ) − 𝐳 ) , \displaystyle g_{\eta,\theta}(\mathbf{z};\mathbf{x})-\mathbf{z}\;=\;\eta\bigl(f_{\theta}(\mathbf{z};\mathbf{x})-\mathbf{z}\bigr), and η > 0 \eta>0 , we have g η , θ ( 𝐳 , 𝐱 ) = 𝐳 g_{\eta,\theta}(\mathbf{z};\mathbf{x})=\mathbf{z} if and only if f θ ( 𝐳 , 𝐱 ) = 𝐳 f_{\theta}(\mathbf{z};\mathbf{x})=\mathbf{z} .
We now study the local stability of the damped iteration around 𝐳 ⋆ \mathbf{z}^{\star} . The Jacobian of g η , θ ( ⋅ , 𝐱 ) g_{\eta,\theta}(\,\cdot\,;\mathbf{x}) at 𝐳 ⋆ \mathbf{z}^{\star} is ∂ g η , θ ∂ 𝐳 ( 𝐳 ⋆ , 𝐱 ) = ( 1 − η ) 𝐈 + η 𝐉 , \displaystyle\frac{\partial g_{\eta,\theta}}{\partial\mathbf{z}}(\mathbf{z}^{\star};\mathbf{x})\;=\;(1-\eta)\mathbf{I}+\eta\mathbf{J}, so for every eigenvalue λ i \lambda_{i} of 𝐉 \mathbf{J} , the corresponding eigenvalue of the damped Jacobian is μ i ( η ) = 1 − η + η λ i = 1 + η ( λ i − 1 ) . \displaystyle\mu_{i}(\eta)\;=\;1-\eta+\eta\lambda_{i}\;=\;1+\eta(\lambda_{i}-1). Local asymptotic stability is implied by | μ i ( η ) | < 1 |\mu_{i}(\eta)|<1 for all i i . Writing λ i = a i + i b i \lambda_{i}=a_{i}+\mathrm{i}\,b_{i} with a i = ℜ ( λ i ) a_{i}=\Re(\lambda_{i}) , | μ i ( η ) | 2 = 1 + 2 η ( a i − 1 ) + η 2 | λ i − 1 | 2 . \displaystyle|\mu_{i}(\eta)|^{2}\;=\;1+2\eta(a_{i}-1)+\eta^{2}|\lambda_{i}-1|^{2}. Hence | μ i ( η ) | < 1 |\mu_{i}(\eta)|<1 if and only if η | λ i − 1 | 2 < 2 ( 1 − a i ) \eta|\lambda_{i}-1|^{2}<2(1-a_{i}) , i.e., 0 < η < 2 ( 1 − ℜ ( λ i ) ) | λ i − 1 | 2 . \displaystyle 0\;<\;\eta\;<\;\frac{2(1-\Re(\lambda_{i}))}{|\lambda_{i}-1|^{2}}. By assumption ℜ ( λ i ) < 1 \Re(\lambda_{i})<1 for every i i , so each upper bound is strictly positive. Setting η 0 = min { 1 , min i 2 ( 1 − ℜ ( λ i ) ) | λ i − 1 | 2 } > 0 , \displaystyle\eta_{0}\;=\;\min\!\left\{\,1,\;\min_{i}\frac{2(1-\Re(\lambda_{i}))}{|\lambda_{i}-1|^{2}}\,\right\}\;>\;0, we obtain | μ i ( η ) | < 1 |\mu_{i}(\eta)|<1 for every i i and every η ∈ ( 0 , η 0 ) \eta\in(0,\eta_{0}) . Therefore 𝐳 ⋆ \mathbf{z}^{\star} is locally asymptotically stable under the damped iteration 𝐳 t + 1 = g η , θ ( 𝐳 t , 𝐱 ) \mathbf{z}_{t+1}=g_{\eta,\theta}(\mathbf{z}_{t};\mathbf{x}) , and the iterates converge to 𝐳 ⋆ \mathbf{z}^{\star} from any sufficiently close initialization. ∎
### A.4 Error of truncated-BPTT
###### Proof.
Since ‖ 𝐉 ‖ 2 = σ < 1 \|\mathbf{J}\|_{2}=\sigma<1 , the Neumann series is convergent, and we have ( 𝐈 − 𝐉 ) − 1 = ∑ j = 0 ∞ 𝐉 j . \left(\mathbf{I}-\mathbf{J}\right)^{-1}=\sum_{j=0}^{\infty}\mathbf{J}^{j}. Therefore, the error of the k k -term truncated approximation is ‖ ( 𝐈 − 𝐉 ) − 1 − ∑ j = 0 k − 1 𝐉 j ‖ F \displaystyle\left\|\left(\mathbf{I}-\mathbf{J}\right)^{-1}-\sum_{j=0}^{k-1}\mathbf{J}^{j}\right\|_{F} = ‖ ∑ j = k ∞ 𝐉 j ‖ F \displaystyle=\left\|\sum_{j=k}^{\infty}\mathbf{J}^{j}\right\|_{F} ≤ ∑ j = k ∞ ‖ 𝐉 j ‖ F . \displaystyle\leq\sum_{j=k}^{\infty}\left\|\mathbf{J}^{j}\right\|_{F}. Using the relation ‖ 𝐀 ‖ F ≤ D ‖ 𝐀 ‖ 2 \|\mathbf{A}\|_{F}\leq\sqrt{D}\|\mathbf{A}\|_{2} for 𝐀 ∈ ℝ D × D \mathbf{A}\in\mathbb{R}^{D\times D} , together with submultiplicativity of the spectral norm, we get ‖ 𝐉 j ‖ F \displaystyle\left\|\mathbf{J}^{j}\right\|_{F} ≤ D ‖ 𝐉 j ‖ 2 \displaystyle\leq\sqrt{D}\left\|\mathbf{J}^{j}\right\|_{2} ≤ D ‖ 𝐉 ‖ 2 j \displaystyle\leq\sqrt{D}\left\|\mathbf{J}\right\|_{2}^{j} = D ⋅ σ j . \displaystyle=\sqrt{D}\cdot\sigma^{j}. Substituting this into the previous inequality gives ‖ ( 𝐈 − 𝐉 ) − 1 − ∑ j = 0 k − 1 𝐉 j ‖ F \displaystyle\left\|\left(\mathbf{I}-\mathbf{J}\right)^{-1}-\sum_{j=0}^{k-1}\mathbf{J}^{j}\right\|_{F} ≤ D ∑ j = k ∞ σ j \displaystyle\leq\sqrt{D}\sum_{j=k}^{\infty}\sigma^{j} = D ⋅ σ k 1 − σ . \displaystyle=\sqrt{D}\cdot\frac{\sigma^{k}}{1-\sigma}. Thus, the approximation error decays as 𝒪 ( σ k ) \mathcal{O}(\sigma^{k}) .
To make the corresponding gradient statement explicit, let 𝐏 = ∂ f θ ∂ θ ( 𝐳 ⋆ , 𝐱 ) \mathbf{P}=\frac{\partial f_{\theta}}{\partial\theta}(\mathbf{z}^{\star};\mathbf{x}) and 𝜹 = ∂ ℒ ∂ 𝐳 ⋆ . \bm{\delta}=\frac{\partial\mathcal{L}}{\partial\mathbf{z}^{\star}}. The exact implicit gradient is ∇ θ ℒ = 𝐏 ⊤ ( 𝐈 − 𝐉 ) − ⊤ 𝜹 , \nabla_{\theta}\mathcal{L}=\mathbf{P}^{\top}\left(\mathbf{I}-\mathbf{J}\right)^{-\top}\bm{\delta}, whereas the k k -step truncated BPTT gradient is ∇ ^ θ ( k ) ℒ = 𝐏 ⊤ ( ∑ j = 0 k − 1 ( 𝐉 ⊤ ) j ) 𝜹 . \widehat{\nabla}_{\theta}^{(k)}\mathcal{L}=\mathbf{P}^{\top}\left(\sum_{j=0}^{k-1}\left(\mathbf{J}^{\top}\right)^{j}\right)\bm{\delta}. Therefore, ‖ ∇ θ ℒ − ∇ ^ θ ( k ) ℒ ‖ 2 \displaystyle\left\|\nabla_{\theta}\mathcal{L}-\widehat{\nabla}_{\theta}^{(k)}\mathcal{L}\right\|_{2} ≤ ‖ 𝐏 ‖ 2 ‖ ( 𝐈 − 𝐉 ) − ⊤ − ∑ j = 0 k − 1 ( 𝐉 ⊤ ) j ‖ 2 ‖ 𝜹 ‖ 2 \displaystyle\leq\left\|\mathbf{P}\right\|_{2}\left\|\left(\mathbf{I}-\mathbf{J}\right)^{-\top}-\sum_{j=0}^{k-1}\left(\mathbf{J}^{\top}\right)^{j}\right\|_{2}\left\|\bm{\delta}\right\|_{2} ≤ ‖ 𝐏 ‖ 2 ( ∑ j = k ∞ ‖ 𝐉 ‖ 2 j ) ‖ 𝜹 ‖ 2 \displaystyle\leq\left\|\mathbf{P}\right\|_{2}\left(\sum_{j=k}^{\infty}\left\|\mathbf{J}\right\|_{2}^{j}\right)\left\|\bm{\delta}\right\|_{2} = ‖ 𝐏 ‖ 2 σ k 1 − σ ‖ 𝜹 ‖ 2 . \displaystyle=\left\|\mathbf{P}\right\|_{2}\frac{\sigma^{k}}{1-\sigma}\left\|\bm{\delta}\right\|_{2}. Hence, the truncated BPTT gradient error also decays exponentially with the number of backward passes k k . This completes the proof. ∎
## Appendix B A Toy Failure Mode for Recurrent Post-norm
In this section, we provide a toy example to showcase a failure mode of post-norm in a small setting. We take random 𝐱 , 𝐲 ∈ ℝ n × d \mathbf{x},\mathbf{y}\in\mathbb{R}^{n\times d} to be the input-output pair of sequences, with sequence length n = 100 n=100 and hidden-size d = 2 d=2 , and set 𝐳 0 0 = 𝐱 \mathbf{z}^{0}_{0}=\mathbf{x} . Let f θ ( 𝐳 , 𝐱 ) f_{\theta}(\mathbf{z};\mathbf{x}) be a neural network with a single sub-layer f θ 1 1 ( 𝐳 ) f_{\theta^{1}}^{1}(\mathbf{z}) defined as: f θ 1 1 ( 𝐳 ) = 𝐳 W ( 𝐰 ) , f_{\theta^{1}}^{1}(\mathbf{z})=\mathbf{z}W(\mathbf{w}), where we define the rank-one map: W ( 𝐰 ) = 𝐰𝟏 ⊤ ∈ ℝ 2 × 2 , 𝐰 = ( w 1 , w 2 ) ⊤ , W(\mathbf{w})=\mathbf{w}\mathbf{1}^{\top}\in\mathbb{R}^{2\times 2},\qquad\mathbf{w}=(w_{1},w_{2})^{\top}, with θ 1 = 𝐰 \theta^{1}=\mathbf{w} .
The post-norm recurrence is defined as 𝐳 i + 1 0 = Norm post ( 𝐳 i 0 + f θ 1 ( 𝐳 i 0 ) ) , \mathbf{z}_{i+1}^{0}=\mathrm{Norm}_{\mathrm{post}}\left(\mathbf{z}_{i}^{0}+f_{\theta^{1}}\left(\mathbf{z}_{i}^{0}\right)\right), while the pre-norm recurrence is defined as
𝐳 i + 1 0 = ( 1 − β ) ⋅ 𝐳 i 0 + β ⋅ f θ 1 ( Norm post ( 𝐳 i 0 ) ) , β = 1 2 . \mathbf{z}_{i+1}^{0}=\left(1-\beta\right)\cdot\mathbf{z}_{i}^{0}+\beta\cdot f_{\theta^{1}}\left(\mathrm{Norm}_{\mathrm{post}}\left(\mathbf{z}_{i}^{0}\right)\right),\qquad\beta=\frac{1}{2}.
Figure 10 gives a minimal version of the normalization trade-off discussed in the main text based on this toy model. For both models we sweep a 200 × 200 200\times 200 grid over 𝐰 ∈ 𝒢 = [ − 5 , 5 ] 2 \mathbf{w}\in\mathcal{G}=[-5,5]^{2} and plot
Δ ℒ ( 𝐰 ) = ℒ ( 𝐰 ) − min 𝐯 ∈ 𝒢 ℒ ( 𝐯 ) , \Delta\mathcal{L}(\mathbf{w})=\mathcal{L}(\mathbf{w})-\min_{\mathbf{v}\in\mathcal{G}}\mathcal{L}(\mathbf{v}), where we define the loss as ℒ ( 𝐰 ) = 1 n d ‖ 𝐳 20 0 ( 𝐰 ) − 𝐲 ‖ F 2 , \mathcal{L}(\mathbf{w})=\frac{1}{nd}\left\|\mathbf{z}_{20}^{0}(\mathbf{w})-\mathbf{y}\right\|_{F}^{2}, i.e., at the 20 t h 20^{th} effective layer.
The figure illustrates that boundedness does not imply trainability . Post-norm keeps the recurrent state bounded by construction: after every step, each row is projected back onto the unit sphere. However, the same projection also removes radial information at every iteration. In this two-parameter slice, the resulting loss is organized into thin angular sectors with sharp ridges and narrow low-loss regions. Thus a random initialization of 𝐰 \mathbf{w} is likely to start in a bounded but poorly conditioned part of the landscape, where the gradient does not point into a useful basin. This is the toy analogue of the optimization difficulty of recurrent post-norm layers.
The right panel is not bare pre-norm; it is pre-norm with residual scaling. This matters because naive pre-norm removes the projection that controls the recurrent state and can lead to activation growth ( Figure 2(b) ). With the scaled update, each row satisfies ‖ 𝐳 i + 1 0 ‖ 2 ≤ ( 1 − β ) ‖ 𝐳 i 0 ‖ 2 + β . \|\mathbf{z}_{i+1}^{0}\|_{2}\leq(1-\beta)\|\mathbf{z}_{i}^{0}\|_{2}+\beta. So the toy dynamics remain bounded while preserving a live residual stream. The broader low-loss region in Figure 10 is therefore consistent with the architecture we use in Theorem 1 : pre-normalization improves signal propagation, while residual scaling replaces the boundedness mechanism that post-normalization provided.
## Appendix C Further Details About the Architecture
##### Fixed-point solver.
Let 𝐳 i ∈ ℝ B × T × d \mathbf{z}_{i}\in\mathbb{R}^{B\times T\times d} denote the i t h i^{th} latent representation (with B B denoting the batch index, T T the sequence index, and d d the hidden size), and 𝐳 i + 1 \mathbf{z}_{i+1} the next latent representation. We index the b t h b^{th} batch dimension as 𝐳 i [ b ] \mathbf{z}_{i}[b] . Convergence is measured per sample 𝐳 i [ b ] \mathbf{z}_{i}[b] by the relative L ∞ L_{\infty} norm of the residual, 𝐫 i [ b ] = ‖ 𝐳 i + 1 [ b ] − 𝐳 i [ b ] ‖ ∞ ‖ 𝐳 i + 1 [ b ] ‖ ∞ + ϵ ∈ ℝ . \mathbf{r}_{i}[b]=\frac{\left\|\mathbf{z}_{i+1}[b]-\mathbf{z}_{i}[b]\right\|_{\infty}}{\left\|\mathbf{z}_{i+1}[b]\right\|_{\infty}+\epsilon}\in\mathbb{R}. A sample is declared converged when r i [ b ] < τ r_{i}[b]<\tau . In practice, we set τ \tau to 0.1 0.1 . But we observe that for a reasonably small choice of τ \tau , the model is not sensitive to this value. Two safeguards bound the loop: (1) a hard cap on the number of iterations, and (2) early termination if the adaptive step size collapses below a minimum.
##### Deep supervision.
We adopt a similar deep supervision mechanism as HRM ( Wang et al., 2025 ) and TRM ( Jolicoeur-Martineau, 2025 ) . Let T sup T_{\text{sup}} denote the deep supervision interval. After every T sup T_{\text{sup}} iterations, the intermediate activations of the model are decoded through the output head, the loss is computed, and truncated-BPTT is performed through the k k latest iterations. Then, the computation graph is detached from the previous step. For each sequence, this process is continued until the fixed-point of the input is reached. The number of backward passes per forward pass is therefore ⌈ k / T sup ⌉ \lceil k/T_{\text{sup}}\rceil . In our model, we set T sup = k T_{\text{sup}}=k , while in TRM and HRM, T sup T_{\text{sup}} is usually set to a larger number. However, we observe that in practice, FPRM performs a smaller number of forward and backward passes during training compared to TRM, lowering the training cost.
##### Depth-wise convolutions.
Depth-wise convolutions have proven effective in improving the performance of looped models, at a small time and parameter complexity ( Shu et al., 2026 ; Gao et al., 2025 ) . Therefore, in FPRM we apply depth-wise convolutions on the latent representations at the beginning of each loop, which we find to be most effective. An overview of FPRM is available in Figure 3 . We consider both 1D and 2D convolutions, and we find the 2D variant to be more effective at 2-dimensional tasks such as Sudoku and ARC, while the 1D variant is essential in state-tracking. However, as observed in Table 3 , depth-wise convolutions seem to have a detrimental impact on the performance of TRM.
## Appendix D Description of Figure 1
In this figure, we categorize the puzzles into three groups: easy, medium, and hard. The grouping is based on difficulty, which is measured by the number of empty cells in the puzzle. The sample sizes for each difficulty level are balanced and set at around 1000 samples. For FPRM, we mark the halting decision for the entire group based on the residual of the group: if the mean residual is smaller than a pre-determined threshold (set to 0.1), then the model makes the halting decision. For TRM, the halting decision is marked when the ACT module signals halting for more than half of the samples. For the sake of exposition, we exclude the hardest puzzles from the groups, since they fail to halt at the current max. set budget of 10000 effective layers.
## Appendix E Fixed-point Residuals and Halting
We provide the test accuracy and the fixed-point residuals achieved by FPRM as a function of effective layer in Figure 12 . The residuals for more difficult problems decay at a much slower rate, indicating that they demand more compute. Furthermore, accuracy stops improving at roughly the same effective layer where the residual plateaus, supporting the use of fixed-points as a halting criterion.
## Appendix F How to effectively spend loops in TRM?