Solve the Loop: Attractor Models for Language and Reasoning — part 4/4
Cleaned-text code points [46518, 56626). Bibliography omitted here; source.md includes it.
For (iii), Lipschitz continuity with constant L L gives ‖ J F ( y ⋆ ) ‖ ≤ L < 1 \|J_{F}(y^{\star})\|\leq L<1 , so the spectral radius satisfies ρ ( J F ( y ⋆ ) ) ≤ ‖ J F ( y ⋆ ) ‖ < 1 \rho(J_{F}(y^{\star}))\leq\|J_{F}(y^{\star})\|<1 . The Neumann series ∑ k ≥ 0 J F ( y ⋆ ) k \sum_{k\geq 0}J_{F}(y^{\star})^{k} therefore converges to ( I − J F ( y ⋆ ) ) − 1 (I-J_{F}(y^{\star}))^{-1} , and in particular I − J F ( y ⋆ ) I-J_{F}(y^{\star}) is invertible. Define G ( y , θ ) ≜ f θ ( y , E ( x ) ) − y G(y,\theta)\triangleq f_{\theta}(y,E(x))-y , which is C 1 C^{1} in both arguments and satisfies G ( y ⋆ , θ ) = 0 G(y^{\star},\theta)=0 with
∂ G ∂ y | y ⋆ = J F ( y ⋆ ) − I , \left.\frac{\partial G}{\partial y}\right|_{y^{\star}}\;=\;J_{F}(y^{\star})-I, (7)
which is invertible. The implicit function theorem applied to G = 0 G=0 yields a unique C 1 C^{1} map θ ↦ y ⋆ ( θ ) \theta\mapsto y^{\star}(\theta) in a neighborhood of the current parameters, with
∂ y ⋆ ∂ θ = − ( ∂ G ∂ y ) − 1 ∂ G ∂ θ = ( I − J F ( y ⋆ ) ) − 1 ∂ f θ ∂ θ | y ⋆ . \frac{\partial y^{\star}}{\partial\theta}\;=\;-\left(\frac{\partial G}{\partial y}\right)^{-1}\frac{\partial G}{\partial\theta}\;=\;\bigl(I-J_{F}(y^{\star})\bigr)^{-1}\left.\frac{\partial f_{\theta}}{\partial\theta}\right|_{y^{\star}}. (8)
∎
Equation ( 6 ) is exactly the gradient computed in the backward pass of Section 3 . The linear solve is convergent for the same reason: since ‖ J F ⊤ ( y ⋆ ) ‖ = ‖ J F ( y ⋆ ) ‖ ≤ L < 1 \|J_{F}^{\top}(y^{\star})\|=\|J_{F}(y^{\star})\|\leq L<1 , the Neumann iteration u ← J F ⊤ ( y ⋆ ) u + v u\leftarrow J_{F}^{\top}(y^{\star})\,u+v converges geometrically with rate L L to the unique solution u = ( I − J F ⊤ ( y ⋆ ) ) − 1 v u=(I-J_{F}^{\top}(y^{\star}))^{-1}v .
### B.2 Looped language models are fixed-point iterators
A LoopLM ( Zhu et al., 2025c ) of depth T T with the same weight-tied block f θ f_{\theta} and warm start y 0 = E ( x ) y_{0}=E(x) produces y T loop ≜ F ( T ) ( E ( x ) ) = F ∘ F ∘ ⋯ ∘ F ⏟ T times ( E ( x ) ) . y_{T}^{\mathrm{loop}}\;\triangleq\;F^{(T)}\bigl(E(x)\bigr)\;=\;\underbrace{F\circ F\circ\cdots\circ F}_{T\text{ times}}\bigl(E(x)\bigr). (9)
This is exactly T T Picard iterations of our residual g θ ( ⋅ , x ) g_{\theta}(\cdot,x) warm-started from the input embedding. See the following corollary:
###### Corollary 1 (LoopLM as a truncated approximation) .
Suppose Assumption 1 holds and the warm start E ( x ) ∈ B r ( y ¯ ) E(x)\in B_{r}(\bar{y}) . Then for every T ≥ 0 T\geq 0 , ‖ y T loop − y ⋆ ‖ ≤ L T ‖ E ( x ) − y ⋆ ‖ . \|y_{T}^{\mathrm{loop}}-y^{\star}\|\;\leq\;L^{T}\,\|E(x)-y^{\star}\|. (10) In particular, y T loop → y ⋆ y_{T}^{\mathrm{loop}}\to y^{\star} as T → ∞ T\to\infty , and the discrepancy between a LoopLM of depth T T and our model decays geometrically in T T .
###### Proof.
Apply Theorem 1 (ii) with y 0 = E ( x ) y_{0}=E(x) . ∎
Two consequences are worth highlighting:
The limit: Our fixed-point model is the T → ∞ T\to\infty limit of LoopLM with shared parameters. Training a LoopLM at any finite depth T T implicitly trains an approximation to the same equilibrium y ⋆ y^{\star} , with the approximation error controlled by ( 10 ).
The iteration count at inference: Picard iteration converges at the linear rate L L . Our forward pass uses Anderson acceleration, which under standard regularity conditions converges superlinearly near y ⋆ y^{\star} ( Anderson, 1965 ) . To reach a target residual tolerance ε \varepsilon , the root finder therefore typically requires fewer evaluations of f θ f_{\theta} than the unroll depth T T a LoopLM would need for a comparable residual. The exact crossover depends on L L and on the solver hyperparameters.
Caveat: Assumption 1 is local and is not guaranteed to hold for an arbitrarily trained transformer block. In our experiments, we generally notice that ‖ J F ( y ⋆ ) ‖ ≤ L < 1 \|J_{F}(y^{\star})\|\leq L<1 , the authors of DEQ also find that using a regularizer on the Jacobian to enforce this helps ( Bai et al., 2021 ) .
### B.3 Where looped language models fall short
#### B.3.1 Attractor Models
The initialization y ~ 0 \tilde{y}_{0} and the fixed point y ~ ⋆ \tilde{y}^{\star} live in the same tied output-embedding space, so either can be decoded by the unembedding E ⊤ E^{\top} . The loss depends only on y ~ ⋆ \tilde{y}^{\star} , and the implicit gradient with respect to the backbone parameters is ∇ θ b ℒ = u ⊤ J y ~ 0 ∇ θ b y ~ 0 , u = ( I − J y ~ ⊤ ) − 1 v . \nabla_{\theta_{b}}\mathcal{L}\;=\;u^{\top}J_{\tilde{y}_{0}}\,\nabla_{\theta_{b}}\tilde{y}_{0},\qquad u=(I-J_{\tilde{y}}^{\top})^{-1}v. Thus, the loss may be viewed as a function of the backbone’s output embedding: perturbations in y ~ 0 \tilde{y}_{0} affect ℒ \mathcal{L} by perturbing y ~ ⋆ \tilde{y}^{\star} in the same space. The backbone is therefore optimized as a standalone next-token predictor whose embedding is decoded by E ⊤ E^{\top} .
When the backbone has strictly greater capacity than the weight-tied attractor, the joint optimum places the prediction work in the backbone. Consequently, y ~ 0 ( x , θ b ⋆ ) \tilde{y}_{0}(x;\theta_{b}^{\star}) approaches the loss-minimizing embedding. The fixed-point constraint 𝒯 θ a ⋆ ( y ~ ⋆ , y ~ 0 ) = y ~ ⋆ \mathcal{T}_{\theta_{a}^{\star}}(\tilde{y}^{\star},\tilde{y}_{0})=\tilde{y}^{\star} is then consistent only when y ~ ⋆ = y ~ 0 \tilde{y}^{\star}=\tilde{y}_{0} , yielding 𝒯 θ a ⋆ ( y ~ 0 , y ~ 0 ) = y ~ 0 . \mathcal{T}_{\theta_{a}^{\star}}(\tilde{y}_{0},\tilde{y}_{0})=\tilde{y}_{0}.
#### B.3.2 Standard looped LMs
In standard looped language models, the latent initialization h 0 h_{0} is uninformative, typically zero or noise, and occupies the role of a hidden state rather than an output embedding. Decoding h 0 E ⊤ h_{0}E^{\top} is therefore not meaningful.
Gradients instead flow by backpropagation through time along the unrolled trajectory ( h 0 , … , h T ) (h_{0},\dots,h_{T}) . There is no term in this gradient that drives h 0 h_{0} toward the loss-bearing final state h T h_{T} in embedding distance: the initialization is fixed by the architecture and is not produced by a separately trained predictor. ∎
#### B.3.3 Implicit-Gradient Barrier
The implicit gradient ∇ θ ℒ ∞ = v ⊤ ( I − J g ⊤ ) − 1 ∂ θ g \nabla_{\theta}\mathcal{L}_{\infty}=v^{\top}(I-J_{g}^{\top})^{-1}\partial_{\theta}g contains the factor ( I − J g ) − 1 (I-J_{g})^{-1} . This inverse is undefined when 1 ∈ σ ( J g ) 1\in\sigma(J_{g}) , and its operator norm scales as ‖ ( I − J g ) − 1 ‖ ∼ dist ( 1 , σ ( J g ) ) − 1 \|(I-J_{g})^{-1}\|\sim\mathrm{dist}(1,\sigma(J_{g}))^{-1} near such a singularity.
Along any continuous gradient-descent path, the eigenvalues of J g ( y ⋆ , θ ) J_{g}(y^{\star};\theta) vary continuously. Therefore, for the dominant eigenvalue to leave the unit disk through + 1 +1 —the canonical loss-of-contraction route for residual iteration maps—it must approach 1 1 . This forces ‖ ( I − J g ) − 1 ‖ → ∞ \|(I-J_{g})^{-1}\|\to\infty .
Unless v v is orthogonal to the offending eigendirection, a codimension-one and hence non-generic condition, the gradient norm diverges. The descent step therefore blows up before the boundary can be crossed, confining θ \theta to the contractive region { ρ ( J g ) < 1 } \{\rho(J_{g})<1\} .
For exits through complex eigenvalues, the same conclusion holds under the one-step JFB approximation, since the truncated Neumann series ∑ k ( J g ⊤ ) k \sum_{k}(J_{g}^{\top})^{k} diverges as ρ ( J g ) → 1 \rho(J_{g})\to 1 . ∎
#### B.3.4 Interpretation
Fixed-loop training has no inherent mechanism that favors contractive iterations. An optimum in which g θ g_{\theta} perturbs the embedding for K K steps and only happens to land accurately at step K K is a valid fixed-loop solution. These are precisely the solutions that fail when extra loops are run at inference.
Equilibrium training cannot reach such solutions, because the implicit gradient creates a barrier of diverging gradients around non-contractive regions. As a result, the trajectory of θ \theta is confined to the regime in which the solver converges, the one-step gradient is a descent direction, and additional iterations remain stable.
The mechanistic observation ( Blayney et al., 2026 ) that fixed points emerge only late in standard looped training is consistent with this picture: the loss landscape contains a basin near contractive solutions, but only equilibrium training applies pressure toward that basin from the beginning of training.
## Appendix C Hyperparamter and Experimental Settings
We use NVIDIA H200 GPUs for all of our experiments.
† \dagger At 770M, Attractor Model uses AdamW LR 5 × 10 − 3 5\times 10^{-3} and Muon LR 6 × 10 − 3 6\times 10^{-3} ; Transformer/Parcae use 6 × 10 − 3 6\times 10^{-3} / 8 × 10 − 3 8\times 10^{-3} . ‡ \ddagger Gradient checkpointing enabled for 770M Attractor Model and Parcae; disabled for Transformer. ⋆ \star torch.compile enabled for Transformer and Parcae; disabled for Attractor Model (implicit-gradient hooks are not compile-compatible).
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .