Solve the Loop: Attractor Models for Language and Reasoning β€” part 3/4

Cleaned-text code points [31096, 46518). Bibliography omitted here; source.md includes it.

Fixed-point convergence. Figure 6 visualizes convergence in two complementary ways. First, we project the trajectory of the final-position representation onto its first two principal components over 16 iterations. Our method rapidly contracts to a fixed point: iterations 8–16 collapse onto a single attractor. Parcae also moves toward a stable state, but does so more slowly; its trajectory remains noisier and continues to drift through later recurrences. This suggests that while finite-loop language models can exhibit fixed-point-like behavior, explicitly training the loop as a fixed-point problem produces faster and cleaner convergence. Second, we track the number of solver iterations required during training. The DEQ baseline requires increasingly many iterations as optimization progresses, consistent with the observations in the original work ( Bai et al., 2019 ) . In contrast, Attractor Models rapidly converge to the minimum iteration count and remain stable. This behavior provides evidence for equilibrium internalization, where optimization shifts work from the iterative solver into the backbone proposal.

Test-time iterations vs. quality. In Figure 7 , we report validation perplexity, CORE accuracy, and CORE-Extended accuracy as we vary the number of inference iterations T T while holding training fixed. For Parcae, quality improves monotonically from T = 1 T=1 until it plateaus near T = 8 T=8 , indicating that the model relies on repeated test-time refinement. In contrast, our method reaches peak performance at T = 1 T=1 at every scale, and even T = 0 T=0 (decoding the backbone proposal y ~ 0 \tilde{y}_{0} directly through the tied unembedding) is already at or near the converged value.

We attribute this behavior to equilibrium internalization : during training, the backbone learns to produce a proposal y ~ 0 \tilde{y}_{0} that already lies close to the fixed point that the solver would otherwise compute through iteration. Thus, the iterative block still shapes the learned representation during training, but at inference time the model has largely internalized the result of that refinement into its backbone initialization. Strikingly, the backbone-only readout at T = 0 T=0 is already stronger than larger standard Transformers trained without attractor assistance, and matches or exceeds finite-loop baselines that require many test-time recurrences. Concretely, our 140M model at T = 0 T=0 (no solver) matches or improves Parcae 140M at T = 8 T=8 recurrent steps; the same pattern holds at 370M and 770M.

### 4.4 Ablations

For the ablations, we use a 60.3M-parameter version of our model trained on 1B tokens of FineWeb-Edu under the same nanochat configuration. All ablation runs share the same data stream, tokenizer, and optimizer; only the ablated component varies.

We isolate the contribution of (i) the location of the equilibrium and its initialization (Table 3 ), (ii) the additive injection of c c (Table 4 ), (iii) and the one-step backward approximation (Table 5 ). All ablations train a 60.3M-parameter model on 1B tokens of FineWeb-Edu under the same nanochat setup, with all other hyperparameters fixed.

Comparison to DEQ. Table 3 compares Attractor Models against a parameter-matched DEQ ( Bai et al., 2019 ) . The DEQ baseline solves for a hidden-state equilibrium initialized from an uninformative state and decodes it with a separate output head. Tying the unembedding closes part of the gap (Val. PPL 42.18 β†’ 38.74 42.18\to 38.74 ), but the largest improvement comes from matching the equilibrium to the design of Attractor Models: the fixed point is placed directly in the tied output-embedding space and the root finder is initialized from the backbone proposal y ~ 0 = 𝒯 ΞΈ b ​ ( E ⁑ ( x ) ) \tilde{y}_{0}=\mathcal{T}_{\theta_{b}}(E(x)) ( β†’ 34.05 \to 34.05 ).

This proposal gives the solver a meaningful, input-dependent starting point rather than requiring the recurrent block to construct the representation from scratch. As a result, the Attractor Model reaches the same residual tolerance in 1.7 Γ— 1.7\times fewer iterations while also achieving lower perplexity. We view this as evidence of equilibrium internalization : the backbone learns to produce an output-side proposal y ~ 0 \tilde{y}_{0} that already lies close to the eventual equilibrium y ~ ⋆ \tilde{y}^{\star} , making the subsequent attractor refinement both easier and faster.

Proposal injection. Table 4 ablates how the backbone proposal y ~ 0 \tilde{y}_{0} is provided to the attractor. We use proposal injection to denote how y ~ 0 \tilde{y}_{0} enters the recurrent refinement map, and persistent injection to denote providing y ~ 0 \tilde{y}_{0} at every refinement step rather than only through the initial state.

When the proposal is used only for initialization, y ~ t = 0 = y ~ 0 \tilde{y}_{t=0}=\tilde{y}_{0} and y ~ t + 1 = 𝒯 ΞΈ a ​ ( y ~ t ) \tilde{y}_{t+1}=\mathcal{T}_{\theta_{a}}(\tilde{y}_{t}) , the recurrent update no longer depends on the input after the first state. Consequently, the fixed point can become proposal-independent, and only 12.4% of validation tokens converge within T max T_{\max} . Persistent injection by concatenation, y ~ t + 1 = 𝒯 ΞΈ a ​ ( [ y ~ t ; y ~ 0 ] ) \tilde{y}_{t+1}=\mathcal{T}_{\theta_{a}}([\tilde{y}_{t};\tilde{y}_{0}]) , restores proposal-dependence and recovers much of the quality, but makes the refinement problem harder: convergence is slower (11.2 vs. 8.4 average iterations) and perplexity is worse (36.81 vs. 34.05). Additive proposal injection, y ~ t + 1 = 𝒯 ΞΈ a ​ ( y ~ t , y ~ 0 ) \tilde{y}_{t+1}=\mathcal{T}_{\theta_{a}}(\tilde{y}_{t},\tilde{y}_{0}) , provides the backbone proposal at every step while keeping the refinement map simple and well-conditioned, yielding the best results across all three metrics.

Backward pass. In table 5 , we compare the one-step backward approximation against full implicit differentiation (Anderson on the linear system ( I βˆ’ J y ~ ⊀ ) ​ u = v (I-J_{\tilde{y}}^{\top})u=v ) and the phantom gradient ( Geng et al., 2022 ) unroll. Full IFT improves PPL by only 0.14 while increasing peak training memory by 4.8 Γ— 4.8\times and step time by 2.7 Γ— 2.7\times ; phantom gradient lies in between. The one-step approximation allows relatively cheap training cost while maintaining nearly all of the original performance of a full backward gradient computation.

Initialization ablation. Table 6 isolates the effect of the solver initialization. Starting the attractor solve from an uninformative state, either zero or Gaussian noise, substantially degrades both quality and convergence. In contrast, initializing from the backbone proposal y ~ 0 = 𝒯 ΞΈ b ​ ( E ⁑ ( x ) ) \tilde{y}_{0}=\mathcal{T}_{\theta_{b}}(E(x)) gives the solver a meaningful output-side starting point, yielding lower perplexity, fewer iterations, and higher downstream accuracy. This supports our hypothesis that the refinement should begin near a coherent prediction embedding rather than constructing one from scratch.

## 5 Conclusion and Future Work

In this work, we propose Attractor Models, a new family of architectures that first produce meaningful prediction embeddings and then refine them through an attractor module by solving for a fixed point. This formulation makes recurrent refinement stable, memory-efficient, and adaptive, while avoiding the cost of explicit unrolling and achieving strong results across both large-scale language modeling and hard reasoning with tiny models. In stark contrast to prior looped language models, training Attractor Models gives rise to equilibrium internalization: the model learns to make the very refinement procedure that trained it largely unnecessary at inference time. Interesting directions for future work is to further study the equilibrium internalization phenomenon and the differences between Attractor Models and finite-loop recurrence.

Acknowledgements. The authors gratefully acknowledge generous support from Coefficient Giving.

## Appendix A Related Work

Looped and recurrent language models. Weight-tied recurrence has re-emerged as an alternative to deeper feed-forward stacks ( Press and Wolf, 2017 ; Inan et al., 2017 ; Kaiser and Sutskever, 2016 ; Lan et al., 2020 ; Dabre and Fujita, 2018 ; Takase and Kiyono, 2023 ; Bae et al., 2025a ) . Universal Transformers share a single block across depth ( Dehghani et al., 2019 ; Sapunov, 2026a ; Sapunov, 2026b ) , while looped and recurrent language models iterate a recurrent update on a hidden state to enable latent reasoning ( Zhu et al., 2025c ; Prairie et al., 2026 ; Geiping et al., 2025a ; Bae et al., 2025b ; Zheng et al., ; Zhang et al., ; Song et al., 2026a ; Knupp et al., 2026a ; Zeitoun et al., 2026a ; McLeish et al., ; Zheng et al., 2026 ; Zhang et al., 2026 ; Song et al., 2026b ; Moosa et al., 2026 ; Knupp et al., 2026b ; Wu et al., 2025 ; Zeitoun et al., 2026b ; Pappone et al., 2025 ; Rauba et al., 2026 ; Zeng et al., 2026 ; Tur et al., 2026 ; Komisarczyk et al., 2026 ; Ghugare et al., 2026 ; Cameron et al., 2026 ; Williams and Tureci, 2026 ; Han, 2026 ; Goyal et al., 2026 ; Tang et al., 2026 ) . Latent (implicit) reasoning methods such as Coconut ( Hao et al., 2025 ) and related methods ( Teoh et al., 2025 ; Chen et al., 2025 ; Xu and Sato, 2026 ; Jeddi et al., 2026 ; Fu et al., 2026 ; Zhou et al., 2025 ; Geiping et al., 2025b ; Hu et al., 2025 ; Fleuret, 2025 ) perform reasoning in continuous representation space rather than through chain-of-thought tokens. These approaches unroll the loop for a fixed number of steps at training time, causing training memory to grow linearly in depth, coupling inference iterations to training, and often degrading quality when iterations are extended at test time ( Zhu et al., 2025c ) . From theoretical and mechanistic analysis perspectives, looped and continuous thinking models have shown to offer benefits over non-recurrent models ( Yang et al., 2024a ; Labovich, 2026b ; Saunshi et al., 2025a ; Xu and Sato, 2025a ; Saunshi et al., 2025b ; Giannou et al., 2023b ; Yang et al., 2024c ; Gatmiry et al., 2024b ; Huang et al., 2025 ; Xu and Sato, 2025b ; Merrill and Sabharwal, 2025 ; Labovich, 2026a ; Zhu et al., 2025b ; Zhu et al., 2025a ; Gatmiry et al., 2024a ; Merrill and Sabharwal, 2026 ) . Mechanistic analysis shows that the recurrent updates of looped LMs in fact converge to a fixed point for the vast majority of tokens, directly motivating our formulation ( Blayney et al., 2026 ) .

Implicit fixed-point models. Deep Equilibrium Models (DEQs) replace finite unrolling with a fixed-point equation z ⋆ = f ΞΈ ​ ( z ⋆ , x ) z^{\star}=f_{\theta}(z^{\star},x) and train through it via implicit differentiation, decoupling effective depth from training memory ( Bai et al., 2019 ) . Solvers typically use Anderson acceleration ( Anderson, 1965 ) , with backward passes ranging from full implicit differentiation to cheap surrogates such as one-step ( Fung et al., 2021 ) and phantom ( Geng et al., 2022 ) gradients. Standard DEQs solve for a hidden state initialized from zero and decode through a separate output head. In Attractor Models, the equilibrium instead lives in the tied output-embedding space and is warm-started from a meaningful backbone prediction; we find this structure essential for stable training at language-modeling scale and show it gives rise to equilibrium internalization (Section 4.3 ).

Tiny recursive reasoners. On small-data algorithmic benchmarks such as Sudoku and maze solving, hierarchical and tiny recursive networks (HRM, TRM) ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) achieve strong accuracy with few parameters, but exhibit a β€œless is more” behavior where performance collapses as model size grows ( Jolicoeur-Martineau, 2025 ) . Attractor Models retain the iterative-refinement benefit of these architectures while scaling cleanly with parameter count).

Positioning. Looped LMs couple three quantities through unrolled BPTT: inference depth, training depth, and training memory ( Zhu et al., 2025c ) . Attractor Models decouple all three: the equilibrium is defined by a residual equation in output-embedding space, the number of solver evaluations is chosen adaptively by tolerance Ξ΅ \varepsilon , and training memory in the recurrent block is constant in effective depth. Empirically, this combines the strengths of feed-forward and recurrent modelsβ€”better perplexity than parameter-matched Transformers, 25–31% lower training FLOPs and constant memory relative to looped LMs, and clean scaling on hard reasoning where specialized recursive networks collapse.

## Appendix B Theory of Looped and Attractor Models

### B.1 Well-posedness and the implicit gradient

Fix an input x x and parameters ΞΈ \theta . Throughout this section, we write F ⁑ ( y ) β‰œ f ΞΈ ​ ( y , E ⁑ ( x ) ) F(y)\triangleq f_{\theta}(y,E(x)) for brevity, so that the fixed-point equation y ⋆ = f ΞΈ ​ ( y ⋆ , E ⁑ ( x ) ) y^{\star}=f_{\theta}(y^{\star},E(x)) becomes y ⋆ = F ⁑ ( y ⋆ ) y^{\star}=F(y^{\star}) . Let J F ​ ( y ) β‰œ βˆ‚ F / βˆ‚ y J_{F}(y)\triangleq\partial F/\partial y denote the Jacobian of F F with respect to its state input. We work in a generic norm βˆ₯ β‹… βˆ₯ \|\cdot\| on ℝ L Γ— d \mathbb{R}^{L\times d} and its induced operator norm; for concreteness, can take both to be Frobenius and spectral, respectively. Assumptions are borrowed from ( Bai et al., 2020 ; Krantz and Parks, 2002a ) .

###### Assumption 1 (Local contraction) .

There exist a point y Β― ∈ ℝ L Γ— d \bar{y}\in\mathbb{R}^{L\times d} , a radius r > 0 r>0 , and a constant L ∈ [ 0 , 1 ) L\in[0,1) such that 1. F F maps the closed ball B r ​ ( y Β― ) β‰œ { y : β€– y βˆ’ y Β― β€– ≀ r } B_{r}(\bar{y})\triangleq\{y:\|y-\bar{y}\|\leq r\} into itself, and

2. F F is L L -Lipschitz on B r ​ ( y Β― ) B_{r}(\bar{y}) : β€– F ⁑ ( y ) βˆ’ F ⁑ ( y β€² ) β€– ≀ L ​ β€– y βˆ’ y β€² β€– for all ​ y , y β€² ∈ B r ​ ( y Β― ) . \|F(y)-F(y^{\prime})\|\;\leq\;L\,\|y-y^{\prime}\|\qquad\text{for all }y,y^{\prime}\in B_{r}(\bar{y}).

When F F is continuously differentiable, a sufficient form of (ii) is sup y ∈ B r ​ ( y Β― ) β€– J F ​ ( y ) β€– ≀ L \sup_{y\in B_{r}(\bar{y})}\|J_{F}(y)\|\leq L .

###### Theorem 1 (Well-posedness and implicit gradient) .

Under Assumption 1 , the following hold. 1. Existence and uniqueness: There is a unique y ⋆ ∈ B r ​ ( y Β― ) y^{\star}\in B_{r}(\bar{y}) with y ⋆ = F ⁑ ( y ⋆ ) y^{\star}=F(y^{\star}) .

2. The picard iteration converges linearly: For any y 0 ∈ B r ​ ( y Β― ) y_{0}\in B_{r}(\bar{y}) , the iterates y k + 1 = F ⁑ ( y k ) y_{k+1}=F(y_{k}) remain in B r ​ ( y Β― ) B_{r}(\bar{y}) and satisfy β€– y k βˆ’ y ⋆ β€– ≀ L k ​ β€– y 0 βˆ’ y ⋆ β€– . \|y_{k}-y^{\star}\|\;\leq\;L^{k}\,\|y_{0}-y^{\star}\|. (5)

3. Validity of the implicit gradient: If F F is continuously differentiable on B r ​ ( y Β― ) B_{r}(\bar{y}) , then I βˆ’ J F ​ ( y ⋆ ) I-J_{F}(y^{\star}) is invertible, y ⋆ y^{\star} depends continuously differentiably on ΞΈ \theta in a neighborhood of the current parameters, and βˆ‚ y ⋆ βˆ‚ ΞΈ = ( I βˆ’ J F ​ ( y ⋆ ) ) βˆ’ 1 ​ βˆ‚ f ΞΈ βˆ‚ ΞΈ | y ⋆ . \frac{\partial y^{\star}}{\partial\theta}\;=\;\bigl(I-J_{F}(y^{\star})\bigr)^{-1}\left.\frac{\partial f_{\theta}}{\partial\theta}\right|_{y^{\star}}. (6)

###### Proof.

Parts (i) and (ii) follow from the Banach fixed-point theorem applied to the contraction F F on the closed (hence complete) ball B r ( y Β― ) βŠ‚ ( ℝ L Γ— d , βˆ₯ β‹… βˆ₯ ) B_{r}(\bar{y})\subset(\mathbb{R}^{L\times d},\|\cdot\|) .