Solve the Loop: Attractor Models for Language and Reasoning — part 2/4
Cleaned-text code points [15912, 31096). Bibliography omitted here; source.md includes it.
The attractor module refines the output embedding. The attractor module is a separate weight-tied refinement network 𝒯 θ a \mathcal{T}_{\theta_{a}} . Starting from the backbone proposal y ~ 0 \tilde{y}_{0} , it repeatedly refines the output embedding according to y ~ t + 1 = 𝒯 θ a ( y ~ t , y ~ 0 ) , where y ~ 0 = 𝒯 θ b ( x ~ ) . \displaystyle\tilde{y}_{t+1}=\mathcal{T}_{\theta_{a}}(\tilde{y}_{t},\tilde{y}_{0}),\;\;\text{where}\;\;\tilde{y}_{0}=\mathcal{T}_{\theta_{b}}(\tilde{x}). (2) Here, we persistently inject the initial guess y ~ 0 \tilde{y}_{0} at every refinement step. This persistent injection keeps the attractor proposal-dependent and prevents it from collapsing to a proposal-independent fixed point. We ablate this conditioning mechanism in Section 4 . Importantly, we warm-start the attractor module at an informative proposal y ~ 0 \tilde{y}_{0} , in contrast to existing work that initialize the recurrent state at uninformative values such as zero or Gaussian noise ( Geiping et al., 2025a ; Prairie et al., 2026 ) ; see Table 6 for a comparison.
Rather than rolling out recurrent steps to reach a fixed point, we directly solve for the convergence: 𝒜 θ a ( y ~ ⋆ , y ~ 0 ) ≔ 𝒯 θ a ( y ~ ⋆ , y ~ 0 ) − y ~ ⋆ = 0 ⇒ y ~ ⋆ = RootFind ( 𝒜 θ a ( ⋅ , y ~ 0 ) , y 0 ) . \displaystyle\mathcal{A}_{\theta_{a}}(\tilde{y}^{\star},\tilde{y}_{0})\coloneqq\mathcal{T}_{\theta_{a}}(\tilde{y}^{\star},\tilde{y}_{0})-\tilde{y}^{\star}=0\;\Rightarrow\;\tilde{y}^{\star}\;=\;\texttt{RootFind}\!\left(\mathcal{A}_{\theta_{a}}(\cdot,\tilde{y}_{0});y_{0}\right). (3) In the forward pass, we compute this equilibrium with a root finder initialized at the backbone proposal. In our implementation, the RootFind algorithm uses Anderson acceleration, which combines a small window of past iterates and residuals to reach the fixed point faster than plain recursion. The solver exits when ‖ 𝒜 θ a ( y ~ t , y ~ 0 ) ‖ 2 / ‖ y ~ t ‖ 2 < ε {\|\mathcal{A}_{\theta_{a}}(\tilde{y}_{t},\tilde{y}_{0})\|_{2}}/{\|\tilde{y}_{t}\|_{2}}<\varepsilon or after T max T_{\max} steps. Thus, the computation is controlled by the convergence of the residual rather than by a learned halting head or a preset loop count. In contrast to fixed unrolling, the number of refinement steps can therefore vary at inference time without changing the model. Finally, the equilibrium embedding is decoded with the tied unembedding.
Parameters of the Attractor Models consist of the tied embedding/unembedding matrices, the backbone module, and parameters of the attractor module: θ ≔ ( θ a , θ b , E ) \theta\coloneqq(\theta_{a},\theta_{b},E) . Compared to looped models, Attractor Models change both the starting point and the endpoint of recurrence: we initialize the loop from the output guess from the backbone network y ~ 0 \tilde{y}_{0} , and the decoded state is the attractor y ~ ⋆ \tilde{y}^{\star} rather than a finite unroll.
### 3.2 Training and Inference of Attractor Models
We now describe the training procedure for Attractor Models. We first explain how to differentiate through the fixed-point solver using implicit differentiation, and then show how the model is optimized with the standard cross-entropy language-modeling loss applied to the output y ⋆ y^{\star} .
Backward pass and implicit differentiation. Because Attractor Models define the output embedding as the solution to a fixed-point equation, we differentiate through the equilibrium using the implicit function theorem ( Krantz and Parks, 2002b ) . Let ℒ \mathcal{L} denote the training loss and let v = ∂ ℒ / ∂ y ~ ⋆ v=\partial\mathcal{L}/\partial\tilde{y}^{\star} . Applying the implicit function theorem to 𝒜 θ a ( y ~ ⋆ , y ~ 0 ) = 0 \mathcal{A}_{\theta_{a}}(\tilde{y}^{\star},\tilde{y}_{0})=0 gives ∂ ℒ ∂ θ = u ⊤ ∂ 𝒯 θ a ( y ~ ⋆ , y ~ 0 ) ∂ θ , u = ( I − J y ~ ⊤ ) − 1 v , where J y ~ = ∂ 𝒯 θ a ( y ~ , y ~ 0 ) ∂ y ~ | y ~ = y ~ ⋆ . \displaystyle\frac{\partial\mathcal{L}}{\partial\theta}=u^{\top}\frac{\partial\mathcal{T}_{\theta_{a}}(\tilde{y}^{\star},\tilde{y}_{0})}{\partial\theta},\quad u=\left(I-J_{\tilde{y}}^{\top}\right)^{-1}v,\quad\text{where}\quad J_{\tilde{y}}=\left.\frac{\partial\mathcal{T}_{\theta_{a}}(\tilde{y},\tilde{y}_{0})}{\partial\tilde{y}}\right|_{\tilde{y}=\tilde{y}^{\star}}. (4) The derivative with respect to θ = ( θ a , θ b , E ) \theta=(\theta_{a},\theta_{b},E) includes the direct dependence on the attractor parameters θ a \theta_{a} , as well as the dependence on the initialization y ~ 0 = 𝒯 θ b ( E ( x ) ) \tilde{y}_{0}=\mathcal{T}_{\theta_{b}}(E(x)) through the backbone parameters θ b \theta_{b} and the tied embedding parameters E E .
Following prior work on implicit models ( Geng et al., 2022 ; Fung et al., 2021 ) , we use the one-step approximation u ≈ v u\approx v . This avoids the extra linear solve for u u and reduces the backward pass to one vector–Jacobian product through 𝒜 θ a \mathcal{A}_{\theta_{a}} . Since we do not backpropagate through every solver step, memory in the attractor block does not grow with the number of forward iterations. In Section 4 , we show that the Anderson solver yields only marginal quality gains, whereas the one-step approximation enables substantially cheaper training.
Remark. Attractor Models are implicit equilibrium models in the spirit of DEQs ( Bai et al., 2019 ) , but the equilibrium plays a different role. Classical DEQs replace the prediction network with a hidden-state equilibrium z ⋆ z^{\star} (single-layer), decoded with a separate output head. We instead keep a standard causal Transformer backbone and add an equilibrium refinement block on top of its prediction state. The fixed point lives directly in the tied embedding space, so every iterate { y 0 , y 1 , … , y ⋆ } \{y_{0},y_{1},\ldots,y^{\star}\} is already a representation in the output space that can be decoded. This gives three practical differences: (i) the solver is initialized from a semantically meaningful proposal y ~ 0 \tilde{y}_{0} rather than from an uninformative state such as zero (as in DEQ), (ii) inference can stop according to a residual tolerance ε \varepsilon rather than a fixed depth or learned halting head, and (iii) DEQ shows that scaling the number of DEQ blocks can harm the performance of their method, whereas we allow for an arbitrary depth backbone transformer and show that we can use a variable number of solver blocks.
Training objective and inference. We train Attractor Models with the standard next-token prediction cross-entropy objective applied to the fixed-point output y ⋆ y^{\star} . Inference reuses the same equilibrium computation. Given an input sequence x x , the backbone first produces the proposal y ~ 0 = 𝒯 θ b ( E ( x ) ) \tilde{y}_{0}=\mathcal{T}_{\theta_{b}}(E(x)) , the attractor solver computes y ~ ⋆ \tilde{y}^{\star} , and the tied unembedding decodes y ~ ⋆ \tilde{y}^{\star} into next-token probabilities. Peak memory is bounded by a single forward through the attractor module and standard KV-caching applies in the backbone. In principle, the solver tolerance ε \varepsilon and maximum iteration budget T max T_{\max} are inference-time hyperparameters: they can be adjusted without retraining the model, turning test-time computation into a budget for approaching the learned attractor. Interestingly, however, we find that trained Attractor Models often require very little test-time refinement, as we describe in the next section.
### 3.3 Equilibrium Internalization and Stability in Attractor Models
Although Attractor Models define predictions through the equilibrium y ~ ⋆ \tilde{y}^{\star} , we observe a surprising phenomenon: after training, the backbone proposal y ~ 0 \tilde{y}_{0} often lies close to the equilibrium (c.f. Figures 6 and 7 ). We refer to this phenomenon as equilibrium internalization . Intuitively, the attractor module appears to act as a moving teacher for the backbone, resulting in a form of automatic curriculum. Early in training, the proposal y ~ 0 \tilde{y}_{0} may be far from a good prediction, and the solver must perform nontrivial refinement to reach y ~ ⋆ \tilde{y}^{\star} . Since y ~ 0 \tilde{y}_{0} and y ~ ⋆ \tilde{y}^{\star} live in the same tied output-embedding space, gradients through the equilibrium also train the backbone proposal to move toward the state that the solver would have found. Thus, when the backbone is sufficiently expressive, much of the prediction work can be internalized into y ~ 0 \tilde{y}_{0} , leaving the attractor to perform a stable refinement.
Stability. Equilibrium training also biases the recurrent map toward convergent dynamics. The implicit gradient contains the inverse factor ( I − J y ~ ⊤ ) − 1 (I-J_{\tilde{y}}^{\top})^{-1} , which becomes ill-conditioned near non-contractive regimes. This creates a barrier against unstable fixed-point dynamics, unlike fixed-loop training, which can learn trajectories that are accurate only at a prescribed step count and fail under extra inference-time loops. We discuss this contrast in detail in Appendix B.3 . We refer to Appendix B for theoretical analysis of Attractor Models and comparison with finite-loop models.
## 4 Experiments
We evaluate Attractor Models in two regimes. We first study language modeling across model sizes, comparing against parameter-matched Transformers and looped-LM baselines. We evaluate scaling behavior, downstream accuracy, and training efficiency. We also present results on hard reasoning tasks, where we test whether the same fixed-point refinement mechanism improves small models on problems that require iterative computation.
### 4.1 Attractor Models Improve Large-Scale Language Modeling
Setup. We follow the nanochat ( Karpathy, 2025 ) pretraining recipe used by Parcae ( Prairie et al., 2026 ) for its main Transformer comparison, training on FineWeb-Edu ( Penedo et al., 2024 ) . To ensure a fair comparison, all models are matched in parameter count and trained with the same data budget, optimizer, and learning-rate schedule as the Parcae baselines; the only change is the recurrent block. We compare at three scales: 140M, 370M, and 770M parameters. Parcae is a middle-looped language model with prelude, coda, and recurrent blocks. The stability in this model comes from a linear injection that bounds the spectral radius of the recurrence below one. In contrast, our architecture uses standard Transformer blocks followed by a fixed-point iteration block, and lets the solver itself control the effective depth. Detailed hyperparameter settings are in Appendix C .
Parameter Scaling. We demonstrate how large-scale pretraining scales with our model. We evaluate our method on 140M, 370M, and 770M parameters against a parameter-matched Transformer and Parcae, a looped-language model ( Prairie et al., 2026 ) . Our model improves monotonically with scale. Across all three sizes, our method achieves the best validation PPL, Lambada PPL, and CORE accuracy. These results show that our fixed-point refinement scales cleanly with model size, with especially large gains on Lambada, where iterative refinement substantially improves long-context prediction.
Training efficiency: Lower FLOPs. The one-step IFT backward pass keeps training memory constant in the number of solver iterations. The memory of standard looped language models like ( Prairie et al., 2026 ) scales linearly with the number of loops. The total training FLOPs (Figure 4 ) follow the same trend: although every step of our recurrent block costs roughly the same as Parcae’s, our solver typically converges below ε \varepsilon in well under T max T_{\max} steps, so the realized depth during training is lower despite identical T max T_{\max} , yielding 25–31% lower training FLOPs across scales.
Training efficiency: O(1) memory. An additional advantage of our method is that the training memory required stays constant with the number of iterations (Figure 5 ). This is because our implicit backward pass does not need to store the intermediate activations from every recurrent step. In contrast, standard looped language models must backpropagate through each unrolled iteration, causing memory usage to grow linearly with the number of loops.
### 4.2 Attractor Models Lead to Significant Gains in Hard Reasoning Tasks
Beyond language modeling, we train and evaluate Attractor Models on Sudoku-Extreme and Maze-Hard from ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ) : two challenging reasoning benchmarks, where non-recurrent Transformers and most frontier LLMs fail.
Setup. We train small models with approximately 1000 training examples for each task and require predicting the full output grid in a single direct forward pass (no autoregressive decoding). We follow the TRM training protocol ( Jolicoeur-Martineau, 2025 ) , using deep-supervision steps.
Method. TRM carries two latents across deep-supervision steps, a current answer y y and a reasoning state z z , and applies a tiny two-layer network T ( n + 1 ) T(n{+}1) times per step to update them. We keep this protocol except for the inner update. Specifically, instead of unrolling T ( n + 1 ) T(n{+}1) applications, we solve directly for the fixed point of the ( y , z ) (y,z) update with our solver. The initialization is handled by deep supervision itself: the previous step’s ( y , z ) (y,z) initializes the solver at the next step, with a learned embedding at step zero, so we do not use a separate backbone.
We present our results in Table 2 . The fixed-depth Transformer fails on both tasks. HRM (27M) achieves 55.0% and 74.5%, respectively on Sudoku-Extreme and Maze-Hard. TRM is the strongest tiny baseline at 7M, achieving 74.7% and 85.3%, but (counter to the goal of scaling) collapses to 0% on both tasks when scaled to 27M parameters. Our model scales naturally with parameter count. We attribute the difference to the explicit fixed-point objective, which appears to provide regularization that bare iterative refinement lacks at higher capacity.
For the backward pass, we use the phantom-gradient scheme ( Geng et al., 2022 ) rather than the one-step approximation used in the language-modeling experiments. This choice is important in the small-data reasoning regime: with only ∼ \sim 1,000 training examples and much smaller networks, the solver dynamics are more sensitive, and the one-step surrogate can provide too crude a training signal. This is consistent with TRM, which reports that replacing its backward pass with a one-step approximation reduces Sudoku-Extreme accuracy from 87.4% to 56.5% ( Jolicoeur-Martineau, 2025 ) .
This setup is still an instance of our Attractor Model framework, where an output-space representation is iteratively refined to an equilibrium, decoded, and differentiated through implicitly. The only difference is the initialization mechanism. Rather than a Transformer backbone producing y ~ 0 \tilde{y}_{0} from the input, deep supervision supplies the initialization. Specifically, the previous supervision step’s ( y , z ) (y,z) initializes the solver at the next step, with a learned embedding at step zero.
### 4.3 Equilibrium Internalization and Test-Time Behavior