Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning — part 3/7
Cleaned-text code points [30969, 46646). Bibliography omitted here; source.md includes it.
So far, our scaling experiments have applied the same test-time budget to every task instance, regardless of difficulty. However, applying a massive, static compute budget universally is highly inefficient. As our landscape analysis suggests, instance difficulty is highly heterogeneous: simple problems quickly fall into favorable attractors, whereas complex ones require extensive iterative refinement. To systematically optimize the compute-accuracy Pareto frontier, we resort to elastic budget inference guided by a learned halting policy ( Jolicoeur-Martineau, 2025 ) . By equipping EqR with a learned halting head, the model dynamically tracks its internal state and terminates computation early upon converging to an attractor, so additional compute is allocated mainly to instances that remain unresolved.
We use a fixed-size inference queue for batching: halted samples are replaced immediately to keep utilization high while preserving per-sample stopping. This allows ACT to convert sample-wise dynamic halting into an actual reduction in the average number of function evaluations (Avg. NFE) at inference time.
Tab. 5 shows that ACT substantially reduces Avg. NFE with only minor accuracy changes: at D = 1024 D{=}1024 , Avg. NFE drops from 1024.0 1024.0 to 58.7 58.7 ( 17.4 × 17.4\times fewer evaluations) while accuracy changes from 96.1 96.1 to 95.3 95.3 ; under breadth scaling ( 64,128 ) (64,128) , Avg. NFE drops from 8192.0 8192.0 to 1400.6 1400.6 ( 5.8 × 5.8\times fewer evaluations) while accuracy changes from 97.9 97.9 to 97.4 97.4 . This suggests most instances terminate early, while a small fraction require long runs. Fig. 7 shows the same effect at the accuracy–compute frontier: EqR+ACT reaches the matched accuracy target with 11.34 × 11.34\times fewer NFEs than the baseline.
Together with earlier scaling results, this spans both ends of compute: large-budget scaling and budget-elastic efficiency. The halting objective’s training-side effect was analyzed in Tab. 8 ( 8(e) ); here we focus on ACT’s inference-side compute–accuracy trade-off.
## 7 Related Work
#### Iterative weight-tied models.
Iterative models apply an update operator repeatedly to refine a latent state, with representative works including weight-tied Transformers such as the Universal Transformer and related variants ( Dehghani et al., 2019 ; Graves, 2017 ; Chowdhury and Caragea, 2025 ; Heo et al., 2025 ) . Deep Equilibrium Models ( Bai et al., 2019 ) take this idea to the implicit limit by defining representations as fixed points, followed by extensive work on convergence diagnostics, stability, and more efficient training methods ( Bai et al., 2021 ; Geng et al., 2021a ; Geng et al., 2021b ; Gu et al., 2020 ; Anil et al., 2022 ; Fung et al., 2022 ) and practical tooling such as TorchDEQ ( Geng and Kolter, 2023 ) .
Path-independent equilibrium models make this connection more explicit: when iterative inference converges to the same fixed point regardless of the trajectory or initialization, additional test-time steps can reliably refine toward a well-defined representation ( Anil et al., 2022 ) . Our setting relaxes the requirement of a globally unique fixed point: we instead ask whether finite rollouts and restarts concentrate around solution-aligned attractors, making path independence both a diagnostic and a property encouraged by our intervention.
Recent weight-tied models further show that iterative latent computation is becoming an active scaling direction across language and visual reasoning ( Geiping et al., 2025 ; Zhu et al., 2025 ; Prairie et al., 2026 ; Song et al., 2026 ; Bae et al., 2025 ; Shu et al., 2026 ) . In parallel, the HRM series of works shows that such iterative models have strong performance over complex and structured reasoning tasks ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Gao et al., 2025 ) . These finite iterative computations still induce a latent dynamical system; we study whether training can shape their trajectories toward solution-aligned attractors, enabling scalable gains from additional test-time compute.
#### Compute allocation in weight-tied models.
For weight-tied models, test-time compute allocation asks how many iterations to spend, where to spend them, and whether the same budget should be used uniformly or adapted across inputs. One line of work studies iteration or recurrent-depth scaling, showing that applying a weight-tied module more times at test time can improve performance beyond the training regime ( Schwarzschild et al., 2021 ; Geiping et al., 2025 ; Prairie et al., 2026 ; Shu et al., 2026 ) . A second line focuses on adaptive allocation, using mechanisms such as token-wise depth ( Song et al., 2026 ) , learned recurrence mixtures ( Bae et al., 2025 ) , adaptive halting ( Graves, 2017 ) , or elastic-depth budget conditioning ( Jeddi et al., 2026 ) to spend different amounts of computation across tokens, examples, or compute budgets. Recent compute-allocation diagnostics ( Moosa et al., 2026 ) and selective latent iterations on hard tokens ( Fu et al., 2025 ) further ask whether adaptive policies allocate compute to genuinely hard tokens rather than merely reducing average depth. These model-specific mechanisms connect to a broader inference-time scaling lesson: additional computation is most useful when allocated to informative candidates, states, or search branches rather than spent uniformly ( Snell et al., 2025 ; Yang et al., 2025 ; Ghosal et al., 2025 ) . In our work, we use depth scaling and breadth scaling to verify that scaling far beyond the training regime can substantially improve performance. We then close the efficiency side by learning a halting module following Graves (2017) , which cuts inference cost by allocating fewer iterations to easier inputs while preserving most of the scaling gains.
## 8 Conclusion
In this work, we present an attractor-based perspective on test-time scaling in iterative reasoning models. Trajectory diagnostics show that depth and breadth help when learned attractors are aligned with the task metric and reachable from diverse initial states. Guided by this view, randomized initialization and path noise reshape the latent landscape, improving the coverage and stability of correct attractors on Sudoku and Maze and making additional computation more reliable. The broader implication is that iterative latent reasoning models should be evaluated not only by final accuracy, but also by whether their internal dynamics make correct solutions stable, reachable, and selectable. We hope these diagnostics and interventions provide a step toward a more mechanistic understanding of scalable iterative reasoning.
Appendix
A Additional Results, Analyses, and Findings . A
A.1 Attractor Formulation and Residual Diagnostics . A.1
A.2 Training-Dynamics Ablations and Diagnostics . A.2
A.3 Additional Experiments on Randomized State Initialization . A.3
A.4 Additional Experiments on Path Stochasticity . A.4
A.5 Generalization Beyond the Main Setting . A.5
A.6 Seed Stability Diagnostics . A.6
B Qualitative Study . B
C Dataset Details and Task Definitions . C
C.1 Dataset and Benchmark Specifications . C.1
C.2 Datasets and Task Definitions Shape Attractor Landscapes . C.2
D Method and Experimental Details . D
D.1 Architecture and Hyperparameters . D.1
D.2 Learning-Rate Control for Feedforward Baselines . D.2
D.3 Evaluation Metrics . D.3
D.4 Compute Accounting for Iterative Inference . D.4
E Extended Related Work and Discussion . E
## Appendix A Additional Results, Analyses, and Findings
This section extends the findings in the main text through additional analyses and experiments, including a residual-diagnostic formulation, training-dynamics ablations from feedforward models to weight-tied iterative models, ablations of training interventions on initialization and path stochasticity, generalization beyond the main setting, and seed stability.
### A.1 Attractor Formulation and Residual Diagnostics
This subsection isolates the attractor formulation and residual diagnostics used to interpret the training ablations below.
A useful idealization of attractor learning follows the DEQ view of an input-conditioned equilibrium layer ( Bai et al., 2019 ) : min θ , 𝐳 ℓ θ ( 𝐳 , 𝐱 , 𝐲 ) s.t. R θ ( 𝐳 , 𝐱 ) = 𝐳 − f θ ( 𝐳 , 𝐱 ) = 0 . \min_{\theta,\mathbf{z}}\;\ell_{\theta}(\mathbf{z};\mathbf{x},\mathbf{y})\qquad\text{s.t.}\qquad R_{\theta}(\mathbf{z};\mathbf{x})=\mathbf{z}-f_{\theta}(\mathbf{z};\mathbf{x})=0. (3) The constraint says that the supervised latent state should be a fixed point of the current update operator. This differs from the Universal Transformer formulation ( Dehghani et al., 2019 ) , where input token embeddings initialize the recurrent state and the shared block is iterated over that state. Here the update is written as an input-conditioned solver f θ ( 𝐳 , 𝐱 ) f_{\theta}(\mathbf{z};\mathbf{x}) : the problem data 𝐱 \mathbf{x} remains available as an external condition at every solver step, as in DEQs and recent recurrent-depth models ( Geiping et al., 2025 ) . In our setting, however, the relevant object is not a globally unique fixed point, but a reachable attractor or low-residual state reached from an initialization 𝐳 0 \mathbf{z}_{0} by iterative computation. We can therefore write the practical surrogate as a bilevel optimization problem: min θ \displaystyle\min_{\theta} 𝔼 ( 𝐱 , 𝐲 ) , 𝐳 0 [ ℓ θ ( 𝐳 θ ⋆ ( 𝐱 , 𝐳 0 ) , 𝐱 , 𝐲 ) ] \displaystyle\mathbb{E}_{(\mathbf{x},\mathbf{y}),\mathbf{z}_{0}}\bigl[\ell_{\theta}(\mathbf{z}^{\star}_{\theta}(\mathbf{x},\mathbf{z}_{0});\mathbf{x},\mathbf{y})\bigr] (4) s . t . \displaystyle\mathrm{s.t.} 𝐳 ⋆ θ ( 𝐱 , 𝐳 0 ) := Solve ( D ) θ ( 𝐳 0 ; 𝐱 ) , ‖ 𝐳 ⋆ θ − f θ ( 𝐳 ⋆ θ ; 𝐱 ) ‖ 2 ≤ ε res . \displaystyle\mathbf{z}^{\star}_{\theta}(\mathbf{x},\mathbf{z}_{0}):=\operatorname{Solve}^{(D)}_{\theta}(\mathbf{z}_{0};\mathbf{x}),\quad\left\|\mathbf{z}^{\star}_{\theta}-f_{\theta}(\mathbf{z}^{\star}_{\theta};\mathbf{x})\right\|_{2}\leq\varepsilon_{\mathrm{res}}. where ℓ θ ( 𝐳 , 𝐱 , 𝐲 ) \ell_{\theta}(\mathbf{z};\mathbf{x},\mathbf{y}) denotes the supervised loss evaluated from latent state 𝐳 \mathbf{z} , Solve θ ( D ) \operatorname{Solve}^{(D)}_{\theta} denotes a D D -step lower-level rollout induced by repeated applications of f θ f_{\theta} , and ε res \varepsilon_{\mathrm{res}} is the residual tolerance for the reached state.
#### Fixed-point residual as a local convergence diagnostic.
The formulation clarifies why residual is a meaningful diagnostic under local stability. Suppose the reached basin contains a fixed point 𝐳 ⋆ = f θ ( 𝐳 ⋆ , 𝐱 ) \mathbf{z}^{\star}=f_{\theta}(\mathbf{z}^{\star};\mathbf{x}) , and f θ f_{\theta} is L L -Lipschitz in that basin for some L < 1 L<1 . Then ‖ 𝐳 − 𝐳 ⋆ ‖ \displaystyle\|\mathbf{z}-\mathbf{z}^{\star}\| ≤ ‖ 𝐳 − f θ ( 𝐳 , 𝐱 ) ‖ + ‖ f θ ( 𝐳 , 𝐱 ) − f θ ( 𝐳 ⋆ , 𝐱 ) ‖ \displaystyle\leq\|\mathbf{z}-f_{\theta}(\mathbf{z};\mathbf{x})\|+\|f_{\theta}(\mathbf{z};\mathbf{x})-f_{\theta}(\mathbf{z}^{\star};\mathbf{x})\| (5) ≤ ‖ R θ ( 𝐳 , 𝐱 ) ‖ + L ‖ 𝐳 − 𝐳 ⋆ ‖ , \displaystyle\leq\|R_{\theta}(\mathbf{z};\mathbf{x})\|+L\|\mathbf{z}-\mathbf{z}^{\star}\|, so ‖ 𝐳 − 𝐳 ⋆ ‖ ≤ ‖ R θ ( 𝐳 , 𝐱 ) ‖ / ( 1 − L ) \|\mathbf{z}-\mathbf{z}^{\star}\|\leq\|R_{\theta}(\mathbf{z};\mathbf{x})\|/(1-L) . Thus, inside a stable basin, small residual implies closeness to the local attractor. To connect residual to correctness, one also needs an output margin. Let s θ , i , a ( 𝐳 , 𝐱 ) s_{\theta,i,a}(\mathbf{z};\mathbf{x}) be the logit for assigning label a a at output location i i , and suppose 𝐳 ⋆ \mathbf{z}^{\star} decodes to the correct output 𝐲 \mathbf{y} . Define the minimum margin at the attractor by γ ( 𝐳 ⋆ ) = min i [ s θ , i , y i ( 𝐳 ⋆ , 𝐱 ) − max a ≠ y i s θ , i , a ( 𝐳 ⋆ , 𝐱 ) ] . \gamma(\mathbf{z}^{\star})=\min_{i}\left[s_{\theta,i,y_{i}}(\mathbf{z}^{\star};\mathbf{x})-\max_{a\neq y_{i}}s_{\theta,i,a}(\mathbf{z}^{\star};\mathbf{x})\right]. (6) If γ ( 𝐳 ⋆ ) > 0 \gamma(\mathbf{z}^{\star})>0 and each true-versus-competing logit gap is G gap G_{\mathrm{gap}} -Lipschitz in the same basin, then states within distance γ ( 𝐳 ⋆ ) / G gap \gamma(\mathbf{z}^{\star})/G_{\mathrm{gap}} decode correctly. Combining this margin condition with Eq. 5 , a sufficient residual condition is ‖ R θ ( 𝐳 , 𝐱 ) ‖ < ( 1 − L ) γ ( 𝐳 ⋆ ) G gap ⟹ 𝐲 ^ θ ( 𝐳 , 𝐱 ) = 𝐲 . \|R_{\theta}(\mathbf{z};\mathbf{x})\|<(1-L)\frac{\gamma(\mathbf{z}^{\star})}{G_{\mathrm{gap}}}\quad\Longrightarrow\quad\hat{\mathbf{y}}_{\theta}(\mathbf{z};\mathbf{x})=\mathbf{y}. (7) Thus residual is a correctness proxy only under local stability, a correct attractor, and positive output margin; low residual near a spurious or low-margin attractor still certifies convergence, not correctness.
#### Implicit gradient conditioning.
The same formulation also clarifies why exact implicit gradient computation through long attractor solvers can be unstable. Let J 𝐳 = ∂ 𝐳 f θ ( 𝐳 ⋆ , 𝐱 ) J_{\mathbf{z}}=\partial_{\mathbf{z}}f_{\theta}(\mathbf{z}^{\star};\mathbf{x}) and J θ = ∂ θ f θ ( 𝐳 ⋆ , 𝐱 ) J_{\theta}=\partial_{\theta}f_{\theta}(\mathbf{z}^{\star};\mathbf{x}) . At an exact locally isolated fixed point, differentiating R θ ( 𝐳 ⋆ , 𝐱 ) = 0 R_{\theta}(\mathbf{z}^{\star};\mathbf{x})=0 for fixed 𝐱 \mathbf{x} gives ( I − J 𝐳 ) d 𝐳 ⋆ = J θ d θ , d 𝐳 ⋆ d θ = ( I − J 𝐳 ) − 1 J θ , (I-J_{\mathbf{z}})\,d\mathbf{z}^{\star}=J_{\theta}\,d\theta,\qquad\frac{d\mathbf{z}^{\star}}{d\theta}=(I-J_{\mathbf{z}})^{-1}J_{\theta}, (8) with Jacobians evaluated at ( 𝐳 ⋆ , θ , 𝐱 ) (\mathbf{z}^{\star},\theta,\mathbf{x}) . Thus, solver sensitivity is controlled by the resolvent ( I − J 𝐳 ) − 1 (I-J_{\mathbf{z}})^{-1} : when this operator is poorly conditioned, small parameter changes can cause large attractor shifts, and approximation errors in the lower-level solve can be amplified in the implicit gradient. This conditioning issue is a standard concern in implicit-layer training, where prior work studies Jacobian regularization, monotone or well-posed operators, and approximate or Jacobian-free backward passes ( Bai et al., 2021 ; Geng et al., 2021a ; Geng et al., 2021b ; Gu et al., 2020 ; Fung et al., 2022 ) . The role of truncation and SOT is therefore not only to reduce memory and compute, but also to keep optimization local while the latent trajectory tracks a changing attractor landscape.
### A.2 From Feedforward Models to Iterative Models: Training-Dynamics Ablations and Diagnostics
This subsection expands the construction path from feedforward models to weight-tied iterative models in Sec. 6.1 , Tab. 2 , using the design axes introduced in Sec. 3 , on Sudoku tasks. The full ablations cover weight-tied structure, hierarchical iterations, gradient truncation, supervision and optimization schedules, adaptive computation variants, and FLOPs and memory accounting. For weight-tied models, we follow the configurations of TRM listed in Tab. 17 . For a fair comparison, the Sudoku feedforward baseline uses 42 distinct blocks, matching equivalent-layer budget N eq = 42 N_{\mathrm{eq}}=42 of the corresponding weight-tied iterative model (Appendix D.4 ). Throughout this subsection, an iteration means one outer-loop step or segment, which may itself contain repeated applications. We count these through equivalent-layer evaluations. A trajectory with D D iterations contains D D outer-loop segments, and an iteration-depth multiplier scales the number of segments rather than the individual layer applications inside each segment.
Iterative models are fundamentally different from feedforward models.