Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers

arxiv:2606.18206

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-06-16

Cites (10) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
• Abstract
• 1 Introduction
• 2 Background and Related Works
• Looped models.
• Adaptive computation.
• Deep equilibrium and fixed-point models.
• Score-based methods.
• Signal propagation in looped models.
• 3 Method
• 3.1 Improving signal propagation with pre-norm
• 3.2 Recovering boundedness via residual scaling
• Layer-wise residual scaling.
• Iteration-wise input mixing.
• Theorem 2 (Small α 2 \alpha_{2} implies convergence) .
• 3.3 Oscillation around the fixed-point
• 3.4 Optimization of fixed-point models
• Proposition 1 (Exponential decay of truncated-BPTT error) .
• 3.5 The fixed-point reasoning model
• 4 Experiments
• 4.1 Dataset description
• Sudoku-Extreme.
• Maze-Hard.
• ARC-1 and ARC-2.
• State tracking.
• 4.2 Puzzle tasks
• 4.3 Adaptivity
• State tracking.
• Sudoku-Extreme.
• Compute–accuracy trade-off in fixed-point halting.
• 4.4 Depth-induced signal propagation issues
• 4.4.1 Boundedness of activation norms and trainability
• 4.4.2 Pre-norm with residual scaling enables higher depth utilization
• State-tracking.
• Sudoku-Extreme.
• 4.4.3 The training dynamics of residual scales
• 5 Discussion
• Looped fixed-point models are adaptive.
• Enabling ACT at inference time.
• The role of hierarchy in HRM and TRM.
• Scaling behavior of FPRM.
• Limitations.
• 6 Conclusion
• Acknowledgments and Disclosure of Funding
• Appendix A Proofs
• A.1 Fixed-point iterations bound
• Proof.
• A.2 Contractive mapping
• Proof.
• A.3 Mitigating oscillation through damping
• Proof.
• A.4 Error of truncated-BPTT
• Proof.
• Appendix B A Toy Failure Mode for Recurrent Post-norm
• Appendix C Further Details About the Architecture
• Fixed-point solver.
• Deep supervision.
• Depth-wise convolutions.
• Appendix D Description of Figure 1
• Appendix E Fixed-point Residuals and Halting
• Appendix F How to effectively spend loops in TRM?
• Appendix G Additional Experimental Details
• Weight initialization.
• Grokking.
• Hyperparameters, device specification
• Instructions for reporting errors