A Mechanistic Analysis of Looped Reasoning Language Models

arxiv:2604.11791

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-04-13

Cites (6) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• A Mechanistic Analysis of Looped Reasoning Language Models
• Abstract
• Keywords:
• 1 Introduction
• 2 Preliminaries
• 2.1 Looped Transformers
• 2.2 Stages of Inference
• 3 Related Work
• Looped and Recurrent Transformers
• Stages of Inference and Attention Dynamics
• Test-time Computation
• 4 Looped Transformers Tend Towards the Same Attention Patterns
• Proposition 4.1 (Cyclic recurrent blocks reach cyclic fixed points) .
• Proposition 4.2 (Recurrent attention patterns change slowly under state convergence) .
• 4.1 Empirical Validation
• 4.2 Impact of Architecture Choices
• 5 Stages of Inference in Looped Models Mirror Feedforward Computation
• 5.1 Self-Organization Into Stages of Inference
• 5.2 Stability to Unseen Numbers of Recurrences
• 6 Conclusion
• Implications of Findings
• Limitations and Future Work
• Impact Statement
• Acknowledgments
• Appendix Contents
• Appendix A Proofs of Propositions
• Proof of Proposition 4.1
• Proof sketch.
• Proof.
• Proof of Proposition 4.2
• Proof.
• Appendix B Additional Experimental Details
• Appendix C Non-Fixed-Point Limiting Behavior
• C.1 How Frequent is Non-Fixed-Point Behavior?
• Long Persona
• Long Persona (Padded)
• Short Math
• C.2 Do Intermediate Layers Exhibit Non-Fixed-Point Behavior?
• C.3 How Does Non-Fixed-Point Behavior Impact Stages of Inference?
• Appendix D Additional Fixed Point Results
• D.1 Cyclic Similarity
• D.2 Fixed Point and Successive Differences
• D.3 Latent Space Trajectories
• D.4 Architecture Choices
• Appendix E Additional Stages of Inference Results
• E.1 Input Independent Metrics
• E.2 Input Dependent Metrics
• E.3 Cyclic Stages of Inference
• E.4 Non-Reasoning Stages of Inference
• E.5 Stability To Unseen Test-Time Recurrences
• E.6 How Architecture Choices Affect the Formation of Stages of Inference
• Appendix F Looped Floorplan
• Instructions for reporting errors