The Topological Trouble With Transformers

arxiv:2604.17121

Original source · Paper JSON

LLMs: architectures, context, training · 2026-04-18

Cites (9) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• The Topological Trouble With Transformers
• Abstract
• 1 Introduction
• 2 State tracking
• 3 Recurrent architectures
• 4 Architectural limitations and workarounds
• 5 Promising directions
• 5.1 Enhanced State-Space Models
• 5.2 Approximating state tracking in feedforward transformers
• 5.3 Coarse recurrence
• 5.4 Leveraging representational alignment
• 5.5 Efficient training of recurrence
• 6 Conclusions
• Acknowledgments and Disclosure of Funding