Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning

arxiv:2604.21999

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-04-23

Cites (7) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning
• Abstract
• 1 Introduction
• 2 Architecture
• 2.1 Universal Transformer with Memory Tokens
• 2.2 Adaptive Computation Time
• 2.3 Deep-Start Initialization
• 3 The Router Initialization Trap
• 3.1 Diagnosis
• 3.2 Deep Start Resolves the Trap
• 3.3 DerfNorm Is Not the Cause
• 4 Memory Tokens for Recursive Reasoning
• 4.1 Task and Setup
• 4.2 Memory-Token Curve
• 4.3 Memory-Depth Tradeoff
• 4.4 Fixed-Depth vs ACT Processing
• 4.5 How Memory Tokens Function: Attention Analysis
• 4.6 Small-Dataset Generalization
• 5 Making ACT Efficient
• 5.1 Lambda Warmup
• 5.2 Inference Beyond Trained Depth
• 6 Related Work and Architectural Convergence
• 6.1 Universal Transformers and Adaptive Computation Time
• 6.2 Memory Tokens and Registers
• 6.3 Recursive Reasoning on Combinatorial Benchmarks
• 6.4 Recursive Transformers at Language-Modeling Scale
• 6.5 Adjacent: External Memory and Adaptive Long Context
• 6.6 Mapping the Design Space
• 6.7 What the Convergence Suggests
• 7 Future Work
• 8 Limitations
• 9 Conclusion
• Appendix A Puzzle Solving Visualization
• Instructions for reporting errors