Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach

arxiv:2502.05171

Original source · Paper JSON

Reasoning and the "physics" of language models · 2025-02-07

Cites (13) · Cited by (21)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Abstract
• 1 Scaling by Thinking in Continuous Space
• 2 Why Train Models with Recurrent Depth?
• ———————— Table of Contents ————————
• 3 A scalable recurrent architecture
• 3.1 Macroscopic Design
• Motivation for this Design.
• Remark 3.1 (Is this a Diffusion Model?) .
• 3.2 Microscopic Design
• 3.3 Training Objective
• Training Recurrent Models through Unrolling.
• Truncated Backpropagation.
• 4 Training a large-scale recurrent-depth Language Model
• 4.1 Training Setup
• Pretraining Data.
• Tokenization and Packing Details.
• Architecture and Initialization.
• Locked-Step Sampling.
• Optimizer and Learning Rate Schedule.
• 4.2 Compute Setup and Hardware
• Device Speed and Parallelization Strategy.
• Training Timeline.
• 4.3 Importance of Norms and Initializations at Scale
• 5 Benchmark Results
• 5.1 Standard Benchmarks
• 5.2 Math and Coding Benchmarks
• 5.3 Where does recurrence help most?
• Recurrence and Context
• 5.4 Improvements through Weight Averaging
• 6 Recurrent Depth simplifies LLMs
• 6.1 Zero-Shot Adaptive Compute at Test-Time
• Remark 6.1 (What about missing KV-cache entries?) .
• 6.2 Zero-Shot KV-cache Sharing
• 6.3 Zero-Shot Continuous Chain-of-Thought
• 6.4 Zero-Shot Self-Speculative Decoding
• 7 What Mechanisms Emerge at Scale in Recurrent-Depth Models
• Path Independence.
• 8 Related Work Overview
• 9 Future Work
• 10 Conclusions
• Acknowledgements
• Appendix A Additional Information
• Potential Implications of This Work
• A.1 Classical Reasoning Problems
• A.2 Implementation Details
• Device Speed Details
• Parallelization Strategy
• RCCL Interconnect Handling
• Pretraining Metrics.
• Appendix B Latent Space Visualizations
• Appendix C Pretraining Data
• Instructions for reporting errors