Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

arxiv:2606.31779

Original source · Paper JSON

Reasoning and the "physics" of language models · 2026-06-30

Cites (9) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
• Abstract
• 1 Introduction
• 2 Preliminaries
• Explicit CoT.
• Latent reasoning
• Looped padded Transformers.
• Prior latent reasoning methods.
• 3 Method
• 3.1 Padded Latent Prefix
• 3.2 Looped Latent Computation
• 3.3 Direct Step-Aligned Supervision
• 3.3.1 Objective
• Step CoT supervision loss.
• Answer supervision loss.
• Key properties of the objective.
• Inference.
• No latent decoding for answer generation.
• 3.3.2 Roles of ℒ step {\mathcal{L}_{\mathrm{step}}} and ℒ ans {\mathcal{L}_{\mathrm{ans}}}
• Parallel chain likelihood.
• Complementary roles.
• 3.4 LOTUS-aux: An Auxiliary Decoder Variant
• Latent supervision via the auxiliary decoder.
• Per-iteration supervision and the full objective.
• Inference.
• 4 Experiments
• 4.1 Setup
• Datasets and backbones.
• Methods.
• 4.2 Main Results
• LOTUS matches explicit CoT at scale, where prior latent methods fall behind.
• The two routings match at 3B, but only direct LOTUS stays robust at smaller scales.
• LOTUS reasons 2.5 × 2.5\times faster than explicit CoT.
• Speedup grows to 6.9 × 6.9\times on natural-language CoT traces.
• 4.3 Ablation Studies
• 4.3.1 Latent Supervision Design
• Both the looped backbone and latent supervision are beneficial.
• The best supervision schedule differs between LOTUS and LOTUS-aux.
• 4.3.2 Looped Architecture Design
• Latent token budget per block c c .
• Inference-time c c .
• Looped iterations R {R} .
• Inference-time R {R} .
• 5 Latent Representation Analysis
• 5.1 Likelihood of the Gold Chain under Different Readout Options
• Metrics.
• LOTUS’s post-loop latent 𝒉 ( R ) {{{\bm{h}}}^{({R})}} contains the gold CoT via a direct readout.
• LOTUS-aux has CoT signal in both 𝒉 ( R ) {{{\bm{h}}}^{({R})}} and 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} .
• 5.2 Mass on Unseen Valid Chains
• Setup.
• Findings.
• 5.3 Disentangling the Two Losses in LOTUS
• Gold-chain NLL.
• ℒ step {\mathcal{L}_{\mathrm{step}}} and ℒ ans {\mathcal{L}_{\mathrm{ans}}} play complementary roles.
• 6 Conclusion
• Limitations.
• Acknowledgments
• Appendix A Related Work
• Looped and recurrent-depth transformers.
• Diffusion language models.
• Pause and padding symbols.
• Compressing and replacing discrete CoT.
• Speculative decoding.
• Appendix B Notation
• Appendix C Closely Related Latent Reasoning Methods
• Coconut [ Hao et al., 2025 ]
• CODI [ Shen et al., 2025 ] and SIM-CoT [ Wei et al., 2025 ]
• Parallel Continuous CoT (PCCoT) [ Wu et al., 2025 ] and KaVa [ Kuzina et al., 2026 ]
• Comparison with latent methods at other compute budgets.
• Natural-language CoT stress-test.
• Appendix D Experimental Details
• Latent prefix budget.
• Training curriculum.
• Optimization details.
• Natural-language training setup.
• LOTUS-aux auxiliary decoder.
• Evaluation.
• Batched KV-cache sharing.
• Appendix E PCL Versus Autoregressive NLL Lower Bounds
• Appendix F Details for Latent Representation Analysis ( Section 5 )
• F.1 Greedy Readout Examples
• Example 1 — Janet’s ducks.
• Example 2 — James’s sprints.
• Example 3 — Carla’s download (failure).
• Example 4 — Melanie’s vacuums (failure).
• F.2 Details for Multi-Path Analysis ( Section 5.2 )
• Datasets.
• Path bank.
• Candidate sets and filtering.
• Metrics.
• F.3 Multi-Path Readout Example
• Instructions for reporting errors