Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — part 4/6

Cleaned-text code points [46850, 61923). Bibliography omitted here; source.md includes it.

We introduce LOTUS, showing that latent reasoning can approach the performance of explicit CoT by supervising a looped padded Transformer in parallel against the gold CoT tokens under the simple cross-entropy objective. On Llama-3.2-3B-Instruct , LOTUS bridges the in-domain gap to explicit CoT on GSM8K, surpasses CoT on the out-of-domain average, and cuts thought-phase latency by 2.5 × 2.5\times . Ablations show the looped backbone, parallel gold CoT supervision, and sufficient block width and loop depth are each necessary. The latent representation analysis further shows the latents are transparent: the gold CoT is recoverable from them by a direct readout, they place graded probability on unseen but valid reasoning chains rather than a single memorized trace, and the step and answer losses contribute complementary structure.

##### Limitations.

We follow prior latent reasoning work ( Hao et al., 2025 ; Shen et al., 2025 ; Wei et al., 2025 ) and evaluate on math benchmarks. Whether the recipe transfers to other domains remains an open direction for future work. The block budget K {K} , per-block width c {c} , and loop depth R {R} are fixed hyperparameters that must be set to cover the expected step count, so chains longer than K {K} steps fall back to autoregressive completion of the tail. Making K {K} , c {c} , and R {R} adaptive is a natural and promising direction for future work.

## Acknowledgments

Anej Svete is supported by the ETH Zürich AI Center doctoral fellowship.

## Appendix A Related Work

We refer the reader to Chen et al. [2025a] , Yu et al. [2026] , Zhu et al. [2025b] , Li et al. [2025] , Feng et al. [2025] , and Maile and Sacramento [April 27, 2026] for broad surveys of the latent reasoning and looped transformers landscapes. Below, we discuss the work most directly adjacent to ours.

##### Looped and recurrent-depth transformers.

Building on Universal Transformers [ Dehghani et al., 2019 ] , a line of work has established the theoretical foundations of looped architectures: Giannou et al. [2023] show Turing completeness via program simulation, placing looped transformers alongside CoT in expressivity [ Merrill and Sabharwal, 2024 , Li et al., 2024b , Nowak et al., 2024 ] . Saunshi et al. [2025] , Merrill and Sabharwal [2025a] , Merrill and Sabharwal [2025b] , Jerad et al. [2026] study the efficient reasoning abilities of looping, establishing their ability to solve problems with fewer model evaluations than CoT. Saunshi et al. [2025] show that a shallow Transformer looped multiple times nearly matches a non-looped Transformer with the equivalent total depth on arithmetic and reasoning tasks, with looping implicitly simulating hidden reasoning steps. Xu and Sato [2025b] , Xu and Sato [2025a] compare looped Transformers to CoT, and Svete and Sabharwal [2026] draw connections to masked diffusion models. The latter connection also inspires approaches to improve looped transformers with time modulation [ Jeddi et al., 2026 , Chen, 2026 ] and efficient sampling [ Geiping et al., 2025b ] . An active line of work scales looped models to large-scale pretraining: Geiping et al. [2025a] train a 3.5B recurrent-depth Transformer whose loop depth can be increased at test time, Zhu et al. [2025c] pretrain LoopLM with token-level recurrence and learned depth allocation, and Zeng et al. [2026b] , Zeng et al. [2026a] pretrain models that loop by superpositioning symbols in continuous space based on the logits produced after each iteration. LoopLM shows that looped computation can scale in pretraining, but its per-token loop at inference time does not by itself create the parallel padded latent workspace that LOTUS uses. Complementary work addresses different ways of implementing adaptive compute with looped architectures: Depth can be modulated per-symbol via gating [ Ng and Wang, 2024 , Song et al., 2026 , Moosa et al., 2026 ] , mixture-of-depths routing [ Raposo et al., 2024 ] , or RL-trained stopping criteria [ Ning et al., 2025 ] . Depth-recurrent attention mixtures augment recurrence with attention mechanisms that can attend to latent representations of past iterations [ Fu et al., 2025 , Knupp et al., 2026 ] . Parameter efficiency and flexibility are addressed by per-iteration LoRA adapters [ Bae et al., 2025a ] , symbol-level recursion routing [ Bae et al., 2025b ] , inner thinking mechanisms [ Chen et al., 2025b ] , and Mixture-of-Experts compatibility [ Csordás et al., 2024 ] . Further work explores retrofitting recurrence into pretrained models [ McLeish et al., 2025 , Koishekenov et al., 2025 ] , compositional and length generalization [ Fan et al., 2025 , Kohli et al., 2026 ] , reinforcement pretraining [ Tang et al., 2026 ] , and whether pretrained models use their depth efficiently [ Csordás et al., 2025 ] .

##### Diffusion language models.

Diffusion language models are another route to parallel, non-autoregressive reasoning. Discrete-token variants face a parallel-decoding trap , where tokens denoised in parallel are conditionally independent, so recovering quality forces a partial return to sequential decoding [ Liu et al., 2025a ] . Notably, Svete and Sabharwal [2026] show masked diffusion models are equivalent to padded looped Transformers, motivating supervising such a backbone directly, as we do. Continuous diffusion variants such as LaDiR [ Kang et al., 2026 ] instead refine latent thoughts with a VAE and a separate diffusion model over many denoising steps for diversity and adaptive compute, while showing limited efficiency gain when reaching CoT-level performance.

##### Pause and padding symbols.

A parallel strand of work studies the affordances of discrete padding symbols. Goyal et al. [2024] and Pfau et al. [2024] show that they improve performance on natural language and formal tasks; Wang et al. [2024] use discrete symbols in the continuous representation space as planning signals; and Chu et al. [2026] introduce span-level pause symbols for efficient and interpretable latent reasoning. These discrete-symbol approaches motivate our use of continuous latent vectors as a richer computation medium, while our looped architecture lets each iteration refine the same latent rather than extending the sequence.

##### Compressing and replacing discrete CoT.

Latent CoT methods replace or compress explicit reasoning symbols with continuous representations. Cheng and Durme [2024] supervise hidden-state representations of salient reasoning steps, Tan et al. [2025] dynamically compress CoT chains into latent thoughts, Wang et al. [2026] autoregressively encode CoT steps into hidden states decoded back to ground-truth symbols (analogously to our approach), and Amos et al. [2026] interleave supervised latent states with generated text. MUX [ Suleymanzade et al., 2026b ] studies continuous reasoning with multiplexed vocabulary-space targets. It uses KL regression to multiplexed targets. Each target is a position-weighted superposition over aligned reasoning subwords, so several subwords are mixed into one sequential latent. In contrast, LOTUS keeps step and token positions explicit in a parallel padded workspace and applies per-position cross-entropy on gold CoT traces. Latent CoT also enables parallel exploration of reasoning paths [ Gozeten et al., 2026 , Liu et al., 2026 ] , and test-time compute can be scaled in latent space via policy optimization [ Ye et al., 2026 ] , instance-level policy gradient [ Li et al., 2026a ] , sampling conditioned on perturbed prompts [ Xu et al., 2025 ] , or parallel search [ Lian et al., 2025 , You et al., 2026 ] . Latent reasoning can be integrated with generation by mixing latent and text symbols [ Fu and Luo, 2026 , Deng et al., 2026 ] , updating the KV cache with an additional model [ Liu et al., 2025b ] , learning meta-symbols that guide generation [ Shah et al., 2025 ] , and semantically aligning padding symbols to spans of CoT symbols [ He et al., 2025 ] . Further work studies latent reasoning with variational autoencoder features [ Su et al., 2025 , Kang et al., 2026 ] and energy-based guidance [ Chen et al., 2025c ] . GRAM [ Baek et al., 2026 ] guides thinking with recursive latent-variable modeling through shared transitions. It uses stochastic high-level updates and learns a target-conditioned posterior and prior through amortized variational inference. GRAM samples distinct stochastic trajectories, whereas LOTUS represents them within a jointly refined continuous workspace. Interpretability studies probe whether latents actually contribute to reasoning [ Zhang et al., 2025 , Rizvi-Martel and Mosbach, 2026 ] , and empirical comparisons examine the effects of different supervision regimes [ Cui et al., 2026 , Suleymanzade et al., 2026a ] .

##### Speculative decoding.

Speculative decoding uses draft models to propose token continuations [ Leviathan et al., 2023 , Chen et al., 2023 ] . Related approaches include Lookahead [ Fu et al., 2024 ] with Jacobi-style updates, Medusa [ Cai et al., 2024 ] with auxiliary prediction heads, and EAGLE [ Li et al., 2024a ] with future hidden features. These methods parallelize proposals within an explicit token trace, whereas LOTUS parallelizes latent reasoning and decodes only the answer suffix.

## Appendix B Notation

Table 12 summarizes the notation used throughout the paper.

## Appendix C Closely Related Latent Reasoning Methods

We position LOTUS and its auxiliary-decoder variant LOTUS-aux against the most closely related latent reasoning methods.

##### Coconut [ Hao et al., 2025 ]

introduced a curriculum-based approach that progressively replaces explicit CoT steps with latent tokens conditioned autoregressively on the previous latent. Both LOTUS and LOTUS-aux depart from Coconut in two ways: (i) they use a padded latent prefix processed by a looped Transformer, so all K {K} steps are decoded in parallel, not autoregressively, giving the 𝒪 ⁡ ( R ) {{{{\mathcal{O}}}({R})}} -iteration thought-phase cost, and (ii) they add step-aligned supervision on the latent blocks, where Coconut supervises only the answer. Our ablation without latent supervision ( Table 4 ) is the Coconut-style regime within our looped architecture, and it underperforms both variants.

##### CODI [ Shen et al., 2025 ] and SIM-CoT [ Wei et al., 2025 ]

retain Coconut’s autoregressive latent decoding but add an auxiliary supervision target: CODI aligns the hidden state of a single designated token with the corresponding teacher hidden state from a CoT model, and SIM-CoT trains an auxiliary autoregressive decoder that aligns each latent token with the corresponding CoT token. Our LOTUS removes the auxiliary path entirely, supervising the model’s own post-loop hidden states through the answer LM head at block granularity. LOTUS-aux keeps an auxiliary autoregressive decoder, but produces all blocks in parallel through the looped backbone and supervises a full block per step rather than one latent per step.

##### Parallel Continuous CoT (PCCoT) [ Wu et al., 2025 ] and KaVa [ Kuzina et al., 2026 ]

form a single Jacobi-iteration family: PCCoT introduces the backbone and KaVa builds on it. They both refine a fixed budget of 24 24 latent tokens over 3 3 Jacobi iterations and differ only in supervision. Like LOTUS, they process latent symbols in parallel and refine them over passes, but differ from our methods in three ways: (i) Architecture : PCCoT and KaVa use right-shifted Jacobi iterations to simulate sequential continuous CoT in parallel [ Hao et al., 2025 ] , so their latents are continuous thought vectors with no fixed alignment to individual gold CoT steps. LOTUS instead pins block i {i} to fixed positions across all loops, so it always maps to gold CoT step i {i} and is naturally supervised against that step’s tokens; (ii) Supervision : PCCoT uses CODI [ Shen et al., 2025 ] , and KaVa supervises indirectly by distilling a compressed teacher KV-cache, whereas both our variants supervise each latent block against its exact gold CoT step; and (iii) Depth : PCCoT reports accuracy peaking at about 3 3 Jacobi iterations and degrading beyond that, which is why KaVa inherits the 3 3 -iteration setting, whereas LOTUS improves monotonically with loop depth up to 6 ( Table 6 ).

##### Comparison with latent methods at other compute budgets.

The main results ( Table 1 , Section 4.2 ) compare LOTUS against latent methods at the same sequential step budget (CODI and CODI + SIM-CoT, both R = 6 {R}{=}6 ). Here we provide a more comprehensive accuracy comparison in Table 13 against latent methods whose sequential compute budget instead differs from LOTUS’s R = 6 {R}{=}6 sequential steps. Coconut and Coconut + SIM-CoT decode latents autoregressively under the curriculum of Hao et al. [2025] ( 10 10 autoregressive latent tokens), while PCCoT and KaVa decode in parallel over a shared Jacobi-iteration backbone ( 24 24 latent tokens in total, refined over 3 3 iterations). We take accuracies from Wei et al. [2025] , Kuzina et al. [2026] , and Wu et al. [2025] for each method. LOTUS attains higher in-domain GSM8K accuracy than KaVa at both shared scales ( 57.3 57.3 vs. 56.5 56.5 at 1 1 B, 70.0 70.0 vs. 65.7 65.7 at 3 3 B), staying within 1.5 1.5 points of explicit CoT while PCCoT and KaVa fall further behind as the backbone grows. The missing entries follow what each source reports: Wei et al. [2025] evaluate Coconut only up to Llama-3.2-1B , attributing Coconut’s failure at larger scale to a latent-instability issue in which the latent representations become homogeneous and lose semantic diversity, causing training to collapse as the latent budget grows. Kuzina et al. [2026] do not run KaVa on GPT-2 .

##### Natural-language CoT stress-test.

For the natural-language CoT stress test in Section 4.2 , we use the Llama-3.2-3B-Instruct backbone. The explicit CoT run reaches 68.41 % 68.41\% GSM8K accuracy with 963.6 963.6 ms thought latency. The looped run reaches 68.13 % 68.13\% accuracy with 140.8 140.8 ms thought latency. The latent baselines trail well behind on GSM8K (PCCoT 47.6 47.6 , CODI 55.9 55.9 , KaVa 60.0 60.0 ). Out of domain, LOTUS leads all methods on SVAMP ( 73.40 73.40 , above explicit CoT’s 71.93 71.93 ) and stays close to explicit CoT on GSM-Hard ( 16.27 16.27 vs. 18.27 18.27 ), ahead of every latent baseline ( Table 14 ). We exclude SIM-CoT from this comparison because Suleymanzade et al. [2026b] report weak SIM-CoT accuracy on the natural-language subset.

## Appendix D Experimental Details

This appendix includes the training and evaluation details in Section 4.1 .

##### Latent prefix budget.

Each of the K {K} blocks holds c {c} latent positions, so the prefix spans K ​ c {K}{c} positions: c = 25 {c}{=}25 ( 150 150 positions) for the Llama backbones and c = 13 {c}{=}13 ( 78 78 positions) for GPT-2 . We use K = 6 {K}{=}6 , which covers the reference CoT step count for 99 % 99\% of GSM8k-Aug examples. Longer problems fall back to autoregressive completion of the unsupervised tail.

##### Training curriculum.