Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — part 1/6

Cleaned-text code points [0, 15682). Bibliography omitted here; source.md includes it.

# Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

###### Abstract

Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model’s hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1 1 B parameters, and the gap widens with scale. Looped , or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes K K latent blocks in parallel for R R iterations, with a cross-entropy loss on each latent position’s gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s ). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by 2.5 × 2.5\times – 6.9 × 6.9\times from compact math expressions to natural language. Projecting LOTUS’s post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential. Code is available at https://github.com/yingfan-bot/lotus .

## 1 Introduction

Scaling inference compute, i.e., letting a model “think” before it answers, has become a dominant lever for increasing language model capabilities, with stronger performance now coming from longer reasoning chains rather than from model size alone ( DeepSeek-AI, 2025 ; OpenAI, 2026 ) . Chain-of-thought (CoT) reasoning ( Wei et al., 2022 ) , where the model emits intermediate reasoning steps, drives this trend. However, since each token must be decoded sequentially, generating a CoT of length N {N} takes N {N} sequential model evaluations, making reasoning costly.

Latent reasoning aims to achieve the same at a fraction of the cost: it carries out the intermediate computation in continuous hidden states rather than decoded tokens, condensing many steps into fewer model evaluations. On small backbones such as GPT-2 ( Radford et al., ) , latent methods ( Hao et al., 2025 ; Shen et al., 2025 ; Wei et al., 2025 ) already match CoT accuracy. Yet at the scales where CoT matters most, the promise breaks down: beyond 1 1 B parameters, no existing latent method matches explicit CoT on math reasoning, and the gap widens with model size ( Wei et al., 2025 ) .

We identify two common issues in latent methods. (P1) Sequential generation: methods like Coconut ( Hao et al., 2025 ) , CODI ( Shen et al., 2025 ) , and SIM-CoT ( Wei et al., 2025 ) produce latent thoughts in a pure autoregressive way, so the number of sequential forward passes still scales linearly with the number of latent tokens, keeping CoT’s sequential generation bottleneck while giving up the readable intermediate steps that explicit CoT provides. (P2) Lack of CoT grounding: Explicit CoT gives every reasoning step a direct, position-aligned token target. Without similarly grounded supervision, latent traces can drift from meaningful computation and destabilize training at scale ( Li et al., 2026b ; Zou et al., 2026 ) . Methods like PCCoT ( Wu et al., 2025 ) and KaVa ( Kuzina et al., 2026 ) mitigate the sequential bottleneck with Jacobi-style iteration but still ground their latents through indirect signals that fall short of CoT-aligned target, such as hidden-state distillation from a teacher CoT model or a compressed teacher key–value cache.

An architecture that resolves both issues at once would refine its latents in a few parallel passes rather than one autoregressive step per token, and would ground them in a target as direct as explicit CoT’s. Looped , or recurrent-depth, Transformers are a natural fit for the first requirement: they reuse the same weights across iterations to add computation depth without extra parameters, a design that recent work scales to billion-parameter pretraining ( Geiping et al., 2025a ; Zhu et al., 2025c ; Zeng et al., 2026b ) . On parallelizable problems, they can provably reach a solution in fewer model iterations CoT ( Saunshi et al., 2025 ) ; for example, LPTs solve graph reachability in a logarithmic number of iterations ( Merrill and Sabharwal, 2025a ) —an exponential improvement over sequential latent methods. This raises the question we study: can a looped Transformer bridge the gap to explicit CoT while reasoning in fewer sequential steps?

We find that pairing a looped padded backbone with simple supervision on the gold CoT tokens is surprisingly effective. We present LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s ). LOTUS places K {K} learnable padded latent blocks of c {c} tokens each between the question and answer, and processes the sequence with the base model R {R} times. An explicit CoT of length N {N} is thus processed in R ≪ N {R}\!\ll\!{N} iterations of dense parallel computation rather than N {N} sequential per-token generations, which addresses P1 . To address P2 , LOTUS supervises the latents directly through the base model’s own LM head: a cross-entropy loss aligns each latent position to its corresponding gold CoT-step token, all in parallel. The supervision target can also be routed through an auxiliary decoder that is conditioned on all latents and scores the gold CoT tokens under teacher forcing, which we instantiate on the identical looped backbone as LOTUS-aux. Both routings reach near-CoT accuracy at the 3B scale. LOTUS is, to our knowledge, the first latent reasoning method at the Llama-3.2-3B-Instruct scale to bridge the in-domain gap to explicit CoT on GSM8k ( Figure 1 ), while surpassing CoT on the out-of-domain average and cutting thinking time by 2.5 × \mathbf{2.5\times} . On a more verbose natural-language CoT stress test, LOTUS is on par with explicit CoT in accuracy while reducing thought-phase latency by 6.9 × \mathbf{6.9\times} . LOTUS’s latent space is also interpretable: reading the post-loop latents through the base LM head recovers the gold reasoning steps, and even surfaces alternative valid intermediate steps the model was never trained on, evidence that the latents are genuinely CoT-aligned rather than opaque. We summarize our contributions as follows: • We propose the recipe of looped padded Transformers with parallel cross-entropy supervision on gold CoT tokens. The method uses a padded latent prefix ( Section 3.1 ), refines it with a looped backbone ( Section 3.2 ), and supervises the post-loop latents through the base LM head ( Section 3.3 ). We also instantiate LOTUS-aux, which routes the same latent supervision through an auxiliary decoder used only during training ( Section 3.4 ).

• We show that LOTUS is, to our knowledge, the first latent reasoning method to bridge the latent–explicit CoT gap on math reasoning at the 3B scale: on Llama-3.2-3B-Instruct , it brings the GSM8K test accuracy to within 1 1 point of explicit CoT and surpasses explicit CoT on the out-of-domain math average. It also cuts thought-phase latency by 2.5 × 2.5\times in the math-expression setting and by 6.9 × 6.9\times in the natural-language CoT stress test ( Section 4.2 ).

• Ablations show that both the parallel supervision design ( Section 4.3.1 ) and the looped architecture design (with enough block width and loop depth, Section 4.3.2 ) are important. The direct LM-head routing is robust across scale, whereas the auxiliary decoder routing matches it at 3B but degrades on smaller backbones ( Section 4.2 ).

• LOTUS yields a transparent, CoT-aligned latent space. Reading the post-loop latents recovers most of the gold intermediate tokens ( Section 5.1 ) while assigning nontrivial mass to unseen-but-valid alternatives ( Section 5.2 ). A loss ablation shows that the CoT supervision anchors the readout to the gold chain, while answer supervision helps select the coherent joint ( Section 5.3 ).

## 2 Preliminaries

A glossary of all notation is provided in Table 12 in Appendix B .

##### Explicit CoT.

A language model (LM) f 𝜽 {f_{{{{\bm{\theta}}}}}} can map a question Q {{Q}} to an answer A {{A}} via multi-step reasoning: generating S {S} intermediate steps T 1 , … , T S {T_{1}},\ldots,{T_{{S}}} before the answer, where T i {T_{{i}}} ( i ∈ { 1 , … , S } {i}\in\{1,\dots,{S}\} ) may span multiple tokens. We write N {N} for the total number of CoT tokens across all S {S} steps. Standard CoT supervision trains all CoT tokens in parallel via teacher forcing, but at inference time the tokens must be generated sequentially, increasing latency. We refer to this setup as Explicit CoT.

##### Latent reasoning

replaces decoded CoT tokens with multi-step inference in the model’s hidden states, which are commonly referred to as latent thoughts. Zhu et al. (2025a) show latent thoughts carry strictly more information than discrete tokens: linearly many latent steps in a Transformer can solve graph reachability that would otherwise require quadratically many explicit CoT steps.

##### Looped padded Transformers.

A looped Transformer ( Dehghani et al., 2019 ) applies f 𝜽 {f_{{{{\bm{\theta}}}}}} ’s backbone multiple times, gaining depth without adding parameters. Padding the input with extra learnable tokens ( Goyal et al., 2024 ; Pfau et al., 2024 ) adds parallel computation to each iteration. The combination of the two creates looped padded Transformers (LPTs), which are more capable than either component alone.

##### Prior latent reasoning methods.

We contrast LOTUS against the most closely related latent reasoning methods, which we summarize here (details in Appendix C ). Coconut ( Hao et al., 2025 ) replaces each CoT step with a c {c} latent tokens generated autoregressively from the previous one, under a curriculum, and supervises only the answer. CODI ( Shen et al., 2025 ) keeps this autoregressive latent decoding but adds a distillation target, aligning one designated latent’s hidden state with the corresponding hidden state of a teacher CoT model. SIM-CoT ( Wei et al., 2025 ) instead trains an auxiliary autoregressive decoder that aligns each latent with its CoT token. PCCoT ( Wu et al., 2025 ) and KaVa ( Kuzina et al., 2026 ) drop the autoregressive nature, refining a fixed budget of latent tokens in parallel over a few Jacobi iterations; they ground the latents indirectly, through CODI-style distillation and a compressed teacher key–value cache, respectively. We refer the reader to Appendix A for an extended discussion of related work on latent reasoning and looped Transformers, and to Appendix C for a detailed comparison with the related latent reasoning methods.

## 3 Method

Although LPTs are expressive enough to carry out efficient parallel reasoning, without structured supervision they often fail to generalize out-of-distribution ( Altabaa et al., 2025 ) . This motivates our central design choice: grounding the latent computation directly in the gold CoT tokens. We turn a standard LM into a latent reasoner with a recipe built from two ingredients: (i) a padded latent prefix processed by an LPT , and (ii) parallel cross-entropy supervision on exact gold CoT tokens . We instantiate this recipe as LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s , Figure 2 ), which reads the supervision directly through the base model’s LM head. We also study LOTUS-aux ( Figure 3 ), which instead routes the supervision through an auxiliary decoder. We detail the padded latent prefix in Section 3.1 , the looped computation in Section 3.2 , and LOTUS’s objective—together with an analysis of why it works ( Section 3.3.2 )—in Section 3.3 , then introduce the LOTUS-aux variant in Section 3.4 .

### 3.1 Padded Latent Prefix

For a question Q {{Q}} , S {S} chain-of-thought (CoT) steps T 1 , … , T S {T_{1}},\dots,{T_{{S}}} , and answer A = ( A 1 , … , A | A | ) {{A}}=(A_{1},\dots,A_{|{{A}}|}) (with A | A | = ⟨ EoS ⟩ A_{|{{A}}|}=\langle\text{EoS}\rangle ), we construct an input 𝒙 = [ Q , ⟨ BoT ⟩ , ⟨ lat ⟩ ⋯ ⟨ lat ⟩ ⏟ block ​ 1 ​ ( c ​ tokens ) , ⋯ , ⟨ lat ⟩ ⋯ ⟨ lat ⟩ ⏟ block ​ K ​ ( c ​ tokens ) , ⟨ EoT ⟩ , A ] , {\bm{x}}\;=\;\big[\,{{Q}},\;{\langle\texttt{BoT}\rangle},\;\underbrace{{\langle\texttt{lat}\rangle}\cdots{\langle\texttt{lat}\rangle}}_{\text{block }1\;({c}\text{ tokens})},\;\cdots,\;\underbrace{{\langle\texttt{lat}\rangle}\cdots{\langle\texttt{lat}\rangle}}_{\text{block }{K}\;({c}\text{ tokens})},\;{\langle\texttt{EoT}\rangle},\;{{A}}\,\big], (1) where ⟨ lat ⟩ {\langle\texttt{lat}\rangle} is a learnable latent token shared across positions, and ⟨ BoT ⟩ , ⟨ EoT ⟩ {\langle\texttt{BoT}\rangle},{\langle\texttt{EoT}\rangle} are learnable special tokens that delimit the latent region. The block budget K {K} and per-block width c {c} are fixed hyperparameters, so the latent region contains K ​ c {K}{c} latent tokens plus the two delimiters. The latent prefix follows prior work on padding or pause tokens ( Goyal et al., 2024 ; Pfau et al., 2024 ) , with each block i {i} intended to encode the latent representation of CoT step T i {T_{{i}}} , i ∈ { 1 , … , K } {i}\in\{1,\dots,{K}\} . Unlike the example-dependent step count S {S} of Section 2 , the block budget K {K} is fixed across examples; we choose it to cover the step counts in our data, so that K ≥ S {K}\geq{S} on almost all examples and each block can align to one CoT step ( Appendix D ).

### 3.2 Looped Latent Computation

Let f 𝜽 {f_{{{{\bm{\theta}}}}}} be the base causal LM and 𝑬 ∈ ℝ K ​ c × d {{{{\bm{E}}}}}\in{{\mathbb{R}}}^{{K}{c}\times{d}} the learnable latent embeddings, indexed as 𝑬 i , j {{{{\bm{E}}}}}_{{i},j} for block i ∈ { 1 , … , K } {i}\in\{1,\dots,{K}\} and intra-block position j ∈ { 1 , … , c } j\in\{1,\dots,{c}\} . LOTUS first runs a single forward pass over the prefix [ Q , ⟨ BoT ⟩ ] [{{Q}},{\langle\texttt{BoT}\rangle}] to populate a KV cache 𝒞 pre {\mathcal{C}}_{\text{pre}} , which is reused throughout training and inference without recomputation. It then iterates f 𝜽 {f_{{{{\bm{\theta}}}}}} for R {R} iterations, each refining the latent embeddings while attending to 𝒞 pre {\mathcal{C}}_{\text{pre}} (the trailing ⟨ EoT ⟩ {\langle\texttt{EoT}\rangle} and the answer suffix A {{A}} do not enter the loop). Writing 𝒉 ( t ) ∈ ℝ K ​ c × d {{{\bm{h}}}^{({t})}}\in{{\mathbb{R}}}^{{K}{c}\times{d}} for the hidden states at the latent positions after iteration t {t} , with entries 𝒉 i , j ( t ) {{{\bm{h}}}^{({t})}_{{i},j}} : 𝒉 ( 0 ) \displaystyle{{{\bm{h}}}^{(0)}} = f 𝜽 ​ ( 𝑬 | 𝒞 pre ) , \displaystyle=\;{f_{{{{\bm{\theta}}}}}}\!\left({{{{\bm{E}}}}}\;\middle|\;{\mathcal{C}}_{\text{pre}}\right), (2) 𝒉 ( t ) \displaystyle{{{\bm{h}}}^{({t})}} = f 𝜽 ( 𝑬 + 𝒉 ( t − 1 ) | 𝒞 pre ) , t = 1 , … , R . \displaystyle=\;{f_{{{{\bm{\theta}}}}}}\!\left({{{{\bm{E}}}}}+{{{\bm{h}}}^{({t}-1)}}\;\middle|\;{\mathcal{C}}_{\text{pre}}\right),\quad{t}=1,\dots,{R}. where the brackets denote subsequence concatenation. This is a finite-unroll, input-injected recurrence over a looped Transformer ( Dehghani et al., 2019 ; Fan et al., 2025 ) , with a fixed unroll depth R {R} and gradients propagated through all R {R} iterations.

### 3.3 Direct Step-Aligned Supervision

#### 3.3.1 Objective

##### Step CoT supervision loss.