# Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

###### Abstract

Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model’s hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1 1 B parameters, and the gap widens with scale. Looped , or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes K K latent blocks in parallel for R R iterations, with a cross-entropy loss on each latent position’s gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s ). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by 2.5 × 2.5\times – 6.9 × 6.9\times from compact math expressions to natural language. Projecting LOTUS’s post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential. Code is available at https://github.com/yingfan-bot/lotus .

## 1 Introduction

Scaling inference compute, i.e., letting a model “think” before it answers, has become a dominant lever for increasing language model capabilities, with stronger performance now coming from longer reasoning chains rather than from model size alone ( DeepSeek-AI, 2025 ; OpenAI, 2026 ) . Chain-of-thought (CoT) reasoning ( Wei et al., 2022 ) , where the model emits intermediate reasoning steps, drives this trend. However, since each token must be decoded sequentially, generating a CoT of length N {N} takes N {N} sequential model evaluations, making reasoning costly.

Latent reasoning aims to achieve the same at a fraction of the cost: it carries out the intermediate computation in continuous hidden states rather than decoded tokens, condensing many steps into fewer model evaluations. On small backbones such as GPT-2 ( Radford et al., ) , latent methods ( Hao et al., 2025 ; Shen et al., 2025 ; Wei et al., 2025 ) already match CoT accuracy. Yet at the scales where CoT matters most, the promise breaks down: beyond 1 1 B parameters, no existing latent method matches explicit CoT on math reasoning, and the gap widens with model size ( Wei et al., 2025 ) .

We identify two common issues in latent methods. (P1) Sequential generation: methods like Coconut ( Hao et al., 2025 ) , CODI ( Shen et al., 2025 ) , and SIM-CoT ( Wei et al., 2025 ) produce latent thoughts in a pure autoregressive way, so the number of sequential forward passes still scales linearly with the number of latent tokens, keeping CoT’s sequential generation bottleneck while giving up the readable intermediate steps that explicit CoT provides. (P2) Lack of CoT grounding: Explicit CoT gives every reasoning step a direct, position-aligned token target. Without similarly grounded supervision, latent traces can drift from meaningful computation and destabilize training at scale ( Li et al., 2026b ; Zou et al., 2026 ) . Methods like PCCoT ( Wu et al., 2025 ) and KaVa ( Kuzina et al., 2026 ) mitigate the sequential bottleneck with Jacobi-style iteration but still ground their latents through indirect signals that fall short of CoT-aligned target, such as hidden-state distillation from a teacher CoT model or a compressed teacher key–value cache.

An architecture that resolves both issues at once would refine its latents in a few parallel passes rather than one autoregressive step per token, and would ground them in a target as direct as explicit CoT’s. Looped , or recurrent-depth, Transformers are a natural fit for the first requirement: they reuse the same weights across iterations to add computation depth without extra parameters, a design that recent work scales to billion-parameter pretraining ( Geiping et al., 2025a ; Zhu et al., 2025c ; Zeng et al., 2026b ) . On parallelizable problems, they can provably reach a solution in fewer model iterations CoT ( Saunshi et al., 2025 ) ; for example, LPTs solve graph reachability in a logarithmic number of iterations ( Merrill and Sabharwal, 2025a ) —an exponential improvement over sequential latent methods. This raises the question we study: can a looped Transformer bridge the gap to explicit CoT while reasoning in fewer sequential steps?

We find that pairing a looped padded backbone with simple supervision on the gold CoT tokens is surprisingly effective. We present LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s ). LOTUS places K {K} learnable padded latent blocks of c {c} tokens each between the question and answer, and processes the sequence with the base model R {R} times. An explicit CoT of length N {N} is thus processed in R ≪ N {R}\!\ll\!{N} iterations of dense parallel computation rather than N {N} sequential per-token generations, which addresses P1 . To address P2 , LOTUS supervises the latents directly through the base model’s own LM head: a cross-entropy loss aligns each latent position to its corresponding gold CoT-step token, all in parallel. The supervision target can also be routed through an auxiliary decoder that is conditioned on all latents and scores the gold CoT tokens under teacher forcing, which we instantiate on the identical looped backbone as LOTUS-aux. Both routings reach near-CoT accuracy at the 3B scale. LOTUS is, to our knowledge, the first latent reasoning method at the Llama-3.2-3B-Instruct scale to bridge the in-domain gap to explicit CoT on GSM8k ( Figure 1 ), while surpassing CoT on the out-of-domain average and cutting thinking time by 2.5 × \mathbf{2.5\times} . On a more verbose natural-language CoT stress test, LOTUS is on par with explicit CoT in accuracy while reducing thought-phase latency by 6.9 × \mathbf{6.9\times} . LOTUS’s latent space is also interpretable: reading the post-loop latents through the base LM head recovers the gold reasoning steps, and even surfaces alternative valid intermediate steps the model was never trained on, evidence that the latents are genuinely CoT-aligned rather than opaque. We summarize our contributions as follows: • We propose the recipe of looped padded Transformers with parallel cross-entropy supervision on gold CoT tokens. The method uses a padded latent prefix ( Section 3.1 ), refines it with a looped backbone ( Section 3.2 ), and supervises the post-loop latents through the base LM head ( Section 3.3 ). We also instantiate LOTUS-aux, which routes the same latent supervision through an auxiliary decoder used only during training ( Section 3.4 ).

• We show that LOTUS is, to our knowledge, the first latent reasoning method to bridge the latent–explicit CoT gap on math reasoning at the 3B scale: on Llama-3.2-3B-Instruct , it brings the GSM8K test accuracy to within 1 1 point of explicit CoT and surpasses explicit CoT on the out-of-domain math average. It also cuts thought-phase latency by 2.5 × 2.5\times in the math-expression setting and by 6.9 × 6.9\times in the natural-language CoT stress test ( Section 4.2 ).

• Ablations show that both the parallel supervision design ( Section 4.3.1 ) and the looped architecture design (with enough block width and loop depth, Section 4.3.2 ) are important. The direct LM-head routing is robust across scale, whereas the auxiliary decoder routing matches it at 3B but degrades on smaller backbones ( Section 4.2 ).

• LOTUS yields a transparent, CoT-aligned latent space. Reading the post-loop latents recovers most of the gold intermediate tokens ( Section 5.1 ) while assigning nontrivial mass to unseen-but-valid alternatives ( Section 5.2 ). A loss ablation shows that the CoT supervision anchors the readout to the gold chain, while answer supervision helps select the coherent joint ( Section 5.3 ).

## 2 Preliminaries

A glossary of all notation is provided in Table 12 in Appendix B .

##### Explicit CoT.

A language model (LM) f 𝜽 {f_{{{{\bm{\theta}}}}}} can map a question Q {{Q}} to an answer A {{A}} via multi-step reasoning: generating S {S} intermediate steps T 1 , … , T S {T_{1}},\ldots,{T_{{S}}} before the answer, where T i {T_{{i}}} ( i ∈ { 1 , … , S } {i}\in\{1,\dots,{S}\} ) may span multiple tokens. We write N {N} for the total number of CoT tokens across all S {S} steps. Standard CoT supervision trains all CoT tokens in parallel via teacher forcing, but at inference time the tokens must be generated sequentially, increasing latency. We refer to this setup as Explicit CoT.

##### Latent reasoning

replaces decoded CoT tokens with multi-step inference in the model’s hidden states, which are commonly referred to as latent thoughts. Zhu et al. (2025a) show latent thoughts carry strictly more information than discrete tokens: linearly many latent steps in a Transformer can solve graph reachability that would otherwise require quadratically many explicit CoT steps.

##### Looped padded Transformers.

A looped Transformer ( Dehghani et al., 2019 ) applies f 𝜽 {f_{{{{\bm{\theta}}}}}} ’s backbone multiple times, gaining depth without adding parameters. Padding the input with extra learnable tokens ( Goyal et al., 2024 ; Pfau et al., 2024 ) adds parallel computation to each iteration. The combination of the two creates looped padded Transformers (LPTs), which are more capable than either component alone.

##### Prior latent reasoning methods.

We contrast LOTUS against the most closely related latent reasoning methods, which we summarize here (details in Appendix C ). Coconut ( Hao et al., 2025 ) replaces each CoT step with a c {c} latent tokens generated autoregressively from the previous one, under a curriculum, and supervises only the answer. CODI ( Shen et al., 2025 ) keeps this autoregressive latent decoding but adds a distillation target, aligning one designated latent’s hidden state with the corresponding hidden state of a teacher CoT model. SIM-CoT ( Wei et al., 2025 ) instead trains an auxiliary autoregressive decoder that aligns each latent with its CoT token. PCCoT ( Wu et al., 2025 ) and KaVa ( Kuzina et al., 2026 ) drop the autoregressive nature, refining a fixed budget of latent tokens in parallel over a few Jacobi iterations; they ground the latents indirectly, through CODI-style distillation and a compressed teacher key–value cache, respectively. We refer the reader to Appendix A for an extended discussion of related work on latent reasoning and looped Transformers, and to Appendix C for a detailed comparison with the related latent reasoning methods.

## 3 Method

Although LPTs are expressive enough to carry out efficient parallel reasoning, without structured supervision they often fail to generalize out-of-distribution ( Altabaa et al., 2025 ) . This motivates our central design choice: grounding the latent computation directly in the gold CoT tokens. We turn a standard LM into a latent reasoner with a recipe built from two ingredients: (i) a padded latent prefix processed by an LPT , and (ii) parallel cross-entropy supervision on exact gold CoT tokens . We instantiate this recipe as LOTUS ( Lo oped T ransformers with parallel s u pervision on latent s , Figure 2 ), which reads the supervision directly through the base model’s LM head. We also study LOTUS-aux ( Figure 3 ), which instead routes the supervision through an auxiliary decoder. We detail the padded latent prefix in Section 3.1 , the looped computation in Section 3.2 , and LOTUS’s objective—together with an analysis of why it works ( Section 3.3.2 )—in Section 3.3 , then introduce the LOTUS-aux variant in Section 3.4 .

### 3.1 Padded Latent Prefix

For a question Q {{Q}} , S {S} chain-of-thought (CoT) steps T 1 , … , T S {T_{1}},\dots,{T_{{S}}} , and answer A = ( A 1 , … , A | A | ) {{A}}=(A_{1},\dots,A_{|{{A}}|}) (with A | A | = ⟨ EoS ⟩ A_{|{{A}}|}=\langle\text{EoS}\rangle ), we construct an input 𝒙 = [ Q , ⟨ BoT ⟩ , ⟨ lat ⟩ ⋯ ⟨ lat ⟩ ⏟ block ​ 1 ​ ( c ​ tokens ) , ⋯ , ⟨ lat ⟩ ⋯ ⟨ lat ⟩ ⏟ block ​ K ​ ( c ​ tokens ) , ⟨ EoT ⟩ , A ] , {\bm{x}}\;=\;\big[\,{{Q}},\;{\langle\texttt{BoT}\rangle},\;\underbrace{{\langle\texttt{lat}\rangle}\cdots{\langle\texttt{lat}\rangle}}_{\text{block }1\;({c}\text{ tokens})},\;\cdots,\;\underbrace{{\langle\texttt{lat}\rangle}\cdots{\langle\texttt{lat}\rangle}}_{\text{block }{K}\;({c}\text{ tokens})},\;{\langle\texttt{EoT}\rangle},\;{{A}}\,\big], (1) where ⟨ lat ⟩ {\langle\texttt{lat}\rangle} is a learnable latent token shared across positions, and ⟨ BoT ⟩ , ⟨ EoT ⟩ {\langle\texttt{BoT}\rangle},{\langle\texttt{EoT}\rangle} are learnable special tokens that delimit the latent region. The block budget K {K} and per-block width c {c} are fixed hyperparameters, so the latent region contains K ​ c {K}{c} latent tokens plus the two delimiters. The latent prefix follows prior work on padding or pause tokens ( Goyal et al., 2024 ; Pfau et al., 2024 ) , with each block i {i} intended to encode the latent representation of CoT step T i {T_{{i}}} , i ∈ { 1 , … , K } {i}\in\{1,\dots,{K}\} . Unlike the example-dependent step count S {S} of Section 2 , the block budget K {K} is fixed across examples; we choose it to cover the step counts in our data, so that K ≥ S {K}\geq{S} on almost all examples and each block can align to one CoT step ( Appendix D ).

### 3.2 Looped Latent Computation

Let f 𝜽 {f_{{{{\bm{\theta}}}}}} be the base causal LM and 𝑬 ∈ ℝ K ​ c × d {{{{\bm{E}}}}}\in{{\mathbb{R}}}^{{K}{c}\times{d}} the learnable latent embeddings, indexed as 𝑬 i , j {{{{\bm{E}}}}}_{{i},j} for block i ∈ { 1 , … , K } {i}\in\{1,\dots,{K}\} and intra-block position j ∈ { 1 , … , c } j\in\{1,\dots,{c}\} . LOTUS first runs a single forward pass over the prefix [ Q , ⟨ BoT ⟩ ] [{{Q}},{\langle\texttt{BoT}\rangle}] to populate a KV cache 𝒞 pre {\mathcal{C}}_{\text{pre}} , which is reused throughout training and inference without recomputation. It then iterates f 𝜽 {f_{{{{\bm{\theta}}}}}} for R {R} iterations, each refining the latent embeddings while attending to 𝒞 pre {\mathcal{C}}_{\text{pre}} (the trailing ⟨ EoT ⟩ {\langle\texttt{EoT}\rangle} and the answer suffix A {{A}} do not enter the loop). Writing 𝒉 ( t ) ∈ ℝ K ​ c × d {{{\bm{h}}}^{({t})}}\in{{\mathbb{R}}}^{{K}{c}\times{d}} for the hidden states at the latent positions after iteration t {t} , with entries 𝒉 i , j ( t ) {{{\bm{h}}}^{({t})}_{{i},j}} : 𝒉 ( 0 ) \displaystyle{{{\bm{h}}}^{(0)}} = f 𝜽 ​ ( 𝑬 | 𝒞 pre ) , \displaystyle=\;{f_{{{{\bm{\theta}}}}}}\!\left({{{{\bm{E}}}}}\;\middle|\;{\mathcal{C}}_{\text{pre}}\right), (2) 𝒉 ( t ) \displaystyle{{{\bm{h}}}^{({t})}} = f 𝜽 ( 𝑬 + 𝒉 ( t − 1 ) | 𝒞 pre ) , t = 1 , … , R . \displaystyle=\;{f_{{{{\bm{\theta}}}}}}\!\left({{{{\bm{E}}}}}+{{{\bm{h}}}^{({t}-1)}}\;\middle|\;{\mathcal{C}}_{\text{pre}}\right),\quad{t}=1,\dots,{R}. where the brackets denote subsequence concatenation. This is a finite-unroll, input-injected recurrence over a looped Transformer ( Dehghani et al., 2019 ; Fan et al., 2025 ) , with a fixed unroll depth R {R} and gradients propagated through all R {R} iterations.

### 3.3 Direct Step-Aligned Supervision

#### 3.3.1 Objective

##### Step CoT supervision loss.

After R {R} iterations of looping ( Section 3.2 ), we supervise each latent position ( i , j ) ({i},j) in the grid with the corresponding CoT step token. Let T i = ( T i , 1 , … , T i , c ) {T_{{i}}}=({T_{{i},1}},\dots,{T_{{i},{c}}}) be the tokenized CoT step i {i} padded or truncated to c {c} tokens. We apply a single batched cross-entropy that directly aligns each block to its target step: ℒ step = 1 N step ​ ∑ i = 1 K ∑ j = 1 c CE ⁡ ( f head ​ ( 𝒉 i , j ( R ) ) , T i , j ) , {\mathcal{L}_{\mathrm{step}}}\;=\;\frac{1}{{N_{\mathrm{step}}}}\sum_{{i}=1}^{{K}}\sum_{j=1}^{{c}}{\mathrm{CE}}\!\left({f_{\mathrm{head}}}({{{\bm{h}}}^{({R})}_{{i},j}}),\,{T_{{i},j}}\right), (3) where f head {f_{\mathrm{head}}} is the LM head and N step = ∑ i | T i | {N_{\mathrm{step}}}=\sum_{{i}}|{T_{{i}}}| is the total number of supervised (non-padding) CoT tokens. 1 1 1 Padding tokens used to pad CoT steps to length c {c} are ignored by the cross-entropy (and excluded from N step {N_{\mathrm{step}}} ).

##### Answer supervision loss.

ℒ ans {\mathcal{L}_{\mathrm{ans}}} is computed in a separate final forward of f 𝜽 {f_{{{{\bm{\theta}}}}}} that reuses the prefix cache 𝒞 pre {\mathcal{C}}_{\text{pre}} and inserts the post-loop latent hidden states 𝒉 ( R ) {{{\bm{h}}}^{({R})}} at the latent rows: 𝒛 = f 𝜽 ​ ( [ ⟨ EoT ⟩ , A ] | [ 𝒞 pre , 𝒉 ( R ) ] ) , {{{\bm{z}}}}\;=\;{f_{{{{\bm{\theta}}}}}}\!\left([{\langle\texttt{EoT}\rangle},\,{{A}}]\;\middle|\;[{\mathcal{C}}_{\text{pre}},\,{{{\bm{h}}}^{({R})}}]\right), (4) where 𝒛 m ∈ ℝ d {{{\bm{z}}}}_{m}\in{{\mathbb{R}}}^{{d}} collects the resulting hidden state at each answer-suffix position m ∈ { 0 , 1 , … , | A | − 1 } m\in\{0,1,\dots,|{{A}}|-1\} ( m = 0 m=0 being the trailing ⟨ EoT ⟩ {\langle\texttt{EoT}\rangle} position, which predicts A 1 A_{1} ). We denote hidden states here by 𝒛 {{{\bm{z}}}} , distinct from the looped forward latents 𝒉 ( t ) {{{\bm{h}}}^{({t})}} , to mark that they come from different sources. Applying the LM head f head {f_{\mathrm{head}}} , standard next-token cross-entropy on the answer suffix gives: ℒ ans = 1 | A | ​ ∑ m = 0 | A | − 1 CE ⁡ ( f head ​ ( 𝒛 m ) , A m + 1 ) . {\mathcal{L}_{\mathrm{ans}}}\;=\;\frac{1}{|{{A}}|}\sum_{m=0}^{|{{A}}|-1}{\mathrm{CE}}\!\left({f_{\mathrm{head}}}({{{\bm{z}}}}_{m}),\,A_{m+1}\right). (5) Given a step supervision weight λ step {\lambda_{\mathrm{step}}} , the full objective is ℒ = ℒ ans + λ step ​ ℒ step . {\mathcal{L}}\;=\;{\mathcal{L}_{\mathrm{ans}}}+{\lambda_{\mathrm{step}}}\,{\mathcal{L}_{\mathrm{step}}}. (6)

##### Key properties of the objective.

The supervision in Equation 6 has three properties: (i) it is direct : the per-block target T i {T_{{i}}} is scored through the same LM head used to produce the answer; (ii) it is parallel : all K {K} blocks are supervised simultaneously; and (iii) it is post-loop : the readout reads 𝒉 ( R ) {{\bm{h}}}^{({R})} after the final iteration, rather than at every iteration. We ablate the design choice in Section 4.3 , and compare against prior latent reasoning methods in Appendix C .

##### Inference.

At inference, LOTUS runs the looped forward and then decodes the answer. We compute the KV cache of the question Q {{Q}} once, iterate the loop R {R} times to obtain the post-loop latents 𝒉 ( R ) {{{\bm{h}}}^{({R})}} , and then decode the answer A {{A}} autoregressively through the base LM head while conditioning on those latents (same as Figure 2 only without computing the losses). The reasoning is carried entirely by the parallel latent blocks, so the only sequential decoding is over the short answer suffix, which is the source of the latency gains reported in Section 4.2 .

##### No latent decoding for answer generation.

Note that the latent reasoning steps are never read out during answer generation. They are projected to the token space through the LM head only to align them with CoT reasoning during training with ℒ step {\mathcal{L}_{\mathrm{step}}} . The answer is always generated from the latent hidden states, both in the final-forward ℒ ans {\mathcal{L}_{\mathrm{ans}}} pass ( Equation 4 ) and at inference.

#### 3.3.2 Roles of ℒ step {\mathcal{L}_{\mathrm{step}}} and ℒ ans {\mathcal{L}_{\mathrm{ans}}}

A natural worry about the LOTUS objective ( Equation 6 ) is that it supervises each latent position independently : Equation 3 scores every gold CoT token in parallel, with no autoregressive factorization of the chain. Why should independent per-position targets train a model that produces a globally coherent answer? We give a lens that resolves this, distinguishing the role of the two losses and explaining why both are needed.

##### Parallel chain likelihood.

The step loss ℒ step {\mathcal{L}_{\mathrm{step}}} supervises a parallel chain likelihood (PCL) over the readout positions. Maximizing the per-position probability p 𝜽 ​ ( T i , j ∣ Q ) p_{{{{\bm{\theta}}}}}({T_{{i},j}}\mid{Q}) of each gold token at its latent position induces the chain likelihood p 𝜽 PCL ​ ( T ∣ Q ) = ∏ i = 1 K ∏ j = 1 c p 𝜽 ​ ( T i , j ∣ Q ) , p_{{{{\bm{\theta}}}}}^{\mathrm{PCL}}({T}\mid{Q})=\prod_{{i}=1}^{{K}}\prod_{j=1}^{{c}}p_{{{{\bm{\theta}}}}}({T_{{i},j}}\mid{Q}), (7) rather than the autoregressive conditionals p 𝜽 ​ ( T i , j ∣ Q , T < i , T i , < j ) p_{{{{\bm{\theta}}}}}({T_{{i},j}}\mid{Q},{T}_{<{i}},{T_{{i},<j}}) . The factorization treats the readout as conditionally independent across positions, but the latent states themselves are not independent: the looped Transformer computes them jointly over the padded workspace, so the dependence the factorization drops is carried by the shared latent computation rather than by the loss.

##### Complementary roles.

This view makes the division of labor between the two losses precise. First, the support inclusion supp ⁡ ( q ⁡ ( T ∣ Q ) ) ⊆ ∏ i = 1 K ∏ j = 1 c supp ⁡ ( q ⁡ ( T i , j ∣ Q ) ) \mathrm{supp}\!\left({q}({T}\mid{Q})\right)\;\subseteq\;\prod_{{i}=1}^{{K}}\prod_{j=1}^{{c}}\mathrm{supp}\!\left({q}({T_{{i},j}}\mid{Q})\right) (8) shows that every gold chain lies inside the Cartesian product of per-position gold token supports. 2 2 2 Minimizing ℒ step {\mathcal{L}_{\mathrm{step}}} , the cross-entropy to these marginals, pulls the model’s PCL toward that product, i.e. minimizes KL ( ∏ i , j q ( T i , j ∣ Q ) ∥ p 𝜽 PCL ( T ∣ Q ) ) \mathrm{KL}\big(\prod_{{i},j}{q}({T_{{i},j}}\mid{Q})\,\big\|\,p_{{{{\bm{\theta}}}}}^{\mathrm{PCL}}({T}\mid{Q})\big) . So ℒ step {\mathcal{L}_{\mathrm{step}}} plays a support-coverage role: it makes each position place mass on the right gold tokens, without needing to reproduce the joint distribution—which LOTUS never samples, since the answer is decoded directly from the jointly computed latent states. Second, ℒ ans {\mathcal{L}_{\mathrm{ans}}} supplies the global selection pressure that PCL alone lacks: because the answer is trained while conditioning on the entire latent configuration, gradients favor jointly computed hidden states that can actually support the correct answer. The two losses are thus complementary by construction—coverage from ℒ step {\mathcal{L}_{\mathrm{step}}} , joint selection from ℒ ans {\mathcal{L}_{\mathrm{ans}}} —which is exactly the behavior we verify empirically in Section 5.3 , where dropping either loss degrades the gold chain likelihood.

To further probe how LOTUS compares to modeling the autoregressive chain likelihood, we introduce an autoregressive decoder variant in Section 3.4 and compare the results in Section 4.2 .

### 3.4 LOTUS-aux: An Auxiliary Decoder Variant

An alternative route for CoT supervision explored in existing work is autoregressive chain likelihood: an auxiliary autoregressive decoder conditioned on the latent blocks scores the CoT tokens under teacher forcing. This creates a trade-off: in exchange for modeling the chain autoregressively, CoT supervision no longer flows directly through the answer-generating base LM head, and teacher forcing introduces an extra train/inference mismatch.

We instantiate this routing on our looped padded backbone as LOTUS-aux. LOTUS-aux reuses the same looped forward and final forward as LOTUS ( Figure 2 ). The only difference is the supervision in the looped forward, which is routed through an auxiliary decoder rather than read directly through the base LM head. The auxiliary decoder mirrors SIM-CoT ( Wei et al., 2025 ) , but instead of generating latents autoregressively and supervising a single latent per CoT step, LOTUS-aux produces all K {K} latent blocks in parallel through the looped backbone and supervises a full c {c} -token block per step. We describe this routing below.

##### Latent supervision via the auxiliary decoder.

As shown in Figure 3 , the auxiliary decoder g ϕ {g_{\phi}} is an extra decoder model that, at each loop iteration t t , reads latent block 𝒉 ( t ) t {{{\bm{h}}}^{(t)}}_{t} (we select block i = t i=t at iteration t t , the c {c} post-loop latents of the t t -th block) as a c {c} -token prefix and predicts the gold CoT step T t = ( T t , 1 , … , T t , | T t | ) {T_{t}}=(T_{t,1},\dots,T_{t,\lvert{T_{t}}\rvert}) under teacher forcing, where | T t | \lvert{T_{t}}\rvert is the step’s token length. Let 𝒛 ~ ( t ) = ( 𝒛 ~ 0 ( t ) , … , 𝒛 ~ | T t | − 1 ( t ) ) {\widetilde{{{\bm{z}}}}^{(t)}}=({\widetilde{{{\bm{z}}}}^{(t)}_{0}},\dots,{\widetilde{{{\bm{z}}}}^{(t)}_{\lvert{T_{t}}\rvert-1}}) denote the output hidden states at each input CoT position m ∈ { 0 , 1 , … , | T t | − 1 } m\in\{0,1,\dots,\lvert{T_{t}}\rvert-1\} ( m = 0 m{=}0 being the last latent position 𝒉 t , c ( t ) {{{\bm{h}}}^{(t)}_{t,{c}}} , which predicts T t , 1 T_{t,1} , and m > 0 m{>}0 the gold token T t , m T_{t,m} ). Mirroring the final-forward pass of Equation 4 , we generate 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} with 𝒛 ~ ( t ) = g ϕ ( [ 𝒉 t , c ( t ) , T t , 1 , … , T t , | T t | − 1 ] | 𝒉 t , 1 ( t ) , … , 𝒉 t , c − 1 ( t ) ) , {\widetilde{{{\bm{z}}}}^{(t)}}\;=\;{g_{\phi}}\!\left([\,{{{\bm{h}}}^{(t)}_{t,{c}}},\;T_{t,1},\dots,T_{t,\lvert{T_{t}}\rvert-1}\,]\;\middle|\;{{{\bm{h}}}^{(t)}_{t,1}},\dots,{{{\bm{h}}}^{(t)}_{t,{c}-1}}\right), (9) where the first c − 1 {c}-1 latent positions 𝒉 t , 1 ( t ) , … , 𝒉 t , c − 1 ( t ) {{{\bm{h}}}^{(t)}_{t,1}},\dots,{{{\bm{h}}}^{(t)}_{t,{c}-1}} are the conditioning prefix. The teacher-forced next-token cross-entropy on the gold CoT gives: ℒ step aux = 1 N step ​ ∑ t = 1 K ∑ m = 0 | T t | − 1 CE ⁡ ( g head ​ ( 𝒛 ~ m ( t ) ) , T t , m + 1 ) , {\mathcal{L}_{\mathrm{step}}^{\mathrm{aux}}}\;=\;\frac{1}{N_{\text{step}}}\sum_{t=1}^{{K}}\sum_{m=0}^{\lvert{T_{t}}\rvert-1}{\mathrm{CE}}\!\left({g_{\mathrm{head}}}({\widetilde{{{\bm{z}}}}^{(t)}_{m}}),\;T_{t,m+1}\right), (10) where N step = ∑ t = 1 K | T t | N_{\text{step}}=\sum_{t=1}^{{K}}\lvert{T_{t}}\rvert is the total number of supervised CoT tokens. The supervision itself remains parallel: under teacher forcing the gold prefix is fed at every position, so all CoT positions are scored in a single forward pass with no sequential generation. In other words, the autoregressive factorization affects only the loss, not the computation.

##### Per-iteration supervision and the full objective.

Equation 10 reads block i = t i=t at iteration t t (per-iteration, rather than at the final iteration R R ). We validate this choice in Section 4.3.1 , where per-iteration readouts outperform reading all blocks at the final iteration R {R} for the auxiliary decoder routing. The answer loss ℒ ans {\mathcal{L}_{\mathrm{ans}}} from Section 3.3 is unchanged, so the full LOTUS-aux objective is ℒ aux = ℒ ans + λ step ​ ℒ step aux {\mathcal{L}^{\mathrm{aux}}}={\mathcal{L}_{\mathrm{ans}}}+{\lambda_{\mathrm{step}}}\,{\mathcal{L}_{\mathrm{step}}^{\mathrm{aux}}} . In our experiments g ϕ {g_{\phi}} is initialized with a copy of the base LM and trained together with the base LM. Training and architectural details are in Appendix D .

##### Inference.

The auxiliary decoder g ϕ {g_{\phi}} is used only at training time. At inference, LOTUS-aux generates the final answer autoregressively from the post-loop latents exactly as LOTUS does ( Figure 2 (b)), so the inference efficiency gain of LOTUS also applies to LOTUS-aux.

## 4 Experiments

Our experiments investigate four questions: how accurate LOTUS is, how efficient it is at inference, how much each design choice contributes, and whether its latent space is interpretable and CoT-aligned. Section 4.1 introduces the experimental setup, Section 4.2 presents accuracy and latency results against explicit CoT and prior latent baselines, Section 4.3 ablates the supervision design, latent block width c {c} , loop depth R {R} , and inference-time robustness to changing c {c} and R {R} without retraining, and Section 5 analyzes the learned latent representations. Full training and evaluation details are in Appendix D .

### 4.1 Setup

##### Datasets and backbones.

Following SIM-CoT ( Wei et al., 2025 ) , we train on GSM8k-Aug ( Deng et al., 2023 ) with 385k training samples and report in-domain accuracy on the GSM8K test set ( Cobbe et al., 2021 ) , together with three out-of-domain benchmarks: GSM-Hard, MultiArith, and SVAMP. We additionally train on a natural-language version of GSM8K-Aug ( Wu et al., 2025 ) as an efficiency stress test, where each reasoning step is a full-sentence rationale rather than a compact math expression. We use three backbones in increasing size: GPT-2 (124M) ( Radford et al., ) , Llama-3.2-1B-Instruct , and Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 ) .

##### Methods.

We report LOTUS ( K = 6 {K}{=}6 blocks of c = 25 {c}{=}25 tokens on the Llama backbones, c = 13 {c}{=}13 on GPT-2 , and R = 6 {R}{=}6 looped iterations) 3 3 3 K {K} is chosen to cover the target maximum CoT step count: GSM8K solutions have at most six steps for 99 % 99\% of examples, so K = 6 {K}{=}6 assigns one block per step and falls back to autoregressive completion otherwise. and LOTUS + CODI , which adds CODI’s single-position distillation loss. The auxiliary decoder variants LOTUS-aux and LOTUS-aux + CODI ( Section 3.4 ) share the same configuration. We compare with Explicit CoT and No-CoT as reference, and with CODI ( Shen et al., 2025 ) and CODI + SIM-CoT ( Wei et al., 2025 ) for each backbone, which share the sequential latent reasoning budget ( R = 6 {R}{=}6 ) with LOTUS and differ only in the number of latent tokens per step. Latent methods that use a different sequential compute budget: Coconut ( Hao et al., 2025 ) , Coconut + SIM-CoT (with 10 sequentially generated latent thought in total), and the parallel-latent methods PCCoT ( Wu et al., 2025 ) and KaVa ( Kuzina et al., 2026 ) (with 3 sequential steps and 24 latent tokens in total) are compared separately in Table 13 and Table 14 ( Appendix C ).

### 4.2 Main Results

##### LOTUS matches explicit CoT at scale, where prior latent methods fall behind.

LOTUS scales well from GPT-2 to Llama-3.2-3B-Instruct ( Table 1 ). Prior latent methods match explicit CoT at small scale but fall behind as the backbone grows: CODI + SIM-CoT is level with explicit CoT at GPT-2 ( 42.6 42.6 vs. 42.7 42.7 ) yet trails it by 9.2 9.2 points at 3B, and KaVa ( Kuzina et al., 2026 ) widens similarly ( 1.9 1.9 to 5.8 5.8 points, Table 13 ). LOTUS instead stays within ∼ 1.5 {\sim}1.5 points of explicit CoT in-domain at each scale, and its gap does not widen. While the smaller-scale results are rather clustered, the decisive comparison is at Llama-3.2-3B-Instruct : LOTUS surpasses explicit CoT on the out-of-domain average and brings the in-domain GSM8K gap to within 1.5 1.5 points. LOTUS + CODI further tightens it to within 1 1 point.

##### The two routings match at 3B, but only direct LOTUS stays robust at smaller scales.

At 3B, LOTUS-aux performs comparably to LOTUS and well above the CODI + SIM-CoT baseline, which uses an auxiliary decoder of the same size as the base LM. The gains therefore come from the looped padded backbone with parallel CoT supervision, regardless of the routing. This parity holds only at 3B: at smaller scales LOTUS-aux degrades, whereas direct LM-head supervision (LOTUS) stays robust across scale.

##### LOTUS reasons 2.5 × 2.5\times faster than explicit CoT.

The thought phase dominates the difference between methods ( Table 2 ): CoT decodes its chain autoregressively, while LOTUS compresses the same “thinking” into R {R} parallel latent iterations, making its thought 2.5 × \mathbf{2.5\times} faster than CoT and 1.2 × \mathbf{1.2\times} faster than SIM-CoT 4 4 4 We measure latency on a single NVIDIA H100 NVL at batch size 1 1 with greedy decoding, split into query-prefill, thought, and answer phases. SIM-CoT and CODI both replace each explicit CoT step with a single latent token decoded sequentially. SIM-CoT additionally adds LoRA adapters (rank 128 128 ) on top of CODI, which inflates every phase. ( 2.1 × \mathbf{2.1\times} faster than CoT in total 5 5 5 LOTUS’s prefill and answer phases are just the standard LM operations ( Figure 2 ), so they match explicit CoT in Table 2 ( 16.7 16.7 vs. 15.9 15.9 ms and 31.5 31.5 vs. 29.5 29.5 ms); the looped latent thought is the only phase that differs. ). CODI is faster than LOTUS in the thought phase because it decodes only a single latent per step, rather than LOTUS’s K ​ c = 150 {K}{c}\!=\!150 parallel positions, but comes at the cost of accuracy. LOTUS is nonetheless only modestly slower ( 133 133 vs. 88 88 ms) because it consumes all positions in parallel. The inference-time width sweep in Table 8 further separates sequential depth from parallel width: with R {R} fixed, increasing the latent prefix from 6 6 to 300 300 positions changes thought latency by only 30 30 ms.

##### Speedup grows to 6.9 × 6.9\times on natural-language CoT traces.

The main GSM8K-Aug setting uses compact math-expression steps, while natural-language CoT requires explicit models to decode much longer reasoning in natural language. In the 3B natural-language CoT stress test ( Table 3 ), LOTUS is on par with explicit CoT in accuracy ( 68.13 68.13 vs. 68.41 68.41 ) while cutting thought-phase latency from 963.6 963.6 ms to 140.8 140.8 ms with a 6.9 × 6.9\times speedup. On this natural-language setting, LOTUS ( 68.13 % 68.13\% GSM8K) far exceeds the latent baselines PCCoT ( 47.6 % 47.6\% ), CODI ( 55.9 % 55.9\% ), and KaVa ( 60.0 % 60.0\% ) reported by Kuzina et al. (2026) ( Table 14 ), staying within variance of explicit CoT ( Table 3 ).

### 4.3 Ablation Studies

We ablate each component of LOTUS to isolate its contribution. Unless noted, all ablations use Llama-3.2-3B-Instruct , evaluate on GSM8K test set ( Cobbe et al., 2021 ) .

#### 4.3.1 Latent Supervision Design

LOTUS supervises the latent blocks via a direct readout ( ℒ step {\mathcal{L}_{\mathrm{step}}} ) of the post-loop latents 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through the main LM head f head {f_{\mathrm{head}}} . We compare it against five alternatives, holding the rest fixed ( K = R = 6 {K}{=}{R}{=}6 , c = 25 {c}{=}25 , also keeping ℒ ans {\mathcal{L}_{\mathrm{ans}}} ): (i) no latent supervision (only ℒ ans {\mathcal{L}_{\mathrm{ans}}} ); (ii) CODI ( Shen et al., 2025 ) , distilling a single pre-answer latent onto the teacher CoT model’s hidden state; (iii) LOTUS (per-iter), reading 𝒉 ( t ) {{{\bm{h}}}^{(t)}} through f head {f_{\mathrm{head}}} at each iteration t t rather than the post-loop 𝒉 ( R ) {{{\bm{h}}}^{({R})}} ; (iv) LOTUS-aux, reading 𝒉 ( t ) {{{\bm{h}}}^{(t)}} through a separate auxiliary decoder g ϕ {g_{\phi}} per iteration (default, Section 3.4 ); and (v) LOTUS-aux (post-loop), reading 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g ϕ {g_{\phi}} instead.

##### Both the looped backbone and latent supervision are beneficial.

Without any latent supervision, the looped backbone alone ( 63.3 % 63.3\% ) already exceeds CODI + SIM-CoT ( 62.3 % 62.3\% , Table 1 ), which uses a single latent per step; the looped padded backbone is thus already beneficial on its own. Every supervised variant exceeds both no latent supervision and CODI-only, confirming that the gold CoT supervision contributes to the gain. LOTUS (post-loop direct, 70.0 % 70.0\% ) matches the best LOTUS-aux schedule (per-iter, 69.9 % 69.9\% ). The direct (LOTUS) and auxiliary decoder (LOTUS-aux) routings thus perform comparably at their best schedule.

##### The best supervision schedule differs between LOTUS and LOTUS-aux.

LOTUS performs best with post-loop readout ( 70.0 % 70.0\% vs. 68.2 % 68.2\% per-iter), which lets the early latent blocks refine fully until the end of the loop. LOTUS-aux instead works best with per-iteration readout ( 69.9 % 69.9\% vs. 68.4 % 68.4\% post-loop), which shortens the gradient path—possibly helpful because its extra decoder already lengthens it.

#### 4.3.2 Looped Architecture Design

##### Latent token budget per block c c .

We train separate models at c ∈ { 1 , 5 , 10 , 25 , 30 } c\in\{1,5,10,25,30\} to measure how the per-block budget affects accuracy, holding K = 6 K{=}6 fixed ( Table 7 ). Accuracy rises sharply from c = 1 c{=}1 ( 49.7 % 49.7\% ) to c = 5 c{=}5 ( 67.5 % 67.5\% ), then climbs only marginally before plateauing with c = 25 c{=}25 and c = 30 c{=}30 tied at 70.0 % 70.0\% . A single token per CoT step is too narrow for the direct-readout supervision, but moderate widths suffice.

##### Inference-time c c .

We sweep the inference-time budget c ∈ { 1 , 5 , 10 , 25 , 30 , 50 } c\in\{1,5,10,25,30,50\} on a single checkpoint trained at c = 25 c{=}25 ( Table 8 ). Reducing c c below the trained value hurts accuracy ( − 19 -19 points at c = 1 c{=}1 , − 11 -11 points at c = 5 c{=}5 , − 6 -6 points at c = 10 c{=}10 ), while exceeding it slightly helps before plateauing ( 70.5 % 70.5\% at both c = 30 c{=}30 and c = 50 c{=}50 ). Thought-phase latency increases only ∼ 30 {\sim}30 ms ( 111 111 to 141 141 ms) as c c varies by 50 × 50\times , since the latents are processed in parallel within the fixed R = 6 {R}{=}6 sequential steps. This contrasts with autoregressive CoT, whose latency grows linearly in the number of thought tokens.

##### Looped iterations R {R} .

We train separate models for R ∈ { 2 , … , 6 } {R}\in\{2,\dots,6\} , fixing K = 6 K{=}6 and c = 25 c{=}25 so that the latent prefix has K ​ c = 150 Kc=150 positions and only the depth of sequential refinement changes ( Table 6 ). Larger R {R} gives the model more refinement iterations before the readout. As a result, accuracy rises steeply with R {R} (from 14.6 % 14.6\% at R = 2 {R}{=}2 to 70.0 % 70.0\% at R = 6 {R}{=}6 ), with gains continuing through R = 5 {R}{=}5 ( 68.1 % 68.1\% ) before nearly saturating at R = 6 {R}{=}6 .

##### Inference-time R {R} .

Reusing the model trained at R = 6 {R}{=}6 , we vary R ∈ { 1 , … , 7 } {R}\in\{1,\dots,7\} at inference with the latent prefix fixed at 150 150 positions ( Table 6 ). Accuracy climbs with R {R} ( 22.7 % 22.7\% at R = 1 {R}{=}1 to 70.0 % 70.0\% at R = 6 {R}{=}6 ), peaks at the trained R = 6 {R}{=}6 , and dips slightly at R = 7 {R}{=}7 ( 69.3 % 69.3\% ). Running fewer iterations leaves the model less opportunity to refine, while running more than at training time brings no further gain.

## 5 Latent Representation Analysis

We analyze the learned latent representations along three axes. Together, these analyses show that LOTUS’s accuracy gains are accompanied by structured latent computation: the post-loop latents carry readable CoT token signal, assign nontrivial mass to unseen valid alternatives, and rely on both gold CoT supervision and answer supervision to form coherent reasoning states. Section 5.1 measures the likelihood assigned to the gold CoT under different readouts, Section 5.2 tests whether the latents place mass on unseen but valid reasoning chains, and Section 5.3 ablates the roles of gold CoT and answer supervision in shaping the representation. Extended block-to-step decode examples are deferred to Section F.1 .

### 5.1 Likelihood of the Gold Chain under Different Readout Options

Recall that 𝒉 ( R ) {{{\bm{h}}}^{({R})}} denotes the post-loop latents and 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} the aux decoder’s output latents for t ∈ { 1 , … , R } t\in\{1,\dots,{R}\} . We evaluate four readouts in Table 9 : (i) LOTUS, reading 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through the base LM head; (ii) LOTUS-aux, reading 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ; (iii) LOTUS-aux, reading 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} under teacher forcing (TF) that matches training (each block’s c {c} latents paired with that block’s gold step tokens); and (iv) LOTUS-aux, reading 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} but under free-running (FR), 6 6 6 LOTUS-aux (FR) gives a fairer reference to LOTUS than LOTUS-aux (TF): at inference the gold CoT is unavailable, so LOTUS-aux must feed its decoder its own generated tokens, exactly the FR setting. The TF row instead feeds the gold prefix the model would not have at test time. feeding the aux decoder’s own generated token in place of the gold tokens. 7 7 7 Here we report greedy decoding results. Replacing them with temperature- T T sampling ( T ∈ { 0.5 , 0.7 , 1.0 , 1.5 } T\in\{0.5,0.7,1.0,1.5\} ) moves FR NLL by ≤ 0.05 \leq 0.05 and top- 1 1 by ≤ 1 \leq 1 pp.

##### Metrics.

We report the negative log-likelihood (NLL) of the gold CoT token sequence T {T} at the latent positions. 8 8 8 Note that the 𝒉 ( R ) {{{\bm{h}}}^{({R})}} readout scores the gold CoT under the PCL ( Section 3.3.2 ) factorization and 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} under an autoregressive one, so NLL is only directly comparable under the same factorization and only for reference across factorizations. Mathematically, PCL NLL has a higher lower bound than the autoregressive NLL (details in Appendix E ). We also report average P ⁡ ( gold ∈ top-1 ) {P(\text{gold}\in\text{top-1})} and P ⁡ ( gold ∈ top-5 ) {P(\text{gold}\in\text{top-5})} against the full 128 ​ K 128\text{K} -vocabulary softmax. All metrics are averaged over the 1,319 1{,}319 GSM8K test questions.

##### LOTUS’s post-loop latent 𝒉 ( R ) {{{\bm{h}}}^{({R})}} contains the gold CoT via a direct readout.

Reading LOTUS’s post-loop latent 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} assigns the gold CoT a low 3.07 3.07 NLL and 70.9 % 70.9\% top- 1 1 accuracy ( Table 9 ), edging out LOTUS-aux (FR) at 64.5 % 64.5\% . The teacher-forced LOTUS-aux reader does better still ( 1.56 1.56 NLL, 84.3 % 84.3\% top- 1 1 ), but with the caveat that it uses the gold prefix as input, so we read it as an upper bound rather than a direct comparison. Still, the results confirm that the latents carry enough information to reconstruct the gold CoT chain.

##### LOTUS-aux has CoT signal in both 𝒉 ( R ) {{{\bm{h}}}^{({R})}} and 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} .

Surprisingly, even though 𝒉 ( R ) {{{\bm{h}}}^{({R})}} is never directly supervised to predict the gold CoT through g head {g_{\mathrm{head}}} in LOTUS-aux, we can still read 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} with a 25.8 % 25.8\% top- 5 5 probability. Greedy decoding also surfaces some meaningful gold tokens, indicating that the CoT content is carried by the latents themselves. In fact, despite being supervised through different heads, LOTUS and LOTUS-aux have a similar effect of making the post-loop latents 𝒉 ( R ) {{{\bm{h}}}^{({R})}} put mass on the CoT steps, with the caveat that LOTUS-aux readouts are less formatted (which is expected since it is not supervised directly). We provide side-by-side 𝒉 ( R ) {{{\bm{h}}}^{({R})}} readout examples for both models in Appendix F .

### 5.2 Mass on Unseen Valid Chains

Beyond the gold chain in the test set, we test whether LOTUS and LOTUS-aux place mass on unseen valid reasoning chains rather than only memorizing the seen trace in the training set. We use the same four readout configurations as in Section 5.1 .

##### Setup.

Given each question, we compare its ground-truth reasoning chain against an unseen-but-valid alternative. Let G G and U U denote the sets of intermediate numbers appearing uniquely in the ground-truth and the unseen reasoning chain, respectively. A random control set serves as a baseline. We report each set’s average per-token negative log-likelihood (NLL) at the position where it surfaces most strongly (lower means more confident). We also report P ⁡ ( U ∈ top- ​ k ) P(U\in\text{top-}k) , the fraction of U U numbers that surface within the top- k k readout at any latent position. See Section F.2 for the full construction and metric details.

##### Findings.

Across all four readouts the ordering G ≪ U ≪ G\!\ll\!U\!\ll\! Random holds ( Table 10 , lower NLL = = surfaces more): each reader assigns much lower NLL to a ground-truth-only number than to an unseen-but-valid one, which in turn sits far below the random control. Crucially, the U U signal is concentrated rather than diffuse: P ⁡ ( U ∈ top- ​ k ) P(U\in\text{top-}k) columns confirm the unseen-valid intermediates genuinely surface in the top- k k (e.g., for LOTUS through the base LM head, 15.3 % 15.3\% are top- 1 1 and 64.0 % 64.0\% fall within top- 5 5 , despite never being trained on). Section F.3 and Figure 4 give a path-level example and visual grid: two valid chains reach the same answer ( 108 108 ) through disjoint intermediates, and the unseen-path numbers ( 24 24 , 18 18 ) surface in the post-loop readout despite never appearing in the trained chain, question, or answer.

### 5.3 Disentangling the Two Losses in LOTUS

We now ask what each of LOTUS’s two supervision losses contributes to the latent representation observed above. The theory in Section 3.3.2 ascribes complementary roles to ℒ step {\mathcal{L}_{\mathrm{step}}} and ℒ ans {\mathcal{L}_{\mathrm{ans}}} . We compare three models that differ in supervision: LOTUS (both losses), ℒ step {\mathcal{L}_{\mathrm{step}}} only, and ℒ ans {\mathcal{L}_{\mathrm{ans}}} only.

##### Gold-chain NLL.

Reading the 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through the base LM head f head {f_{\mathrm{head}}} , Table 11 reports NLL, P ⁡ ( gold ∈ top-1 ) P(\text{gold}\in\text{top-1}) , and P ⁡ ( gold ∈ top-5 ) P(\text{gold}\in\text{top-5}) on the GSM8K test set, using the same protocol as Table 9 .

##### ℒ step {\mathcal{L}_{\mathrm{step}}} and ℒ ans {\mathcal{L}_{\mathrm{ans}}} play complementary roles.

LOTUS beats both ablations ( Table 11 ). ℒ step {\mathcal{L}_{\mathrm{step}}} -only, lacking the cross-block coupling and the answer grounding that ℒ ans {\mathcal{L}_{\mathrm{ans}}} supplies, is worst on every column. ℒ ans {\mathcal{L}_{\mathrm{ans}}} -only, without directly tying 𝒉 ( R ) {{{\bm{h}}}^{({R})}} to the gold CoT, lands nearer the gold tokens (NLL 5.97 5.97 , top- 5 5 26.5 % 26.5\% ) yet has a top- 1 1 rate similar to ℒ step {\mathcal{L}_{\mathrm{step}}} -only and stays far short of LOTUS (NLL 3.07 3.07 , top- 5 5 85.8 % 85.8\% ). The answer loss alone settles 𝒉 ( R ) {{{\bm{h}}}^{({R})}} into an answer-consistent neighborhood without aligning to the gold chain, matching the complementary roles posited in Section 3.3.2 .

## 6 Conclusion

We introduce LOTUS, showing that latent reasoning can approach the performance of explicit CoT by supervising a looped padded Transformer in parallel against the gold CoT tokens under the simple cross-entropy objective. On Llama-3.2-3B-Instruct , LOTUS bridges the in-domain gap to explicit CoT on GSM8K, surpasses CoT on the out-of-domain average, and cuts thought-phase latency by 2.5 × 2.5\times . Ablations show the looped backbone, parallel gold CoT supervision, and sufficient block width and loop depth are each necessary. The latent representation analysis further shows the latents are transparent: the gold CoT is recoverable from them by a direct readout, they place graded probability on unseen but valid reasoning chains rather than a single memorized trace, and the step and answer losses contribute complementary structure.

##### Limitations.

We follow prior latent reasoning work ( Hao et al., 2025 ; Shen et al., 2025 ; Wei et al., 2025 ) and evaluate on math benchmarks. Whether the recipe transfers to other domains remains an open direction for future work. The block budget K {K} , per-block width c {c} , and loop depth R {R} are fixed hyperparameters that must be set to cover the expected step count, so chains longer than K {K} steps fall back to autoregressive completion of the tail. Making K {K} , c {c} , and R {R} adaptive is a natural and promising direction for future work.

## Acknowledgments

Anej Svete is supported by the ETH Zürich AI Center doctoral fellowship.

## Appendix A Related Work

We refer the reader to Chen et al. [2025a] , Yu et al. [2026] , Zhu et al. [2025b] , Li et al. [2025] , Feng et al. [2025] , and Maile and Sacramento [April 27, 2026] for broad surveys of the latent reasoning and looped transformers landscapes. Below, we discuss the work most directly adjacent to ours.

##### Looped and recurrent-depth transformers.

Building on Universal Transformers [ Dehghani et al., 2019 ] , a line of work has established the theoretical foundations of looped architectures: Giannou et al. [2023] show Turing completeness via program simulation, placing looped transformers alongside CoT in expressivity [ Merrill and Sabharwal, 2024 , Li et al., 2024b , Nowak et al., 2024 ] . Saunshi et al. [2025] , Merrill and Sabharwal [2025a] , Merrill and Sabharwal [2025b] , Jerad et al. [2026] study the efficient reasoning abilities of looping, establishing their ability to solve problems with fewer model evaluations than CoT. Saunshi et al. [2025] show that a shallow Transformer looped multiple times nearly matches a non-looped Transformer with the equivalent total depth on arithmetic and reasoning tasks, with looping implicitly simulating hidden reasoning steps. Xu and Sato [2025b] , Xu and Sato [2025a] compare looped Transformers to CoT, and Svete and Sabharwal [2026] draw connections to masked diffusion models. The latter connection also inspires approaches to improve looped transformers with time modulation [ Jeddi et al., 2026 , Chen, 2026 ] and efficient sampling [ Geiping et al., 2025b ] . An active line of work scales looped models to large-scale pretraining: Geiping et al. [2025a] train a 3.5B recurrent-depth Transformer whose loop depth can be increased at test time, Zhu et al. [2025c] pretrain LoopLM with token-level recurrence and learned depth allocation, and Zeng et al. [2026b] , Zeng et al. [2026a] pretrain models that loop by superpositioning symbols in continuous space based on the logits produced after each iteration. LoopLM shows that looped computation can scale in pretraining, but its per-token loop at inference time does not by itself create the parallel padded latent workspace that LOTUS uses. Complementary work addresses different ways of implementing adaptive compute with looped architectures: Depth can be modulated per-symbol via gating [ Ng and Wang, 2024 , Song et al., 2026 , Moosa et al., 2026 ] , mixture-of-depths routing [ Raposo et al., 2024 ] , or RL-trained stopping criteria [ Ning et al., 2025 ] . Depth-recurrent attention mixtures augment recurrence with attention mechanisms that can attend to latent representations of past iterations [ Fu et al., 2025 , Knupp et al., 2026 ] . Parameter efficiency and flexibility are addressed by per-iteration LoRA adapters [ Bae et al., 2025a ] , symbol-level recursion routing [ Bae et al., 2025b ] , inner thinking mechanisms [ Chen et al., 2025b ] , and Mixture-of-Experts compatibility [ Csordás et al., 2024 ] . Further work explores retrofitting recurrence into pretrained models [ McLeish et al., 2025 , Koishekenov et al., 2025 ] , compositional and length generalization [ Fan et al., 2025 , Kohli et al., 2026 ] , reinforcement pretraining [ Tang et al., 2026 ] , and whether pretrained models use their depth efficiently [ Csordás et al., 2025 ] .

##### Diffusion language models.

Diffusion language models are another route to parallel, non-autoregressive reasoning. Discrete-token variants face a parallel-decoding trap , where tokens denoised in parallel are conditionally independent, so recovering quality forces a partial return to sequential decoding [ Liu et al., 2025a ] . Notably, Svete and Sabharwal [2026] show masked diffusion models are equivalent to padded looped Transformers, motivating supervising such a backbone directly, as we do. Continuous diffusion variants such as LaDiR [ Kang et al., 2026 ] instead refine latent thoughts with a VAE and a separate diffusion model over many denoising steps for diversity and adaptive compute, while showing limited efficiency gain when reaching CoT-level performance.

##### Pause and padding symbols.

A parallel strand of work studies the affordances of discrete padding symbols. Goyal et al. [2024] and Pfau et al. [2024] show that they improve performance on natural language and formal tasks; Wang et al. [2024] use discrete symbols in the continuous representation space as planning signals; and Chu et al. [2026] introduce span-level pause symbols for efficient and interpretable latent reasoning. These discrete-symbol approaches motivate our use of continuous latent vectors as a richer computation medium, while our looped architecture lets each iteration refine the same latent rather than extending the sequence.

##### Compressing and replacing discrete CoT.

Latent CoT methods replace or compress explicit reasoning symbols with continuous representations. Cheng and Durme [2024] supervise hidden-state representations of salient reasoning steps, Tan et al. [2025] dynamically compress CoT chains into latent thoughts, Wang et al. [2026] autoregressively encode CoT steps into hidden states decoded back to ground-truth symbols (analogously to our approach), and Amos et al. [2026] interleave supervised latent states with generated text. MUX [ Suleymanzade et al., 2026b ] studies continuous reasoning with multiplexed vocabulary-space targets. It uses KL regression to multiplexed targets. Each target is a position-weighted superposition over aligned reasoning subwords, so several subwords are mixed into one sequential latent. In contrast, LOTUS keeps step and token positions explicit in a parallel padded workspace and applies per-position cross-entropy on gold CoT traces. Latent CoT also enables parallel exploration of reasoning paths [ Gozeten et al., 2026 , Liu et al., 2026 ] , and test-time compute can be scaled in latent space via policy optimization [ Ye et al., 2026 ] , instance-level policy gradient [ Li et al., 2026a ] , sampling conditioned on perturbed prompts [ Xu et al., 2025 ] , or parallel search [ Lian et al., 2025 , You et al., 2026 ] . Latent reasoning can be integrated with generation by mixing latent and text symbols [ Fu and Luo, 2026 , Deng et al., 2026 ] , updating the KV cache with an additional model [ Liu et al., 2025b ] , learning meta-symbols that guide generation [ Shah et al., 2025 ] , and semantically aligning padding symbols to spans of CoT symbols [ He et al., 2025 ] . Further work studies latent reasoning with variational autoencoder features [ Su et al., 2025 , Kang et al., 2026 ] and energy-based guidance [ Chen et al., 2025c ] . GRAM [ Baek et al., 2026 ] guides thinking with recursive latent-variable modeling through shared transitions. It uses stochastic high-level updates and learns a target-conditioned posterior and prior through amortized variational inference. GRAM samples distinct stochastic trajectories, whereas LOTUS represents them within a jointly refined continuous workspace. Interpretability studies probe whether latents actually contribute to reasoning [ Zhang et al., 2025 , Rizvi-Martel and Mosbach, 2026 ] , and empirical comparisons examine the effects of different supervision regimes [ Cui et al., 2026 , Suleymanzade et al., 2026a ] .

##### Speculative decoding.

Speculative decoding uses draft models to propose token continuations [ Leviathan et al., 2023 , Chen et al., 2023 ] . Related approaches include Lookahead [ Fu et al., 2024 ] with Jacobi-style updates, Medusa [ Cai et al., 2024 ] with auxiliary prediction heads, and EAGLE [ Li et al., 2024a ] with future hidden features. These methods parallelize proposals within an explicit token trace, whereas LOTUS parallelizes latent reasoning and decodes only the answer suffix.

## Appendix B Notation

Table 12 summarizes the notation used throughout the paper.

## Appendix C Closely Related Latent Reasoning Methods

We position LOTUS and its auxiliary-decoder variant LOTUS-aux against the most closely related latent reasoning methods.

##### Coconut [ Hao et al., 2025 ]

introduced a curriculum-based approach that progressively replaces explicit CoT steps with latent tokens conditioned autoregressively on the previous latent. Both LOTUS and LOTUS-aux depart from Coconut in two ways: (i) they use a padded latent prefix processed by a looped Transformer, so all K {K} steps are decoded in parallel, not autoregressively, giving the 𝒪 ⁡ ( R ) {{{{\mathcal{O}}}({R})}} -iteration thought-phase cost, and (ii) they add step-aligned supervision on the latent blocks, where Coconut supervises only the answer. Our ablation without latent supervision ( Table 4 ) is the Coconut-style regime within our looped architecture, and it underperforms both variants.

##### CODI [ Shen et al., 2025 ] and SIM-CoT [ Wei et al., 2025 ]

retain Coconut’s autoregressive latent decoding but add an auxiliary supervision target: CODI aligns the hidden state of a single designated token with the corresponding teacher hidden state from a CoT model, and SIM-CoT trains an auxiliary autoregressive decoder that aligns each latent token with the corresponding CoT token. Our LOTUS removes the auxiliary path entirely, supervising the model’s own post-loop hidden states through the answer LM head at block granularity. LOTUS-aux keeps an auxiliary autoregressive decoder, but produces all blocks in parallel through the looped backbone and supervises a full block per step rather than one latent per step.

##### Parallel Continuous CoT (PCCoT) [ Wu et al., 2025 ] and KaVa [ Kuzina et al., 2026 ]

form a single Jacobi-iteration family: PCCoT introduces the backbone and KaVa builds on it. They both refine a fixed budget of 24 24 latent tokens over 3 3 Jacobi iterations and differ only in supervision. Like LOTUS, they process latent symbols in parallel and refine them over passes, but differ from our methods in three ways: (i) Architecture : PCCoT and KaVa use right-shifted Jacobi iterations to simulate sequential continuous CoT in parallel [ Hao et al., 2025 ] , so their latents are continuous thought vectors with no fixed alignment to individual gold CoT steps. LOTUS instead pins block i {i} to fixed positions across all loops, so it always maps to gold CoT step i {i} and is naturally supervised against that step’s tokens; (ii) Supervision : PCCoT uses CODI [ Shen et al., 2025 ] , and KaVa supervises indirectly by distilling a compressed teacher KV-cache, whereas both our variants supervise each latent block against its exact gold CoT step; and (iii) Depth : PCCoT reports accuracy peaking at about 3 3 Jacobi iterations and degrading beyond that, which is why KaVa inherits the 3 3 -iteration setting, whereas LOTUS improves monotonically with loop depth up to 6 ( Table 6 ).

##### Comparison with latent methods at other compute budgets.

The main results ( Table 1 , Section 4.2 ) compare LOTUS against latent methods at the same sequential step budget (CODI and CODI + SIM-CoT, both R = 6 {R}{=}6 ). Here we provide a more comprehensive accuracy comparison in Table 13 against latent methods whose sequential compute budget instead differs from LOTUS’s R = 6 {R}{=}6 sequential steps. Coconut and Coconut + SIM-CoT decode latents autoregressively under the curriculum of Hao et al. [2025] ( 10 10 autoregressive latent tokens), while PCCoT and KaVa decode in parallel over a shared Jacobi-iteration backbone ( 24 24 latent tokens in total, refined over 3 3 iterations). We take accuracies from Wei et al. [2025] , Kuzina et al. [2026] , and Wu et al. [2025] for each method. LOTUS attains higher in-domain GSM8K accuracy than KaVa at both shared scales ( 57.3 57.3 vs. 56.5 56.5 at 1 1 B, 70.0 70.0 vs. 65.7 65.7 at 3 3 B), staying within 1.5 1.5 points of explicit CoT while PCCoT and KaVa fall further behind as the backbone grows. The missing entries follow what each source reports: Wei et al. [2025] evaluate Coconut only up to Llama-3.2-1B , attributing Coconut’s failure at larger scale to a latent-instability issue in which the latent representations become homogeneous and lose semantic diversity, causing training to collapse as the latent budget grows. Kuzina et al. [2026] do not run KaVa on GPT-2 .

##### Natural-language CoT stress-test.

For the natural-language CoT stress test in Section 4.2 , we use the Llama-3.2-3B-Instruct backbone. The explicit CoT run reaches 68.41 % 68.41\% GSM8K accuracy with 963.6 963.6 ms thought latency. The looped run reaches 68.13 % 68.13\% accuracy with 140.8 140.8 ms thought latency. The latent baselines trail well behind on GSM8K (PCCoT 47.6 47.6 , CODI 55.9 55.9 , KaVa 60.0 60.0 ). Out of domain, LOTUS leads all methods on SVAMP ( 73.40 73.40 , above explicit CoT’s 71.93 71.93 ) and stays close to explicit CoT on GSM-Hard ( 16.27 16.27 vs. 18.27 18.27 ), ahead of every latent baseline ( Table 14 ). We exclude SIM-CoT from this comparison because Suleymanzade et al. [2026b] report weak SIM-CoT accuracy on the natural-language subset.

## Appendix D Experimental Details

This appendix includes the training and evaluation details in Section 4.1 .

##### Latent prefix budget.

Each of the K {K} blocks holds c {c} latent positions, so the prefix spans K ​ c {K}{c} positions: c = 25 {c}{=}25 ( 150 150 positions) for the Llama backbones and c = 13 {c}{=}13 ( 78 78 positions) for GPT-2 . We use K = 6 {K}{=}6 , which covers the reference CoT step count for 99 % 99\% of GSM8k-Aug examples. Longer problems fall back to autoregressive completion of the unsupervised tail.

##### Training curriculum.

We follow the staged curriculum of Hao et al. [2025] , converting one additional CoT step into a latent block every E stage E_{\text{stage}} epochs. At epoch e e , we use K e = min ⁡ ( ⌊ e / E stage ⌋ , K ) , {K}_{e}\;=\;\min\!\left(\lfloor e/E_{\text{stage}}\rfloor,\;{K}\right), latent blocks: up to the first K e {K}_{e} CoT steps become latent blocks, and the rest remain visible tokens. Thus e = 0 e{=}0 is standard CoT fine-tuning and the curriculum saturates at K e = K {K}_{e}{=}{K} . We set the loop count at epoch e e to R e = r ​ K e {R}_{e}=r\,{K}_{e} , where r r is the loops-per-block ratio. Since each pass refines all blocks in parallel, K e {K}_{e} controls the latent workspace width and R e {R}_{e} controls the depth. Unless stated otherwise, E stage = 1 E_{\text{stage}}{=}1 and r = 1 r{=}1 , so the saturated model uses R = K = 6 {R}{=}{K}{=}6 . The loop-depth ablation ( Section 4.3.2 ) fixes K = 6 {K}{=}6 and varies r r to decouple depth from width.

##### Optimization details.

We initialize from each backbone’s standard explicit CoT checkpoint, and fine-tune all parameters with AdamW and gradient-norm clipping at 1.0 1.0 , in bf16 for the Llama backbones and fp32 for GPT-2 . The learning rate is constant: × 10 − 4 1\!\times\!10^{-4} for GPT-2 and Llama-3.2-1B-Instruct , and × 10 − 5 5\!\times\!10^{-5} for Llama-3.2-3B-Instruct . The training objective ( Equation 6 ) weights the step loss at λ step = 0.05 {\lambda_{\mathrm{step}}}{=}0.05 (for both LOTUS and LOTUS-aux). Adding CODI weights its loss at 1.0 1.0 . We train for 30 30 epochs with an overall batch size of 128 128 , and select the best checkpoint by GSM8k validation accuracy, though the final epoch checkpoints perform similarly.

##### Natural-language training setup.

For the natural-language stress-test on LOTUS, we keep the same 3B looped backbone but switch to the natural-language GSM8k-Aug training set [ Wu et al., 2025 ] , increase latent width to c = 50 c{=}50 and set λ step {\lambda_{\mathrm{step}}} to 0.033 0.033 . We initialize from the CoT checkpoint trained on the natural-language GSM8k-Aug. Other settings are identical to those used for the original GSM8k-Aug training.

##### LOTUS-aux auxiliary decoder.

For LOTUS-aux, the auxiliary decoder is a full-size deep copy of the base model, initialized from its weights but trained together with the main model. This full-size choice matches SIM-CoT [ Wei et al., 2025 ] , whose auxiliary decoder is architecturally identical to the base model. We also tried lightweight auxiliary decoders but found them unstable in training.

##### Evaluation.

Inference uses greedy decoding with batch size 1 1 . This isolates per-example latency from batching effects and calculates the average latency per sample accurately. Latency is measured on a single NVIDIA H100 and decomposed into query, thought, and answer phases ( Table 2 ).

##### Batched KV-cache sharing.

For training, we left-pad shorter examples so the ⟨ lat ⟩ {\langle\texttt{lat}\rangle} positions align across the batch. This lets us compute each example’s question-prefix KV cache once and reuse it across all R {R} looped iterations.

## Appendix E PCL Versus Autoregressive NLL Lower Bounds

Here we provide the explanation mentioned in Section 5.1 . PCL ( Section 3.3.2 ) scores each gold token in parallel, − ∑ i , j log p 𝜽 ( T i , j ∣ 𝒉 ( R ) ) -\sum_{i,j}\log{{p_{{{{\bm{\theta}}}}}}}({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}}) , whereas the autoregressive factorization also conditions on the preceding gold tokens, − ∑ i , j log p 𝜽 ( T i , j ∣ 𝒉 ( R ) , T < ( i , j ) ) -\sum_{i,j}\log{{p_{{{{\bm{\theta}}}}}}}({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}},{T}_{<({i},j)}) , where T < ( i , j ) {T}_{<({i},j)} denotes all gold tokens before position ( i , j ) ({i},j) in reading order. The lowest expected NLL each can reach is the corresponding conditional entropy, and since conditioning never increases entropy, H ⁡ ( T i , j ∣ 𝒉 ( R ) ) ≥ H ⁡ ( T i , j ∣ 𝒉 ( R ) , T < ( i , j ) ) , H\!\big({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}}\big)\;\geq\;H\!\big({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}},{T}_{<({i},j)}\big), (11) with equality only when the gold tokens are conditionally independent given the latents. So PCL has a strictly higher NLL floor whenever cross-token dependence exists.

## Appendix F Details for Latent Representation Analysis ( Section 5 )

### F.1 Greedy Readout Examples

Here we give examples of two correctly answered and two incorrectly answered GSM8K test problems. We show three readouts of the post-loop latents: (A) the per-block top- 1 1 token sequence reading 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through LOTUS’s base LM head f head {f_{\mathrm{head}}} ; (B) the same readout applied to LOTUS-aux’s post-loop latents 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through its auxiliary head g head {g_{\mathrm{head}}} ; and (C) the autoregressive CoT produced by LOTUS-aux’s trained auxiliary decoder from 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} , conditioned on the same latents and decoded greedily from the start-of-CoT token. (A) and (B) show the top- 1 1 token stream per block verbatim (newlines rendered as \n ), truncated at 60 60 characters per block with ... marking the cut. (C) shows the raw aux-decoder output. We additionally report the final answer the model generates autoregressively from 𝒉 ( R ) {{{\bm{h}}}^{({R})}} after all R = 6 {R}{=}6 loops. In each (A) / (B) readout we list the K = 6 {K}{=}6 latent blocks top to bottom, labeled B0 – B5 .

For readability, blue marks the readable gold content: well-formed <<…>> CoT steps and exact gold chain numbers (step operands and intermediate results), including a number stuttered back-to-back such as 540540 for 540 540 . Wrong digit runs (e.g. 120120 or 3636 ) are left unmarked, and within each readout line we highlight only the first surfacing of each gold value.

##### Example 1 — Janet’s ducks.

Question: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market? Reference: <<16-3-4=9>> , <<9*2=18>> . Gold answer 18 18 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<3+4=7>> \n00>>\n7>>\n>>\n>>\n 16 16-- 9 999999<< B1: <<16-7=9>> \n00>>\n>>\n>>\n>>\n>>\n*2== 18 18>>\n>>\n>>\n1818<< B2: <<9*2=18>> \n00>>\n>>\n>>\n18>>\n>>\n>>\n= 1818>>\n18>>\n>>\n... B3: <<18*2=18>> \n>>\n>>\n>>\n>>\n>>\n>>\n>>\n>>\n= 9 9>>\n>>\n18>>... B4: ###134* 7 =>>\n 18 >>\n>>\n>>\n>>\n>>\n>>\n>>\n2=2 .>>\n181818>>... B5: >>\n 18 =21818>>\n>>\n>>\n===.###2== ,>>\n1818222 (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: <<’+) 7 ))<< 16 16--7= 1313\n\n###22*22 B1: =2189)#########2222= 18 18#########22* 9 = B2: 18 18 per#########2922 =1818 fresh###### 9 9* no 16 = 18 B3: 18 fresh<|eot_id|>######### 9 * no no nothing=1818 Clean<|eot_... B4: 18 ly<|eot_id|>###### no nothing no nothing 1818<|eot_id|>###... B5: ############ nothing nothing nothing= 18 1818############### n... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<3+4=7>> \n <<16-7=9>> \n <<9*2=18>> \n <<18=18>> \n <<9*2=18>> \n <<18+18=18>> \n <<18*2=18>> \n <<18=18>> Final answer (autoregressive, after 6 6 loops). LOTUS: ### 18 . LOTUS-aux: ### 18 . Gold: 18 18 . Both correct. Reading the chains. (A): the gold intermediates 𝟗 \mathbf{9} and 𝟏𝟖 \mathbf{18} surface from B1 onward and stabilize by B5. (B): the same numerics ( 9 , 18 , 16 , 7 9,18,16,7 ) are present but interleaved with <|eot_id|> runs and filler. (C): the aux decoder produces the gold first two steps <<3+4=7>> , <<16-7=9>> , <<9*2=18>> and then loops on 18 18 .

##### Example 2 — James’s sprints.

Question: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week? Reference: <<3*3=9>> , <<9*60=540>> . Gold answer 540 540 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<3*60=180>> \n00>>\n180180>>\n>>\n3== 540 540>>\n>>\n>>\n>>\n>... B1: <<180*3=540>> \n00>>\n>>\n>>\n540>>\n### 540>>\n>>\n >>\n... B2: <<540*540=540>> \n00>>\n>>\n>>\n>>\n>>\n>>\n### 540>>\n>>\n ... B3: <<540=180=540>> \n540>>\n>>\n>>\n>>\n>>\n>>\n### 540>>\n>>\n... B4: ### 540 = =>>\n###>>\n>>\n>>\n 2 ###### 540540>>\n>>\n ### B5: ### == >>\n######### >>\n### >>\n>>\n >>\n (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 3 3*3= 180 180180*###9993== 540 540)###### 9 B1: 9 * 60 = 540 540))#########9* 3 nothing = 540540)############ B2: /* nothing= 540 540)#########540+### nothing 540############... B3: nothing is 540 540###############540,### also 540540540#####... B4: is 540 540540############540,### also 540540#################... B5: 540 540###############540### definitely 540540540############... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<3*60=180>> \n <<180*3=540>> \n <<540=540>> \n <<540/60=9>> \n <<540=540>> \n <<540/10=540>> \n <<540>> Final answer (autoregressive, after 6 6 loops). LOTUS: ### 540 . LOTUS-aux: ### 540 . Gold: 540 540 . Both correct. Reading the chains. (A): the gold step <<3*60=180>> appears in B0 already (a valid alternative decomposition: meters per day first), and 𝟓𝟒𝟎 \mathbf{540} stabilizes from B1. (B): 180,540 , 9 , 3 180,540,9,3 are all present but with heavy ### runs around them. (C): the aux decoder reproduces 𝟏𝟖𝟎 → 𝟓𝟒𝟎 \mathbf{180}\to\mathbf{540} cleanly, then loops.

##### Example 3 — Carla’s download (failure).

Question: Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file? Reference: <<200*40*.01=80>> , <<80/2=40>> , <<200/2=100>> , <<40+100+20=160>> . Gold answer 160 160 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<200/2=6=120>> \n6>>\n>>\n>>\n>>\n>>\n2>>\n505050>>\n605050<... B1: <<40*.2=40>> \n120>>\n666>>\n>>\n>>\n>>\n>>\n60>>\n 80 80>>\n>>... B2: <<60+40=120>> \n8>>\n>>\n>>\n>>\n>>\n>>\n>>\n6050>>\n>>\n>>\n... B3: <<140+20=120>> \n8>>\n33>>\n>>\n>>\n>>\n>>\n2020=120>>\n>>\n>... B4: <<140+20=120>> \n120>>\n33>>\n>>\n>>\n>>\n###+60+/1>>\n>>\n>>... B5: 160 160+ 20 =120>>\n180>>\n>>\n>>\n>>\n180>>\n###+600+100120>>\... (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 100200/) 100 100))<<100100** 40 percentage,404040,<<100-+ B1: 20 20= 80 + and### 100 100+20++ 120120)###100100+220+ B2: 120120 plus### 100 100*10020 140 plus###12080+2100= 120240)\n... B3: ###120120+ 20 20+120140140######120+202020 140140)\n######120+ B4: +2= 140140)\n)\n######120120+ 20 2020 140100)\n Ris###### B5: ###120+ 20 20= 100 100 Ris#########120+20= 100100 Ris##########... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<100+20=120>> \n <<120+20=140>> \n <<100+20=120>> \n### 140 Final answer (autoregressive, after 6 6 loops). LOTUS: ### 180 . LOTUS-aux: ### 120 . Gold: 160 160 . Reading the chains. The latents recover most of the right intermediates ( 80,100 , 𝟒𝟎 , 𝟐𝟎 \mathbf{80,100,40,20} ) but fail to combine them: (A) flickers between 120,140,160,180 120,140,160,180 in the late blocks; (B) commits to 120 + 20 120+20 chains; (C) outputs <<100+20=120>>, <<120+20=140>>, ... and emits 140 140 . Both methods land on plausible-but-wrong totals.

##### Example 4 — Melanie’s vacuums (failure).

Question: Melanie is a door-to-door saleswoman. She sold a third of her vacuum cleaners at the green house, 2 more to the red house, and half of what was left at the orange house. If Melanie has 5 vacuum cleaners left, how many did she start with? Reference: <<5*2=10>> , <<10+2=12>> . Gold answer 18 18 . One extra step is omitted from the reference; the true chain is → → → 18 5\!\to\!10\!\to\!12\!\to\!18 ( 12 × 3 / 2 = 18 12\times 3/2=18 ).

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<5*2=10>> \n33>>\n 12 >>\n12>>\n22=1312>>\n1212121313<< B1: <<10+2=17>> \n 5 >>\n5>>\n17>>\n>>\n1//1181818.55<< B2: <<9/(1/2/5=20.5>> \n20>>\n15.18>>\n>>\n>>\n>>\n5>>\n<< B3: <<15*1=30>> \n15>>\n 5 >>\n5>>\n20>>\n5=151818>>\n>>\n55<< B4: <<3=15>> \n30>>\n15>>\n 5 >>\n5>>\n>>\n>>\n55515>>\n15>>\n5>>\n... B5: <<15+5=15>> \n15>>\n5>>\n>>\n>>\n17>>\n55515>>\n 18 >>\n18>>\n>... (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 105* half 10 10)+101010+2)= 12 12)<<1212*3 B1: 3)1512..<< 12 1212*3 third = 3636.###121212*3) B2: 3636)###### 12 +33= 3636)######36+125= B3: 3636/#########36+ 12 ++ 4836 Egg#########36+12+= B4: 483636#########+ 5 == 4836############36+5 3636### B5: ############15+3 3636###############15###1 = 363615### (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<10+2=12>> \n <<10+2=12>> \n <<10+2=12>> \n <<12+2=14>> \n... Final answer (autoregressive, after 6 6 loops). LOTUS: ### 30 . LOTUS-aux: ### 36 . Gold: 18 18 . Reading the chains. (A): LOTUS’s latents flicker between 15 , 18 , 30 15,18,30 across blocks. The valid sub-chain → → → 𝟏𝟖 \mathbf{5\!\to\!10\!\to\!12\!\to\!18} surfaces in B5, but the final readout commits to 30 30 . (B): LOTUS-aux locks onto a wrong × 3 \times 3 branch (presumably “a third” read as multiply-by-three), with 36 36 saturating B4 to B5. (C): the aux decoder loops on <<10+2=12>> .

What these readouts are (and are not). The readouts (A), (B), (C) are just diagnostic projections of the post-loop continuous latents back into the discrete token space. They are not part of the inference pipeline. At inference, the final answer is generated autoregressively from the continuous post-loop latents (the “Final answer” line in each example), conditioning on the latent representation itself rather than on any discretized chain. The intermediate streams are intended to give intuition about what numbers the latents place mass on , not as a quality measure of the model’s reasoning: a noisy or repetitive (A)/(B) stream is compatible with a correct final answer, and a clean (C) chain is not required for the model to be correct.

### F.2 Details for Multi-Path Analysis ( Section 5.2 )

##### Datasets.

Note that GSM8K-Aug [ Deng et al., 2023 ] augments the original GSM8K [ Cobbe et al., 2021 ] training split (about 7.5 7.5 K problems) into the 385,620 385{,}620 training examples we use. GSM8K is thus a subset of GSM8K-Aug, and the multi-path bank below is built from the original GSM8K problems, which are also in the training set.

##### Path bank.

For each question we identify its trained path (its gold chain in the training data) and an unseen-but-valid alternative drawn from a bank of up to ten correct paths obtained by rejection-sampling Llama-3.1-8B [ Grattafiori et al., 2024 ] on the original GSM8K training questions ( 7,444 7{,}444 questions with 58,155 58{,}155 verified paths in total), with each alternative verified absent from training.

##### Candidate sets and filtering.

We partition intermediate numbers into three disjoint sets: G G (trained-path only), U U (unseen-path only), and a random control from other questions matched to U U in both cardinality and single/multi-token composition. Numbers shared by both paths or appearing in the question or answer are excluded, since they cannot indicate which chain a readout recovered. The random control is fixed and shared across all four readouts and verified disjoint from the trained path, unseen path, question, and answer numbers. Intersecting the bank with our training set and keeping questions with ≥ 2 \geq 2 disjoint intermediates per side yields 653 653 candidate questions. We then apply a token-clean filter, requiring every only- U U number to share no sub-token with any trained-path, question, or answer number, leaving the final 341 341 questions, spanning 871 871 G G , 974 974 U U , and 974 974 Random numbers. Random stays matched to U U in cardinality and composition under every criterion.

##### Metrics.

We score a target number by its best-position, per-token NLL. A number spans one or more consecutive sub-tokens, and at every latent position the readout gives a softmax distribution over the vocabulary. We slide the number’s sub-tokens across the latent positions. At each placement we average the sub-tokens’ NLLs at their aligned positions, and keep the lowest such per-token average over all placements (for a single-token number, simply the negative log of its highest readout probability across positions). Per-token normalization puts single and multi-token numbers on the same scale, and scoring only the most probable span avoids double counting overlaps. We average this over the numbers in sets G G , U U , and Random within each question, then over the 341 341 questions. The P ⁡ ( U ∈ top- ​ k ) P(U\in\text{top-}k) columns instead report a hit rate: the fraction of U U numbers recovered within the top- k k candidates (top- k k membership at any position for single-token numbers, and a consecutive top- k k match for multi-token ones), counting each number once however many positions it surfaces at, averaged within each question and then over the 341 341 questions.

### F.3 Multi-Path Readout Example

This example (referenced from Section 5.2 ) shows a question that admits two valid reasoning chains with the same answer 108 108 , read out by LOTUS ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through the base LM head f head {f_{\mathrm{head}}} , i.e. the first row of Table 10 ). Q: The bus driver drives an average of 2 hours each day, 5 days a week. From Monday to Wednesday he drove at an average speed of 12 kilometers per hour, and from Thursday to Friday at an average speed of 9 kilometers per hour. How many kilometers did the driver travel during these 5 days? Trained path G G : 3*2=6 -> 6*12=72 -> 2*2=4 -> 4*9=36 -> 72+36=108 . Unseen path U U : 2*12=24 -> 24+24+24=72 -> 2*9=18 -> 18+18=36 -> 72+36=108 . only G = { 3 , 4 , 6 } \text{only}_{G}=\{3,4,6\} , only U = { 24 , 18 } \text{only}_{U}=\{24,18\} (question and final answer excluded). Figure 4 visualizes the same example, aligning the trained path, the unseen valid path, and the latent readout positions where the only- U U intermediates surface. The only- U U intermediates 24 24 and 18 18 are the per-day distances in the alternative decomposition, which forms daily distances before summing rather than aggregating hours first. Although neither appears in the trained reference path, both surface in the post-loop latent readout through the base LM head: P ⁡ ( 24 ) = 0.77 P(\texttt{24}){=}0.77 (top- 1 1 , position 142 142 ) and P ⁡ ( 18 ) = 0.14 P(\texttt{18}){=}0.14 (top- 2 2 , position 116 116 ). Every number here is a single token, so no only- U U number shares a sub-token with any trained-path, question, or answer number, confirming the surfacing is genuine unseen-chain content rather than a digit-overlap artifact.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .