A Formal Comparison Between Chain of Thought and Latent Thought — part 1/9

Cleaned-text code points [0, 15974). Bibliography omitted here; source.md includes it.

# A Formal Comparison Between Chain of Thought and Latent Thought

###### Abstract

Chain of thought (CoT) elicits reasoning in large language models by explicitly generating intermediate tokens. In contrast, latent thought reasoning operates directly in the continuous latent space, enabling computation beyond discrete linguistic representations. While both approaches exploit iterative computation, their comparative capabilities remain underexplored. In this work, we present a formal analysis showing that latent thought admits more efficient parallel computation than inherently sequential CoT. In contrast, CoT enables approximate counting and sampling through stochastic decoding. These separations suggest the tasks for which depth-driven recursion is more suitable, thereby offering practical guidance for choosing between reasoning paradigms. Code is available at https://github.com/kevin671/cot-vs-loop .

###### Keywords:

## 1 Introduction

Transformer-based large language models (LLMs) ( Vaswani et al., 2017 ) have shown strong performance across diverse tasks and have recently been extended to complex reasoning. Rather than directly predicting final answers, generating intermediate reasoning steps, known as chain of thought (CoT) ( Wei et al., 2022 ) , enhances reasoning abilities. This naturally raises the question: why is CoT effective for complex tasks? Recent studies have approached this question by framing reasoning as a computational problem and analyzing its complexity ( Feng et al., 2023 ; Merrill and Sabharwal, 2024 ; Li et al., 2024 ; Nowak et al., 2024 ) , showing that CoT improves performance by increasing the model’s effective depth through iterative computation, thereby enabling the solution of problems that would otherwise be infeasible.

As an alternative to CoT, recent work has explored latent thought, which reasons directly in the hidden state space rather than in the discrete token space. This paradigm includes chain of continuous thought (Coconut) ( Hao et al., 2025 ) , which replaces next tokens with hidden state, and looped Transformer (looped TF) , in which output hidden states are iteratively fed back as inputs ( Dehghani et al., 2019 ) . Such iterative architectures have been shown to enhance expressivity: Coconut enables the simultaneous exploration of multiple traces ( Zhu et al., 2025a ; Gozeten et al., 2025 ) , while looped TF satisfies universality ( Giannou et al., 2023 ; Xu and Sato, 2025 ) and demonstrates competitive empirical performance ( Csordás et al., 2024 ; Bae et al., 2025 ; Zhu et al., 2025b ) .

These reasoning paradigms share the core idea of iteratively applying Transformers to enhance expressive power, which naturally leads to a fundamental question: What is the separation between chain of thought and latent thought? Recent studies characterize how expressivity scales with the number of iterations. Specifically, it has been shown that looped TF subsumes deterministic CoT ( Saunshi et al., 2025 ) , and exhibits a strict separation with only a logarithmic number of iterations ( Merrill and Sabharwal, 2025a ) . Nevertheless, fundamental questions remain open: Does the separation extend beyond the logarithmic regime? Is latent thought always more expressive than CoT?

### 1.1 Our Contributions

In this work, we address both questions by clarifying the respective strengths and limitations of the two reasoning paradigms through a formal complexity-theoretic analysis of their expressive power. Specifically, we show that latent thought gains efficiency from its parallelizability, yielding separations beyond the polylogarithmic regime. In contrast, CoT benefits from stochasticity, which enables approximate counting. An overview is given in Fig. 1 .

##### Latent thought enables parallel reasoning.

By formalizing decision problems as the evaluation of directed acyclic graphs (DAGs), we reveal the parallel computational capability of latent thought utilizing continuous hidden states. This analysis can be formalized by relating the class of decision problems realizable by the model to Boolean circuits. In particular, Boolean circuits composed of logic gates such as AND, OR, NOT, and Majority, with polylogarithmic depth log k ⁡ n \log^{k}n for k ∈ ℕ k\in\mathbb{N} and input size n n , define the class 𝖳𝖢 k \mathsf{TC}^{k} , a canonical model of parallel computation. Circuit complexity plays a central role in analyzing the computational power of Transformer models: fixed-depth Transformers without CoT are known to be upper-bounded by 𝖳𝖢 0 \mathsf{TC}^{0} ( Merrill and Sabharwal, 2023 ) , and subsequent studies analyze how the expressivity of CoT scales their computational power in terms of Boolean circuit complexity ( Li et al., 2024 ) . We show that latent thought with log k ⁡ n \log^{k}n iterations exactly captures the power of 𝖳𝖢 k \mathsf{TC}^{k} (Thm. 3.12 ); in contrast, CoT with log k ⁡ n \log^{k}n steps cannot realize the full power of 𝖳𝖢 k \mathsf{TC}^{k} (Thm. 3.13 ) due to its inherent sequentiality. This yields a strict separation in favor of latent thought in polylogarithmic regime (Thm. 3.15 ), showing its efficiency in terms of the required number of iterations.

##### CoT enables approximate counting.

A counting problem is a fundamental task in mathematics and computer science that determines the number of solutions satisfying a given set of constraints, including satisfying assignments of Boolean formulas, graph colorings, and partition functions ( Arora and Barak, 2009 ) . While exact counting for the complexity class # ​ 𝖯 \#\mathsf{P} is generally computationally intractable, approximation provides a feasible alternative. We show that CoT supports fully polynomial-time randomized approximation schemes ( 𝖥𝖯𝖱𝖠𝖲 \mathsf{FPRAS} ), yielding reliable estimates even in cases where exact counting via deterministic latent thought reasoning is intractable ( Lemma 4.3 ). Furthermore, leveraging classical results connecting approximate counting and sampling ( Jerrum et al., 1986 ) , we extend this separation to distribution modeling: there exist target distributions that CoT can approximately represent and sample from, but that remain inaccessible to latent thought ( Theorem 4.4 ). To the best of our knowledge, this constitutes the first formal separation in favor of CoT.

## 2 Background

### 2.1 Models of Computation

We define a class of reasoning paradigms in which a Transformer block ( Vaswani et al., 2017 ) is applied iteratively. Informally, CoT generates intermediate reasoning steps explicitly as tokens in an autoregressive manner. Formal definitions and illustrations are given in Appendix A .

###### Definition 2.1 (CoT, following Merrill and Sabharwal (2024) ) .

Let 𝒱 {\mathcal{V}} be a vocabulary, and let TF dec : 𝒱 ∗ → 𝒱 \mathrm{TF}_{\mathrm{dec}}:{\mathcal{V}}^{*}\to{\mathcal{V}} denote an decoder-only Transformer. Given an input sequence x = ( x 1 , … , x n ) ∈ 𝒱 n x=(x_{1},\dots,x_{n})\in{\mathcal{V}}^{n} , the outputs of CoT are defined by f cot 0 ​ ( x ) ≔ x , f cot k + 1 ​ ( x ) ≔ f cot k ​ ( x ) ⋅ TF dec ​ ( f cot k ​ ( x ) ) , f_{\mathrm{cot}}^{0}(x)\coloneq x,\quad f_{\mathrm{cot}}^{k+1}(x)\coloneq f_{\mathrm{cot}}^{k}(x)\cdot\mathrm{TF}_{\mathrm{dec}}(f_{\mathrm{cot}}^{k}(x)), where ⋅ \cdot denotes concatenation. We define the output to be the last tokens of f cot T ⁡ ( n ) ​ ( x ) ∈ 𝒱 n + T ⁡ ( n ) f_{\mathrm{cot}}^{T(n)}(x)\in{\mathcal{V}}^{\,n+T(n)} .

Coconut feeds the final hidden state as the embedding of the next token. Although the original Coconut model ( Hao et al., 2025 ) can generate both language tokens and hidden states, we focus exclusively on hidden state reasoning steps, in order to compare its representational power with that of CoT. Here, 𝔽 \mathbb{F} denotes the set of finite-precision floating-point numbers, and d ∈ ℕ d\in\mathbb{N} denotes the embedding dimension.

###### Definition 2.2 (Coconut) .

Let 𝒱 {\mathcal{V}} be a vocabulary and let TF dec Coconut : 𝒱 ∗ × ( 𝔽 d ) ∗ → 𝔽 d \mathrm{TF}^{\mathrm{Coconut}}_{\mathrm{dec}}:{\mathcal{V}}^{*}\times(\mathbb{F}^{d})^{*}\to\mathbb{F}^{d} be a decoder-only Transformer that maps a fixed token prefix together with a hidden state to the next hidden state. Given an input sequence x = ( x 1 , … , x n ) ∈ 𝒱 n x=(x_{1},\dots,x_{n})\in{\mathcal{V}}^{n} , we define the hidden states recursively by h 0 ≔ ( e ⁡ ( x i ) ) i = 1 n , h k + 1 ≔ TF dec Coconut ​ ( x , h k ) , h^{0}\coloneq\bigl(e(x_{i})\bigr)_{i=1}^{n},\quad h^{k+1}\coloneq\mathrm{TF}^{\mathrm{Coconut}}_{\mathrm{dec}}(x,h^{k}), where e : 𝒱 → 𝔽 d e:{\mathcal{V}}\to\mathbb{F}^{d} denotes an embedding. The output after T ⁡ ( n ) T(n) steps is obtained by decoding a suffix of the hidden state sequence ending at h T ⁡ ( n ) h^{T(n)} .

Looped TFs, by contrast, feed the entire model output back into the input without generating explicit tokens, recomputing all hidden states of the sequence at every iteration.

###### Definition 2.3 (Looped TF) .

Let TF : 𝔽 d × ∗ → 𝔽 d × ∗ \mathrm{TF}:\mathbb{F}^{d\times*}\to\mathbb{F}^{d\times*} denote a Transformer block. Given an input sequence x = ( x 1 , … , x n ) ∈ 𝒱 n x=(x_{1},\dots,x_{n})\in{\mathcal{V}}^{n} , the outputs are defined recursively by f loop 0 ​ ( x ) ≔ ( e ⁡ ( x i ) ) i = 1 n , f loop k + 1 ​ ( x ) ≔ TF ⁡ ( f loop k ​ ( x ) ) , f_{\mathrm{loop}}^{0}(x)\coloneq\bigl(e(x_{i})\bigr)_{i=1}^{n},\quad f_{\mathrm{loop}}^{k+1}(x)\coloneq\mathrm{TF}(f_{\mathrm{loop}}^{k}(x)), where e : 𝒱 → 𝔽 d e:{\mathcal{V}}\to\mathbb{F}^{d} denotes an embedding. The output after T ⁡ ( n ) T(n) loop iterations is the decoded last tokens of f loop T ⁡ ( n ) ​ ( x ) f_{\mathrm{loop}}^{T(n)}(x) .

Here, we assume that the input for looped TF may include sufficient padding so that its length is always at least as large as the output length, as in ( Merrill and Sabharwal, 2025b ) . The definitions of the models describe their core architectures; the specific details may vary depending on the tasks to which they are applied.

### 2.2 Related Work

To understand the expressive power of Transformers, previous work studies which classes of problems can be solved and with what computational efficiency. These questions can be naturally analyzed within the framework of computational complexity theory. Such studies on CoT and latent thought are summarized below and in Table 1 .

##### Computational power of chain of thought.

The expressivity of Transformers is limited by bounded depth ( Merrill and Sabharwal, 2023 ) , whereas CoT enhances their expressiveness by effectively increasing the number of sequential computational steps, enabling the solution of problems that would otherwise be intractable for fixed-depth architectures ( Feng et al., 2023 ) . Recent work has investigated how the expressivity of CoT scales with the number of reasoning steps, formalizing CoT for decision problems and computational complexity classes ( Merrill and Sabharwal, 2024 ; Li et al., 2024 ) . Beyond decision problems, CoT has been further formalized in a probabilistic setting for representing probability distributions over strings ( Nowak et al., 2024 ) .

##### Computational power of latent thought.

Latent thought is an alternative paradigm for increasing the number of computational steps without being constrained to the language space, with the potential to enhance model expressivity. In particular, Coconut has been shown to enable the simultaneous exploration of multiple candidate reasoning traces ( Zhu et al., 2025a ; Gozeten et al., 2025 ) . Looped TFs can simulate iterative algorithms ( Yang et al., 2024 ; de Luca and Fountoulakis, 2024 ) and, more generally, realize polynomial-time computations ( Giannou et al., 2023 ) . Recent results further demonstrate advantages over chain of thought reasoning: looped TFs can subsume the class of deterministic computations realizable by CoT using the same number of iterations ( Saunshi et al., 2025 ) , and exhibit a strict separation within the same logarithmic iterations ( Merrill and Sabharwal, 2025a ) . Concurrent work ( Zhu et al., 2025a ; Merrill and Sabharwal, 2025b ) has also identified connections between parallel reasoning and Coconut and diffusion models.

## 3 Latent Thought Enables Parallel Reasoning

We formalize the reasoning problem as a graph evaluation problem. Section 3.2 illustrates how each model approaches the same problem differently, providing intuitive insight into their contrasting capabilities. Building on these observations, Section 3.3 characterizes their expressive power and establishes a formal separation between them.

### 3.1 Problem Setting

Reasoning problems that can be solved by straight-line programs admit representations as directed acyclic graphs (DAGs) ( Aho and Ullman, 1972 ) , as illustrated in Fig. 2 (a).

###### Definition 3.1 (Computation graph) .

Let Σ \Sigma be a finite alphabet, and let ℱ \mathcal{F} denote a finite set of functions f : Σ ∗ → Σ f:\Sigma^{*}\to\Sigma . A computation graph is a directed acyclic graph G n = ( V n , E n ) G_{n}=(V_{n},E_{n}) that defines a function F G n : Σ n → Σ m ⁡ ( n ) F_{G_{n}}:\Sigma^{n}\to\Sigma^{m(n)} , where m ⁡ ( n ) m(n) denotes the output length. Here V n V_{n} denotes the set of nodes, consisting of (i) n n input nodes with in-degree 0 0 , (ii) function nodes labeled by f ∈ ℱ f\in\mathcal{F} , which take as arguments the predecessor nodes specified by their incoming edges in E n E_{n} , and (iii) m ⁡ ( n ) m(n) output nodes with out-degree 0 0 . The overall function is obtained by evaluating the graph in topological order. The size of the graph is | V n | |V_{n}| , denoted by size ⁡ ( G n ) \mathrm{size}(G_{n}) , and its depth is the length of the longest path from an input to an output node, denoted by depth ⁡ ( G n ) \mathrm{depth}(G_{n}) .

##### Assumptions on models.

Our goal is to evaluate the computational efficiency of each model via an asymptotic analysis of how the required number of reasoning steps or loops scales with the input size n n . Beyond time complexity , we also allow the space complexity of the model to scale with the input size n n . In particular, the embedding dimension in Transformer blocks can be viewed as analogous to the number of processors in classical parallel computation models. Accordingly, we adopt a non-uniform computational model, in which a different model is allowed for each input size. This non-uniform setting is standard in the study of circuit complexity and parallel computation ( Cook, 1985 ) , and is consistent with prior analyses of Transformers and CoT ( Sanford et al., 2024b ; Li et al., 2024 ) .

##### On the fairness of comparing steps and loops.

We analyze expressivity in terms of the number of reasoning steps. Although this may appear unfair in terms of raw computation, it is justified when comparing latency. Specifically, CoT benefits from KV caching, which makes each step computationally inexpensive; however, accessing cached states is typically memory-bound, leaving compute resources underutilized. In contrast, looped TFs recompute over the full sequence at each iteration, incurring higher arithmetic cost but achieving higher arithmetic intensity and better utilization of modern parallel hardware. As a result, the latency of looped TFs is comparable to that of CoT.

### 3.2 CoT Suffices with Size-scaled Steps and Latent Thought Suffices with Depth-scaled Iterations

We show how each model can evaluate DAGs, which provides a lower bound on their expressivity and offers intuition for the distinctions between the models, in terms of sequentiality and parallelizability. Before presenting our main result, we first state the underlying assumptions.

###### Definition 3.2 ( Merrill and Sabharwal, 2023 ) .

The model is log-precision , where each scalar is stored with O ⁡ ( log ⁡ n ) O(\log n) bits and every arithmetic operation is rounded to that precision.

###### Assumption 3.3 (Poly-size graph) .