SIM-CoT: Supervised Implicit Chain-of-Thought — part 1/4

Cleaned-text code points [0, 15734). Bibliography omitted here; source.md includes it.

# SIM-CoT: Supervised Implicit Chain-of-Thought

###### Abstract

Implicit Chain-of-Thought (CoT) methods offer a token-efficient alternative to explicit CoT reasoning in Large Language Models (LLMs), but a persistent performance gap has limited their adoption. We identify a core latent instability issue when scaling the computational budget of implicit CoT: as the number of reasoning tokens increases, training often becomes unstable and collapses. Our analysis shows that this instability arises from latent representations becoming homogeneous and losing semantic diversity, caused by insufficient step-level supervision in current implicit CoT methods. To address this, we propose SIM-CoT , a plug-and-play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space. SIM-CoT employs an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information. The auxiliary decoder is removed at inference, preserving the efficiency of implicit CoT with no added overhead. It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis. SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. It further surpasses the explicit CoT baseline on GPT-2 by 2.1% with 2.3 × \times greater token efficiency, while closing the performance gap on larger models like LLaMA-3.1 8B.

## 1 Introduction

“ Measure what is measurable, and make measurable what is not so. ” — Galileo Galilei

The strong reasoning capabilities of Large Language Models (LLMs) ( OpenAI, 2024 ; Google, 2024 ; Anthropic, 2024 ) are often unlocked through explicit Chain-of-Thought (CoT) prompting ( Wei et al., 2022 ) . The explicit CoT approach enables LLMs to solve complex problems in a step-by-step manner, yielding high performance in domains like mathematics and programming ( Guo et al., 2025 ; Muennighoff et al., 2025 ) . Despite its advantages, explicit CoT also faces several limitations. For instance, explicit CoT approaches must verbalize intermediate thoughts from a fixed vocabulary, thereby precluding the exploration of alternative solution paths ( Li et al., 2025 ; Zhang et al., 2025b ) . Additionally, the generation of extensive intermediate sequences significantly increases inference cost and can result in redundant over-thinking steps or unnecessary verbosity ( Chen et al., 2024b ) .

To address the flexibility and efficiency issues of explicit CoT methods, recent implicit CoT approaches ( Hao et al., 2025 ; Zhang et al., 2025b ; Li et al., 2025 ) have been proposed by representing reasoning in a continuous latent space rather than as a sequence of discrete text tokens. The implicit CoT methods allow each latent representation to encode richer information than a single explicit token, often with a significantly smaller number of latents than the length of an explicit reasoning chain. Early representative implicit work like Coconut ( Hao et al., 2025 ) improves efficiency while still capturing useful intermediate structure. More recent approaches, such as CODI ( Shen et al., 2025b ) , further apply trajectory-level distillation from explicit reasoning paths to enhance performance. Despite these advancements, a performance gap still exists between existing implicit CoT methods and their explicit counterparts. The implicit CoT approaches are fast , token-efficient but less accurate , which currently limits their broader application.

To narrow the performance gap, inspired by the success of explicit CoT that scales computational budget for better performance, we explore a similar strategy for implicit CoT methods by increasing the number of implicit tokens. However, in Fig. 1 (a) , we reveal one underlying latent instability issue in current implicit CoT approaches. As we extend the number of implicit tokens from the default three ( Hao et al., 2025 ) to five, the training process initially improves accuracy but becomes unstable and sometimes collapses entirely. To interpret the latent instability issue , we analyze implicit tokens from models trained on math reasoning data GSM8K-Aug ( Deng et al., 2024 ) . We follow previous works ( Hao et al., 2025 ; Deng et al., 2024 ) to project the implicit tokens through the LM head and examine their top decoded tokens for analysis. As shown in Fig. 1 (b) , failed models tend to collapse into homogeneous latent states. While successful reasoning requires capturing both numerical and operator information, the implicit tokens of failed models primarily represent numbers, almost completely losing the critical operator information. Fig. 1 (c) further demonstrates that a model’s collapse is accompanied by two changes: a reduction in the inter-latent distance and a drift of the latent states away from the central vocabulary embedding space. The latent representations of failed models become too similar and lose their semantic connection to the tokens they are meant to represent. Fig. 1 (d) provides an example of the semantic homogenization. A normal model (top) maintains a large distance between its two latent tokens, allowing them to capture distinct information for numbers and operators. In contrast, a failed model (bottom)’s latent tokens become homogeneous, with both states decoding to similar information, primarily numbers.

Our observation ( Fig. 1 ) reveals the reasons for the latent instability issue: a lack of sufficient step-level supervision for existing implicit methods to maintain the rich and varied internal representations. Without stronger guidance, the latent space collapses, losing its diversity and making it impossible to reliably encode the distinct, step-level reasoning needed for complex reasoning tasks. Motivated by our findings, we propose S upervised IM plicit-CoT ( SIM-CoT ), a plug-and-play module that introduces step-level supervision for implicit CoT approaches to alleviate the latent instability issue. Instead of supervising only the final answer ( Hao et al., 2025 ) or the trajectory ( Shen et al., 2025b ) , SIM-CoT uses an auxiliary decoder to align each implicit token with its corresponding explicit reasoning step during training. The step-level supervision for implicit tokens stabilizes optimization, prevents collapse, and ensures that latent tokens capture meaningful reasoning content. Crucially, because the auxiliary decoder is removed during inference, our approach incurs virtually no extra computational cost, making it as efficient as standard implicit CoT approaches. Beyond accuracy , stability , and efficiency , the auxiliary decoder also affords interpretability of implicit reasoning. During training, it defines a projection from latent tokens to the explicit reasoning vocabulary, enabling us to decode each latent step into a human-interpretable summary for verification or error diagnosis.

Experiments show that SIM-CoT acts as a plug-and-play module that boosts both accuracy and stability. We show that SIM-CoT can be effortlessly combined with various implicit CoT approaches such as Coconut ( Hao et al., 2025 ) , CODI ( Shen et al., 2025b ) , and training-free approaches ( Zhang et al., 2025b ) to further enhance reasoning performance. On GPT-2, SIM-CoT surpasses both the strong explicit baseline (supervised fine-tuning on explicit CoT data) by 2.1%, and outperforms existing implicit methods Coconut and CODI by 8.2% and 4.3%, respectively. The performance trend holds as the method scales to larger models such as the LLaMA series. SIM-CoT achieves improvements over CODI of 3.4% (LLaMA-3.2 1B), 1.5% (LLaMA-3.2 3B), and 3.0% (LLaMA-3.1 8B), in addition to a 9.0% gain over Coconut on the LLaMA-3.2 1B model. Furthermore, while previous implicit CoT approaches (e.g., Coconut) collapse when scaled to 8 or 16 implicit tokens, SIM-CoT remains stable and continues to boost performance.

In summary, our contributions are as follows: 1) We provide a systematic analysis of the latent instability issue of implicit CoT approaches, showing that instability and collapse arise from insufficient supervision. 2) We introduce SIM-CoT, which applies step-level supervision to the model’s implicit tokens. SIM-CoT not only integrates seamlessly with existing implicit CoT approaches and boosts performance with minimal inference overhead, but also affords interpretability of implicit reasoning by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization of semantic roles and diagnosis. 3) Through extensive experiments, we demonstrate that SIM-CoT not only improves accuracy in the in-domain dataset, but also generalizes effectively to out-of-domain datasets. The performance gains are consistent across a range of LLMs, including GPT-2 and recent LLaMA 3 models (1B, 3B, and 8B).

## 2 Analysis of Implicit COT: the Latent Instability Issue

We first present an analysis ( Fig. 1 ) of the limitations in implicit latent CoT approaches. We follow Coconut ( Hao et al., 2025 ) and analyze implicit latents by projecting them through the LM head and examining the top-8 decoded tokens to understand the semantic and geometric properties.

Latent Instability Issue. Fig. 1 (a) shows the training process of Coconut when the number of implicit latent tokens is progressively increased. Initially, as the number of latents increases from one to four, the model’s accuracy generally improves, suggesting that using more latents can enhance performance. However, a significant drop in accuracy occurs when the number of latents is scaled to five, with performance collapsing to its worst point of 12.5%. The latent instability issue indicates that the implicit reasoning approach is sensitive to the choice of the number of latent tokens, as shown by the sharp drop and subsequent fluctuations in accuracy after adding the fifth latent.

Information Loss. Fig. 1 (b) presents an analysis of how different levels of accuracy are affected by the number of latent tokens, using accuracy metrics at three levels: number, operator, and answer. The bar chart reveals a clear trend: as the number of latent tokens increases from 1 to 5, there is a general decline in performance across all three metrics, especially for the operator accuracy. The strong correlation between increased latent tokens and declining performance, particularly the sharp fall during failure, suggests that implicit latents do not consistently capture the necessary compositional reasoning process without more explicit, fine-grained supervision.

Shifted Distance. Fig. 1 (c) examines the geometric properties of the latent representations during training. Two metrics are analyzed: the Latent Distance (red), which measures the average distance between pairs of latent vectors, and the Vocab Distance (blue), which measures the average distance from each latent vector to the center of the vocabulary embedding space. When the latent CoT model collapses, the latent distance decreases sharply, indicating that the latent vectors are collapsing and becoming nearly identical, losing their distinctiveness. Simultaneously, the vocab distance increases, showing that these collapsing latents are drifting away from the main lexical embedding space and are no longer grounded in the fundamental token representations used by the model.

Semantic Homogenization. Fig. 1 (d) provides a qualitative analysis of the content of the latent tokens in a normal case versus a failed model. In the normal implicit model (middle), the decoded tokens from the latents are diverse and meaningful. In the failed implicit model (bottom), the semantic content of the latents becomes highly homogeneous. Latent 1 and Latent 2 contain mainly numbers, lacking operators or symbolic information needed for calculation. This shows that successful training produces latents with step-wise reasoning, while without explicit supervision, the latent space collapses into uniform numerical forms.

Summary. Our analysis across Fig. 1 (a-d) highlights a crucial trade-off between diversity and stability. When the model collapses, it loses both its diversity (as the latents become too similar) and its stability (as the latents move away from the token space), leading to catastrophic information loss and a complete failure of the reasoning process, as shown by the sharp drop in overall accuracy. These combined findings show that without proper guidance, the latent space degenerates, losing its ability to represent distinct reasoning steps. These challenges motivate our proposed method, which introduces step-level implicit supervision to stabilize the training process and enrich unique semantic content of each latent, all while maintaining efficiency during inference.

## 3 Methodology

Overview. As shown in Fig. 2 , early implicit reasoning studies differ mainly in supervision granularity: Coconut (top left) uses answer-level supervision, while CODI (top right) introduces trajectory-level signals via distillation. Both remain coarse and do not tell the model which latent should encode which step. We propose SIM-CoT , which provides step-level implicit supervision : During an implicit phase , the LLM runs for a fixed number K K of reasoning steps; at each step k k it takes the last hidden state as the implicit latent z k z_{k} and appends it to the sequence as the next “token” vector. After K K steps, the model switches back to explicit decoding over the vocabulary to generate the final answer. A decoder is used only in training to align each z k z_{k} with the textual content of the k k -th reasoning step; at inference, the decoder is removed, so the runtime is essentially that of direct answer generation plus K K forward positions, which is far shorter than explicit CoT token lengths.

### 3.1 Notation

Let 𝒱 \mathcal{V} be the vocabulary and E ∈ ℝ | 𝒱 | × d E\in\mathbb{R}^{|\mathcal{V}|\times d} the token embedding matrix. A question is x = ( x 1 , … , x T ) ∈ 𝒱 T x=(x_{1},\ldots,x_{T})\in\mathcal{V}^{T} with embedded prefix U ( 0 ) = ( e ⁡ ( x 1 ) , … , e ⁡ ( x T ) ) , e ⁡ ( ⋅ ) ∈ ℝ d . U^{(0)}=\big(e(x_{1}),\ldots,e(x_{T})\big),\quad e(\cdot)\in\mathbb{R}^{d}. We run an autoregressive LLM F θ F_{\theta} on any prefix U = ( u 1 , … , u m ) U=(u_{1},\ldots,u_{m}) of d d -dimensional vectors (tokens or latents). Denote the last-layer hidden state at the final position by H θ ​ ( U ) ∈ ℝ d . H_{\theta}(U)\in\mathbb{R}^{d}.

For supervision, the k k -th textual step is s k = ( y k , 1 , … , y k , L k ) ∈ 𝒱 L k s_{k}=(y_{k,1},\ldots,y_{k,L_{k}})\in\mathcal{V}^{L_{k}} , and the answer is a = ( a 1 , … , a L a ) ∈ 𝒱 L a a=(a_{1},\ldots,a_{L_{a}})\in\mathcal{V}^{L_{a}} . The auxiliary decoder has parameters ϕ \phi ; the LLM has parameters θ \theta .

### 3.2 Implicit Phase: Latent Construction by Last Hidden States

We fix the number of implicit reasoning steps K K in advance. For each step k = 1 , … , K k=1,\ldots,K , z k = H θ ​ ( U ( k − 1 ) ) ∈ ℝ d , U ( k ) = U ( k − 1 ) ⊕ z k , z_{k}\;=\;H_{\theta}\!\big(U^{(k-1)}\big)\in\mathbb{R}^{d},\qquad U^{(k)}\;=\;U^{(k-1)}\;\oplus\;z_{k}, (1) where ⊕ \oplus denotes concatenation along the time axis. The implicit chain-of-thought is therefore represented as a continuous sequence of hidden states z 1 : K = ( z 1 , … , z K ) z_{1:K}=(z_{1},\ldots,z_{K}) , which are autoregressively generated and appended to the context before the model switches to explicit decoding.

### 3.3 Explicit Phase: Answer Decoding over the Vocabulary