SIM-CoT: Supervised Implicit Chain-of-Thought — part 3/4

Cleaned-text code points [31382, 47090). Bibliography omitted here; source.md includes it.

In this appendix, we provide additional analyses and supporting materials to complement the main text. First, in Sec. A , we investigate the scaling behavior of SIM-CoT on larger backbones and show its robustness across different model sizes. In Sec. B , we analyze the impact of decoder size on performance, highlighting the trade-offs between moderate and excessively large decoders. In Sec. C , we present experiments on combining soft thinking with SIM-CoT, including setup, results, and a detailed formulation with pseudocode. In Sec. D , we provide an overview of related work in explicit and implicit chain-of-thought reasoning. In Secs. E and F , we describe our implementation, hyperparameter configurations, and training/inference procedures, including benchmark and dataset details. In Sec. G , we introduce the SIM-CoT training procedure and provide pseudocode for step-level supervision. In Sec. H , we offer geometric diagnostics of the latent space, analyzing inter-latent distances and distance to the vocabulary center. In Sec. I , we discuss interpretability analysis, including latent visualization, distance analysis, and summary findings. Finally, we provide future directions, declarations on LLM usage and additional case studies on GSM8k to further illustrate the reasoning process and visualization choices.

## Appendix A Additional Analysis on Scaling to Larger Backbones

As shown in Table 3 , On LLaMA 3.2 3B , SIM-CoT improves over CODI by + 1.5 +1.5 points on GSM8k-Aug and + 1.6 +1.6 points on SVAMP, while maintaining comparable performance on GSM-Hard and MultiArith. This indicates that step-level implicit supervision can strengthen strong implicit reasoning baselines.

On LLaMA 3.1 8B , SIM-CoT yields + 3.0 +3.0 points on GSM8k-Aug, + 1.3 +1.3 points on SVAMP, and + 0.8 +0.8 points on MultiArith relative to CODI, with stable accuracy on GSM-Hard. Compared with SFT-CoT, it achieves higher accuracy on MultiArith ( 100.0 100.0 vs. 98.3 98.3 ) and SVAMP ( 79.4 79.4 vs. 73.1 73.1 ), while remaining similar on GSM-Hard.

These results confirm that SIM-CoT scales effectively to larger backbones, providing consistent gains while reducing reliance on trajectory-level supervision.

## Appendix B Additional Analysis on Decoder Sizes

Compared to the baseline, as shown in Table 4 (a), integrating a 1B-scale decoder leads to consistent improvements across all benchmarks. However, simply replacing the decoder with larger versions (3B or 8B) does not bring further benefits and slightly degrades performance.

These results suggest that moderate decoder scaling can enhance reasoning ability, but excessively large decoders may introduce optimization difficulties or misalignment with the backbone, ultimately limiting generalization. Another plausible explanation is that the 1B encoder and 1B decoder originate from the same model, thereby sharing a more compatible representation space that facilitates learning. In contrast, larger decoders (3B or 8B) may require additional projection to align with the 1B backbone, which could introduce mismatches and hinder training stability.

## Appendix C Additional Analysis on Soft Thinking

Soft thinking ( Zhang et al., 2025b ; Wu et al., 2025 ) is a training-free method for implicit reasoning in which the latent space is represented as a weighted average over the vocabulary embedding space. In contrast, SIM-CoT learns latent representations directly from data during training. To our knowledge, no prior work has evaluated a combination of these two approaches; our experiments provide the first such evaluation.

### C.1 Setup

We apply the proposed soft thinking mechanism on top of both Coconut and SIM-CoT, while adopting GPT-2 as the backbone model. To assess the effectiveness of this approach, we perform evaluations on a diverse set of mathematical reasoning benchmarks. The in-domain evaluation is carried out on GSM8k-Aug, which provides augmented training and testing samples closely aligned with the original GSM8k distribution. To further examine generalization beyond the training domain, we include three out-of-domain benchmarks: GSM-Hard, which contains more challenging arithmetic problems with subtle variations in reasoning steps; MultiArith, which evaluates performance on multi-step arithmetic operations requiring careful sequencing of addition, subtraction, multiplication, and division; and SVAMP, which focuses on variations of elementary word problems designed to test robustness to superficial changes in problem statements.

### C.2 Results

Table 5 (b) reports the results. Adding soft thinking improves accuracy in most cases. For Coconut, improvements are observed on GSM-Hard (+0.2) and MultiArith (+1.7), with a slight decrease on SVAMP (-0.2). For SIM-CoT, soft thinking consistently enhances performance: GSM8k-Aug (+0.2), GSM-Hard (+0.1), MultiArith (+0.7), and SVAMP (+0.1).

### C.3 Formulation

Let z ∈ ℝ d z\in\mathbb{R}^{d} denote a continuous latent token, and E ∈ ℝ | 𝒱 | × d E\in\mathbb{R}^{|\mathcal{V}|\times d} be the embedding matrix of the vocabulary 𝒱 \mathcal{V} . Our goal is to enrich the representational capacity of z z by incorporating soft thinking, which allows the latent space to draw information not only from its continuous representation but also from the semantic structure of the vocabulary. The process can be described in three steps.

#### Step 1. Vocabulary distribution.

The continuous latent token z z is first mapped into a probability distribution over the vocabulary space: p = softmax ​ ( W ​ z ) , p=\text{softmax}(Wz), where W ∈ ℝ | 𝒱 | × d W\in\mathbb{R}^{|\mathcal{V}|\times d} is the output projection matrix and p ∈ ℝ | 𝒱 | p\in\mathbb{R}^{|\mathcal{V}|} is the resulting distribution. This step can be viewed as interpreting the latent token in terms of vocabulary-level semantics, where each token in 𝒱 \mathcal{V} is assigned a likelihood according to its relevance to z z .

#### Step 2. Soft-thinking embedding.

Using the distribution p p , we compute a weighted mixture of vocabulary embeddings: z soft = E ⊤ ​ p = ∑ v ∈ 𝒱 p v ​ E v , z_{\text{soft}}=E^{\top}p=\sum_{v\in\mathcal{V}}p_{v}\,E_{v}, where E v E_{v} is the embedding vector corresponding to token v v . This operation can be seen as constructing a ”soft token” that captures multiple semantic hypotheses simultaneously, instead of committing to a single discrete vocabulary token. As a result, z soft z_{\text{soft}} provides richer and smoother information than a hard token lookup.

#### Step 3. Combination.

Finally, we combine the original continuous latent z z with the soft-thinking embedding z soft z_{\text{soft}} : z ′ = α ​ z + β ​ z soft , z^{\prime}=\alpha z+\beta z_{\text{soft}}, where α = continuous_weight \alpha=\texttt{continuous\_weight} and β = soft_weight \beta=\texttt{soft\_weight} are hyperparameters that balance the contribution of the continuous and soft-thinking components. This formulation allows z ′ z^{\prime} to retain the model’s learned continuous representations while also grounding them in the vocabulary space. Intuitively, the continuous part encourages compact reasoning within the latent space, whereas the soft-thinking component brings in semantic priors from the vocabulary, leading to more stable and interpretable reasoning.

The pseudocode implementation of the above process is presented as follows.

### C.4 Analysis

The results demonstrate that soft thinking complements training-based implicit reasoning. The hybrid latent z ′ z^{\prime} integrates semantics learned through training and distributional information from vocabulary mixing, which enables the model to explore diverse intermediate states rather than committing to a single deterministic path. This leads to improvements in both in-domain and out-of-domain benchmarks. Our findings suggest that combining training-free construction with training-based supervision provides gains beyond either approach in isolation.

## Appendix D Related Work

Explicit chain-of-thought reasoning. Chain-of-thought (CoT) prompting enables large language models (LLMs) to generate intermediate reasoning steps before producing the final answer ( Wei et al., 2022 ) . This approach has been widely studied and extended in many directions. Self-consistency samples multiple reasoning paths and selects the majority answer to improve reliability ( Wang et al., 2023 ) . Least-to-most prompting decomposes a complex question into simpler sub-problems and solves them in order ( Zhou et al., 2023 ) . Reflection-based reasoning allows the model to revise or verify its own intermediate steps, leading to better correctness ( Shinn et al., 2023 ; Madaan et al., 2023 ) . Other works focus on using external tools or symbolic solvers together with explicit reasoning, which further improves accuracy in mathematics and program synthesis ( Yao et al., 2023b ) . Methods such as progressive-hint prompting ( Zheng et al., 2023 ) and step-level feedback ( Wei et al., 2025a ) study how supervision can be incorporated into explicit reasoning to make reasoning more structured. Despite these advances, explicit CoT has clear drawbacks. Because it generates long token sequences, inference cost grows rapidly with reasoning length, and many intermediate steps are redundant or irrelevant to the final answer. Moreover, since explicit reasoning is restricted to tokens from a fixed vocabulary, it often commits to a single trajectory and shows limited reasoning diversity ( Li et al., 2025 ; Zhang et al., 2025b ; Xu et al., 2025 ) .

Implicit chain-of-thought reasoning. Implicit CoT performs multi-step computation in a continuous latent space instead of emitting long textual traces, reducing decoded length while keeping internal structure. Prior work follows four practical routes. First, knowledge internalization trains models to carry out reasoning internally by progressively removing explicit traces or by using dedicated control embeddings; examples include iCoT-SI ( Deng et al., 2024 ) , which removes steps during training to internalize reasoning. Second, architectural modification controls compute by reusing or skipping layers, or by adding light recurrence, so models can refine hidden states without lengthening outputs ( Saunshi et al., 2025 ; Chen et al., 2025 ; Cheng & Van Durme, 2024 ; Su et al., 2025 ; Mohtashami et al., 2023 ; Geiping et al., 2025 ) . Third, training-free methods construct continuous latents directly from the model’s probability distribution over the vocabulary; Soft Thinking mixes embeddings by probability to form “concept” tokens that explore alternative paths without updating weights, which improves efficiency and diversity but does not bind each latent to step-level semantics ( Zhang et al., 2025b ; Wu et al., 2025 ) .

The fourth route, auto-regressive latent reasoning , updates and concatenates latent states in place of some token-level decoding and is the most relevant to our work ( Xu et al., 2025 ; Tan et al., 2025 ) . Coconut applies answer-level supervision—training on the final answer while leaving intermediate latents weakly constrained ( Hao et al., 2025 ) . CODI adds trajectory-level distillation by aligning an implicit trajectory with an explicit CoT trace, narrowing the gap to explicit CoT but giving only coarse guidance to intermediate steps ( Shen et al., 2025b ) . However, the implicit token length in CODI is fixed during training, which limits its flexibility and makes it less suitable for scaling to variable or longer reasoning chains. Our framework remains in the auto-regressive setting but changes the supervision: during training, each latent is aligned with its corresponding textual step ( step-level supervision), distributing learning signals across the full latent chain to improve stability and semantic fidelity of intermediate states; at inference, the decoder is discarded, ensuring that the decoding cost remains identical to that of standard implicit CoT methods (e.g., Coconut).

## Appendix E Implementation and Training Details

We provide the full hyperparameter settings, training procedures, and additional analysis used in our experiments. Unless otherwise specified, we use the Adam optimizer with β 1 = 0.9 \beta_{1}=0.9 , β 2 = 0.999 \beta_{2}=0.999 , and weight decay of 0.1 0.1 . Batch size is set to 128 for GPT-2 and LLaMA 1B, and 64 for LLaMA 3B and 8B. Early stopping is applied with a patience of 3 epochs. We now describe the training setups for Coconut, CODI, and our SIM-CoT.

### E.1 Coconut Training Setup

Following Hao et al. (2025) , GPT-2 and LLaMA 1B ( Radford et al., 2019 ; Meta, 2024 ) are trained with a fixed learning rate of 1 × 10 − 4 1\times 10^{-4} . One implicit latent corresponds to two implicit tokens. A curriculum is applied: every three epochs, one explicit reasoning step is replaced by an implicit latent until the maximum number of latent steps is reached. After this expansion, training continues for 15 additional epochs.

### E.2 CODI Training Setup

For larger backbones such as LLaMA 3B and LLaMA 8B, we adopt task-specific hyperparameter settings to ensure stable training. In particular, we use a learning rate of 3 × 10 − 4 3\times 10^{-4} for LLaMA 3B and train for 8 epochs, while for LLaMA 8B the learning rate is reduced to 1 × 10 − 4 1\times 10^{-4} with 6 training epochs. These choices are motivated by the increased sensitivity of larger models to optimization dynamics, where smaller learning rates and fewer epochs help to prevent overfitting and instability.

When reproducing CODI on GPT-2 and LLaMA 1B, we strictly follow the configurations reported by Shen et al. (2025b) . Specifically, we use a learning rate of 3 × 10 − 3 3\times 10^{-3} with 40 epochs for GPT-2, and a learning rate of 8 × 10 − 4 8\times 10^{-4} with 10 epochs for LLaMA 1B. Adopting these settings ensures that our results are directly comparable to prior work and isolates the effect of our proposed method, rather than confounding it with differences in optimization schedules.

### E.3 Summary of Hyperparameters

## Appendix F Training and Inference Details

#### Curriculum for K K .

We use a curriculum schedule to gradually increase the number of implicit steps. Each latent corresponds to two implicit tokens. Let K max K_{\max} denote the maximum number of latents. Starting from K ( 0 ) = 0 K^{(0)}=0 , the number of implicit steps after epoch e e is K ( e ) = min ⁡ ( K max , ⌊ e Δ ​ e ⌋ ) , K^{(e)}\;=\;\min\!\left(K_{\max},\;\Big\lfloor\tfrac{e}{\Delta e}\Big\rfloor\right), where Δ ​ e \Delta e is the update interval in epochs. Once K ( e ) K^{(e)} reaches K max K_{\max} , it remains fixed for the remainder of training.

#### Inference and Efficiency.

At inference time, the auxiliary decoder is removed and only the base model is executed: U ( 0 ) = [ e ( x 1 ) , … , e ( x T ) ] , for k = 1 , … , K : z k = H θ ( U ( k − 1 ) ) , U ( k ) = U ( k − 1 ) ⊕ z k , U^{(0)}=[\,e(x_{1}),\ldots,e(x_{T})\,],\quad\text{for }k=1,\ldots,K:\;z_{k}=H_{\theta}(U^{(k-1)}),\quad U^{(k)}=U^{(k-1)}\oplus z_{k}, and after K K implicit steps the model switches back to explicit decoding to generate the final answer sequence a a as in Eq. equation 4 . The total decoding length is T + K + L a T+K+L_{a} , where T T is the input length, K K the number of implicit steps, and L a L_{a} the answer length. In practice, the cost is comparable to other implicit reasoning methods because K K is moderate. In tasks where explicit CoT requires long trajectories ( L CoT ≫ K L_{\text{CoT}}\gg K ), the implicit formulation reduces decoding positions, providing efficiency gains without loss of reasoning accuracy.

#### Benchmark Detail.