Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — part 5/6

Cleaned-text code points [61923, 77395). Bibliography omitted here; source.md includes it.

We follow the staged curriculum of Hao et al. [2025] , converting one additional CoT step into a latent block every E stage E_{\text{stage}} epochs. At epoch e e , we use K e = min ⁡ ( ⌊ e / E stage ⌋ , K ) , {K}_{e}\;=\;\min\!\left(\lfloor e/E_{\text{stage}}\rfloor,\;{K}\right), latent blocks: up to the first K e {K}_{e} CoT steps become latent blocks, and the rest remain visible tokens. Thus e = 0 e{=}0 is standard CoT fine-tuning and the curriculum saturates at K e = K {K}_{e}{=}{K} . We set the loop count at epoch e e to R e = r ​ K e {R}_{e}=r\,{K}_{e} , where r r is the loops-per-block ratio. Since each pass refines all blocks in parallel, K e {K}_{e} controls the latent workspace width and R e {R}_{e} controls the depth. Unless stated otherwise, E stage = 1 E_{\text{stage}}{=}1 and r = 1 r{=}1 , so the saturated model uses R = K = 6 {R}{=}{K}{=}6 . The loop-depth ablation ( Section 4.3.2 ) fixes K = 6 {K}{=}6 and varies r r to decouple depth from width.

##### Optimization details.

We initialize from each backbone’s standard explicit CoT checkpoint, and fine-tune all parameters with AdamW and gradient-norm clipping at 1.0 1.0 , in bf16 for the Llama backbones and fp32 for GPT-2 . The learning rate is constant: × 10 − 4 1\!\times\!10^{-4} for GPT-2 and Llama-3.2-1B-Instruct , and × 10 − 5 5\!\times\!10^{-5} for Llama-3.2-3B-Instruct . The training objective ( Equation 6 ) weights the step loss at λ step = 0.05 {\lambda_{\mathrm{step}}}{=}0.05 (for both LOTUS and LOTUS-aux). Adding CODI weights its loss at 1.0 1.0 . We train for 30 30 epochs with an overall batch size of 128 128 , and select the best checkpoint by GSM8k validation accuracy, though the final epoch checkpoints perform similarly.

##### Natural-language training setup.

For the natural-language stress-test on LOTUS, we keep the same 3B looped backbone but switch to the natural-language GSM8k-Aug training set [ Wu et al., 2025 ] , increase latent width to c = 50 c{=}50 and set λ step {\lambda_{\mathrm{step}}} to 0.033 0.033 . We initialize from the CoT checkpoint trained on the natural-language GSM8k-Aug. Other settings are identical to those used for the original GSM8k-Aug training.

##### LOTUS-aux auxiliary decoder.

For LOTUS-aux, the auxiliary decoder is a full-size deep copy of the base model, initialized from its weights but trained together with the main model. This full-size choice matches SIM-CoT [ Wei et al., 2025 ] , whose auxiliary decoder is architecturally identical to the base model. We also tried lightweight auxiliary decoders but found them unstable in training.

##### Evaluation.

Inference uses greedy decoding with batch size 1 1 . This isolates per-example latency from batching effects and calculates the average latency per sample accurately. Latency is measured on a single NVIDIA H100 and decomposed into query, thought, and answer phases ( Table 2 ).

##### Batched KV-cache sharing.

For training, we left-pad shorter examples so the ⟨ lat ⟩ {\langle\texttt{lat}\rangle} positions align across the batch. This lets us compute each example’s question-prefix KV cache once and reuse it across all R {R} looped iterations.

## Appendix E PCL Versus Autoregressive NLL Lower Bounds

Here we provide the explanation mentioned in Section 5.1 . PCL ( Section 3.3.2 ) scores each gold token in parallel, − ∑ i , j log p 𝜽 ( T i , j ∣ 𝒉 ( R ) ) -\sum_{i,j}\log{{p_{{{{\bm{\theta}}}}}}}({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}}) , whereas the autoregressive factorization also conditions on the preceding gold tokens, − ∑ i , j log p 𝜽 ( T i , j ∣ 𝒉 ( R ) , T < ( i , j ) ) -\sum_{i,j}\log{{p_{{{{\bm{\theta}}}}}}}({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}},{T}_{<({i},j)}) , where T < ( i , j ) {T}_{<({i},j)} denotes all gold tokens before position ( i , j ) ({i},j) in reading order. The lowest expected NLL each can reach is the corresponding conditional entropy, and since conditioning never increases entropy, H ⁡ ( T i , j ∣ 𝒉 ( R ) ) ≥ H ⁡ ( T i , j ∣ 𝒉 ( R ) , T < ( i , j ) ) , H\!\big({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}}\big)\;\geq\;H\!\big({T_{{i},j}}\mid{{{\bm{h}}}^{({R})}},{T}_{<({i},j)}\big), (11) with equality only when the gold tokens are conditionally independent given the latents. So PCL has a strictly higher NLL floor whenever cross-token dependence exists.

## Appendix F Details for Latent Representation Analysis ( Section 5 )

### F.1 Greedy Readout Examples

Here we give examples of two correctly answered and two incorrectly answered GSM8K test problems. We show three readouts of the post-loop latents: (A) the per-block top- 1 1 token sequence reading 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through LOTUS’s base LM head f head {f_{\mathrm{head}}} ; (B) the same readout applied to LOTUS-aux’s post-loop latents 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through its auxiliary head g head {g_{\mathrm{head}}} ; and (C) the autoregressive CoT produced by LOTUS-aux’s trained auxiliary decoder from 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} , conditioned on the same latents and decoded greedily from the start-of-CoT token. (A) and (B) show the top- 1 1 token stream per block verbatim (newlines rendered as \n ), truncated at 60 60 characters per block with ... marking the cut. (C) shows the raw aux-decoder output. We additionally report the final answer the model generates autoregressively from 𝒉 ( R ) {{{\bm{h}}}^{({R})}} after all R = 6 {R}{=}6 loops. In each (A) / (B) readout we list the K = 6 {K}{=}6 latent blocks top to bottom, labeled B0 – B5 .

For readability, blue marks the readable gold content: well-formed <<…>> CoT steps and exact gold chain numbers (step operands and intermediate results), including a number stuttered back-to-back such as 540540 for 540 540 . Wrong digit runs (e.g. 120120 or 3636 ) are left unmarked, and within each readout line we highlight only the first surfacing of each gold value.

##### Example 1 — Janet’s ducks.

Question: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market? Reference: <<16-3-4=9>> , <<9*2=18>> . Gold answer 18 18 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<3+4=7>> \n00>>\n7>>\n>>\n>>\n 16 16-- 9 999999<< B1: <<16-7=9>> \n00>>\n>>\n>>\n>>\n>>\n*2== 18 18>>\n>>\n>>\n1818<< B2: <<9*2=18>> \n00>>\n>>\n>>\n18>>\n>>\n>>\n= 1818>>\n18>>\n>>\n... B3: <<18*2=18>> \n>>\n>>\n>>\n>>\n>>\n>>\n>>\n>>\n= 9 9>>\n>>\n18>>... B4: ###134* 7 =>>\n 18 >>\n>>\n>>\n>>\n>>\n>>\n>>\n2=2 .>>\n181818>>... B5: >>\n 18 =21818>>\n>>\n>>\n===.###2== ,>>\n1818222 (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: <<’+) 7 ))<< 16 16--7= 1313\n\n###22*22 B1: =2189)#########2222= 18 18#########22* 9 = B2: 18 18 per#########2922 =1818 fresh###### 9 9* no 16 = 18 B3: 18 fresh<|eot_id|>######### 9 * no no nothing=1818 Clean<|eot_... B4: 18 ly<|eot_id|>###### no nothing no nothing 1818<|eot_id|>###... B5: ############ nothing nothing nothing= 18 1818############### n... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<3+4=7>> \n <<16-7=9>> \n <<9*2=18>> \n <<18=18>> \n <<9*2=18>> \n <<18+18=18>> \n <<18*2=18>> \n <<18=18>> Final answer (autoregressive, after 6 6 loops). LOTUS: ### 18 . LOTUS-aux: ### 18 . Gold: 18 18 . Both correct. Reading the chains. (A): the gold intermediates 𝟗 \mathbf{9} and 𝟏𝟖 \mathbf{18} surface from B1 onward and stabilize by B5. (B): the same numerics ( 9 , 18 , 16 , 7 9,18,16,7 ) are present but interleaved with <|eot_id|> runs and filler. (C): the aux decoder produces the gold first two steps <<3+4=7>> , <<16-7=9>> , <<9*2=18>> and then loops on 18 18 .

##### Example 2 — James’s sprints.

Question: James decides to run 3 sprints 3 times a week. He runs 60 meters each sprint. How many total meters does he run a week? Reference: <<3*3=9>> , <<9*60=540>> . Gold answer 540 540 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<3*60=180>> \n00>>\n180180>>\n>>\n3== 540 540>>\n>>\n>>\n>>\n>... B1: <<180*3=540>> \n00>>\n>>\n>>\n540>>\n### 540>>\n>>\n >>\n... B2: <<540*540=540>> \n00>>\n>>\n>>\n>>\n>>\n>>\n### 540>>\n>>\n ... B3: <<540=180=540>> \n540>>\n>>\n>>\n>>\n>>\n>>\n### 540>>\n>>\n... B4: ### 540 = =>>\n###>>\n>>\n>>\n 2 ###### 540540>>\n>>\n ### B5: ### == >>\n######### >>\n### >>\n>>\n >>\n (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 3 3*3= 180 180180*###9993== 540 540)###### 9 B1: 9 * 60 = 540 540))#########9* 3 nothing = 540540)############ B2: /* nothing= 540 540)#########540+### nothing 540############... B3: nothing is 540 540###############540,### also 540540540#####... B4: is 540 540540############540,### also 540540#################... B5: 540 540###############540### definitely 540540540############... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<3*60=180>> \n <<180*3=540>> \n <<540=540>> \n <<540/60=9>> \n <<540=540>> \n <<540/10=540>> \n <<540>> Final answer (autoregressive, after 6 6 loops). LOTUS: ### 540 . LOTUS-aux: ### 540 . Gold: 540 540 . Both correct. Reading the chains. (A): the gold step <<3*60=180>> appears in B0 already (a valid alternative decomposition: meters per day first), and 𝟓𝟒𝟎 \mathbf{540} stabilizes from B1. (B): 180,540 , 9 , 3 180,540,9,3 are all present but with heavy ### runs around them. (C): the aux decoder reproduces 𝟏𝟖𝟎 → 𝟓𝟒𝟎 \mathbf{180}\to\mathbf{540} cleanly, then loops.

##### Example 3 — Carla’s download (failure).

Question: Carla is downloading a 200 GB file. Normally she can download 2 GB/minute, but 40% of the way through the download, Windows forces a restart to install updates, which takes 20 minutes. Then Carla has to restart the download from the beginning. How long does it take to download the file? Reference: <<200*40*.01=80>> , <<80/2=40>> , <<200/2=100>> , <<40+100+20=160>> . Gold answer 160 160 .

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<200/2=6=120>> \n6>>\n>>\n>>\n>>\n>>\n2>>\n505050>>\n605050<... B1: <<40*.2=40>> \n120>>\n666>>\n>>\n>>\n>>\n>>\n60>>\n 80 80>>\n>>... B2: <<60+40=120>> \n8>>\n>>\n>>\n>>\n>>\n>>\n>>\n6050>>\n>>\n>>\n... B3: <<140+20=120>> \n8>>\n33>>\n>>\n>>\n>>\n>>\n2020=120>>\n>>\n>... B4: <<140+20=120>> \n120>>\n33>>\n>>\n>>\n>>\n###+60+/1>>\n>>\n>>... B5: 160 160+ 20 =120>>\n180>>\n>>\n>>\n>>\n180>>\n###+600+100120>>\... (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 100200/) 100 100))<<100100** 40 percentage,404040,<<100-+ B1: 20 20= 80 + and### 100 100+20++ 120120)###100100+220+ B2: 120120 plus### 100 100*10020 140 plus###12080+2100= 120240)\n... B3: ###120120+ 20 20+120140140######120+202020 140140)\n######120+ B4: +2= 140140)\n)\n######120120+ 20 2020 140100)\n Ris###### B5: ###120+ 20 20= 100 100 Ris#########120+20= 100100 Ris##########... (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<100+20=120>> \n <<120+20=140>> \n <<100+20=120>> \n### 140 Final answer (autoregressive, after 6 6 loops). LOTUS: ### 180 . LOTUS-aux: ### 120 . Gold: 160 160 . Reading the chains. The latents recover most of the right intermediates ( 80,100 , 𝟒𝟎 , 𝟐𝟎 \mathbf{80,100,40,20} ) but fail to combine them: (A) flickers between 120,140,160,180 120,140,160,180 in the late blocks; (B) commits to 120 + 20 120+20 chains; (C) outputs <<100+20=120>>, <<120+20=140>>, ... and emits 140 140 . Both methods land on plausible-but-wrong totals.

##### Example 4 — Melanie’s vacuums (failure).

Question: Melanie is a door-to-door saleswoman. She sold a third of her vacuum cleaners at the green house, 2 more to the red house, and half of what was left at the orange house. If Melanie has 5 vacuum cleaners left, how many did she start with? Reference: <<5*2=10>> , <<10+2=12>> . Gold answer 18 18 . One extra step is omitted from the reference; the true chain is → → → 18 5\!\to\!10\!\to\!12\!\to\!18 ( 12 × 3 / 2 = 18 12\times 3/2=18 ).

(A) LOTUS LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through f head {f_{\mathrm{head}}} ). B0: <<5*2=10>> \n33>>\n 12 >>\n12>>\n22=1312>>\n1212121313<< B1: <<10+2=17>> \n 5 >>\n5>>\n17>>\n>>\n1//1181818.55<< B2: <<9/(1/2/5=20.5>> \n20>>\n15.18>>\n>>\n>>\n>>\n5>>\n<< B3: <<15*1=30>> \n15>>\n 5 >>\n5>>\n20>>\n5=151818>>\n>>\n55<< B4: <<3=15>> \n30>>\n15>>\n 5 >>\n5>>\n>>\n>>\n55515>>\n15>>\n5>>\n... B5: <<15+5=15>> \n15>>\n5>>\n>>\n>>\n17>>\n55515>>\n 18 >>\n18>>\n>... (B) LOTUS-aux LM head ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through g head {g_{\mathrm{head}}} ). B0: 105* half 10 10)+101010+2)= 12 12)<<1212*3 B1: 3)1512..<< 12 1212*3 third = 3636.###121212*3) B2: 3636)###### 12 +33= 3636)######36+125= B3: 3636/#########36+ 12 ++ 4836 Egg#########36+12+= B4: 483636#########+ 5 == 4836############36+5 3636### B5: ############15+3 3636###############15###1 = 363615### (C) LOTUS-aux auxiliary decoder ( 𝒛 ~ ( t ) {\widetilde{{{\bm{z}}}}^{(t)}} through g head {g_{\mathrm{head}}} , FR). <<10+2=12>> \n <<10+2=12>> \n <<10+2=12>> \n <<12+2=14>> \n... Final answer (autoregressive, after 6 6 loops). LOTUS: ### 30 . LOTUS-aux: ### 36 . Gold: 18 18 . Reading the chains. (A): LOTUS’s latents flicker between 15 , 18 , 30 15,18,30 across blocks. The valid sub-chain → → → 𝟏𝟖 \mathbf{5\!\to\!10\!\to\!12\!\to\!18} surfaces in B5, but the final readout commits to 30 30 . (B): LOTUS-aux locks onto a wrong × 3 \times 3 branch (presumably “a third” read as multiply-by-three), with 36 36 saturating B4 to B5. (C): the aux decoder loops on <<10+2=12>> .

What these readouts are (and are not). The readouts (A), (B), (C) are just diagnostic projections of the post-loop continuous latents back into the discrete token space. They are not part of the inference pipeline. At inference, the final answer is generated autoregressively from the continuous post-loop latents (the “Final answer” line in each example), conditioning on the latent representation itself rather than on any discretized chain. The intermediate streams are intended to give intuition about what numbers the latents place mass on , not as a quality measure of the model’s reasoning: a noisy or repetitive (A)/(B) stream is compatible with a correct final answer, and a clean (C) chain is not required for the model to be correct.

### F.2 Details for Multi-Path Analysis ( Section 5.2 )

##### Datasets.

Note that GSM8K-Aug [ Deng et al., 2023 ] augments the original GSM8K [ Cobbe et al., 2021 ] training split (about 7.5 7.5 K problems) into the 385,620 385{,}620 training examples we use. GSM8K is thus a subset of GSM8K-Aug, and the multi-path bank below is built from the original GSM8K problems, which are also in the training set.

##### Path bank.

For each question we identify its trained path (its gold chain in the training data) and an unseen-but-valid alternative drawn from a bank of up to ten correct paths obtained by rejection-sampling Llama-3.1-8B [ Grattafiori et al., 2024 ] on the original GSM8K training questions ( 7,444 7{,}444 questions with 58,155 58{,}155 verified paths in total), with each alternative verified absent from training.

##### Candidate sets and filtering.