Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers — part 6/6

Cleaned-text code points [77395, 82279). Bibliography omitted here; source.md includes it.

We partition intermediate numbers into three disjoint sets: G G (trained-path only), U U (unseen-path only), and a random control from other questions matched to U U in both cardinality and single/multi-token composition. Numbers shared by both paths or appearing in the question or answer are excluded, since they cannot indicate which chain a readout recovered. The random control is fixed and shared across all four readouts and verified disjoint from the trained path, unseen path, question, and answer numbers. Intersecting the bank with our training set and keeping questions with ≥ 2 \geq 2 disjoint intermediates per side yields 653 653 candidate questions. We then apply a token-clean filter, requiring every only- U U number to share no sub-token with any trained-path, question, or answer number, leaving the final 341 341 questions, spanning 871 871 G G , 974 974 U U , and 974 974 Random numbers. Random stays matched to U U in cardinality and composition under every criterion.

##### Metrics.

We score a target number by its best-position, per-token NLL. A number spans one or more consecutive sub-tokens, and at every latent position the readout gives a softmax distribution over the vocabulary. We slide the number’s sub-tokens across the latent positions. At each placement we average the sub-tokens’ NLLs at their aligned positions, and keep the lowest such per-token average over all placements (for a single-token number, simply the negative log of its highest readout probability across positions). Per-token normalization puts single and multi-token numbers on the same scale, and scoring only the most probable span avoids double counting overlaps. We average this over the numbers in sets G G , U U , and Random within each question, then over the 341 341 questions. The P ⁡ ( U ∈ top- ​ k ) P(U\in\text{top-}k) columns instead report a hit rate: the fraction of U U numbers recovered within the top- k k candidates (top- k k membership at any position for single-token numbers, and a consecutive top- k k match for multi-token ones), counting each number once however many positions it surfaces at, averaged within each question and then over the 341 341 questions.

### F.3 Multi-Path Readout Example

This example (referenced from Section 5.2 ) shows a question that admits two valid reasoning chains with the same answer 108 108 , read out by LOTUS ( 𝒉 ( R ) {{{\bm{h}}}^{({R})}} through the base LM head f head {f_{\mathrm{head}}} , i.e. the first row of Table 10 ). Q: The bus driver drives an average of 2 hours each day, 5 days a week. From Monday to Wednesday he drove at an average speed of 12 kilometers per hour, and from Thursday to Friday at an average speed of 9 kilometers per hour. How many kilometers did the driver travel during these 5 days? Trained path G G : 3*2=6 -> 6*12=72 -> 2*2=4 -> 4*9=36 -> 72+36=108 . Unseen path U U : 2*12=24 -> 24+24+24=72 -> 2*9=18 -> 18+18=36 -> 72+36=108 . only G = { 3 , 4 , 6 } \text{only}_{G}=\{3,4,6\} , only U = { 24 , 18 } \text{only}_{U}=\{24,18\} (question and final answer excluded). Figure 4 visualizes the same example, aligning the trained path, the unseen valid path, and the latent readout positions where the only- U U intermediates surface. The only- U U intermediates 24 24 and 18 18 are the per-day distances in the alternative decomposition, which forms daily distances before summing rather than aggregating hours first. Although neither appears in the trained reference path, both surface in the post-loop latent readout through the base LM head: P ⁡ ( 24 ) = 0.77 P(\texttt{24}){=}0.77 (top- 1 1 , position 142 142 ) and P ⁡ ( 18 ) = 0.14 P(\texttt{18}){=}0.14 (top- 2 2 , position 116 116 ). Every number here is a single token, so no only- U U number shares a sub-token with any trained-path, question, or answer number, confirming the surfacing is genuine unseen-chain content rather than a digit-overlap artifact.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .