Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs — part 4/5
Cleaned-text code points [45771, 61695). Bibliography omitted here; source.md includes it.
Thinking-mode runs use Qwen3’s recommended sampling with a 32,768 32{,}768 -token budget ( 38,912 38{,}912 for Sudoku), and Countdown-4, Sudoku, and MBPP use fixed random test subsets ( n = 300 / 200 / 300 n{=}300/200/300 ) to bound the compute spent. Two entries in Table 2 carry footnotes. The Sudoku result ( 0.5 0.5 ) is not a truncation artifact: only 8 8 of 200 200 generations were cut off before emitting an answer, so thinking-mode accuracy is bounded at ≤ 4.5 % {\leq}4.5\% even if all eight would have been correct; the remaining generations produced complete but constraint-violating grids. The MBPP result is name-normalized, because the bare MBPP prompt does not pin the required function name and raw extraction therefore under-scores thinking mode.
#### Adapting TRM to Countdown-4.
Countdown-4 is a sequence-to-sequence task in which x x (four input integers and a target) and y y (a sequence of arithmetic expressions) occupy different spaces. This differs from Sudoku, where TRM operates on the input grid directly and the output shares the input’s shape. To adapt TRM we construct a padded sequence [ x , PAD ] [x,\text{PAD}] , so that the model attends to the x x tokens through bidirectional attention and writes its answer into the pad positions. This is a natural but not necessarily optimal construction, and TRM’s low Countdown-4 score should be read with that caveat (§ 4.3 ); a natively sequential TRM is left to future work.
## Appendix C Parameter-Efficient Baselines and Data Efficiency
#### Do soft prompts alone explain the gain?
A natural alternative hypothesis is that LRT’s improvement comes from task supervision plus trainable parameters in the decoder’s input, rather than from instance-conditioned latent computation. Table 7 tests this directly on Countdown-4 with two parameter-efficient prompt-adaptation methods trained on the same data at a matched ≈ 11 {\approx}11 M trainable budget. Prefix-tuning ( Li and Liang, 2021 ) reaches 12.5 12.5 and P-tuning v2 ( Liu et al., 2022 ) reaches 28.4 28.4 – the latter essentially matching the no-latent Direct baseline ( 27.8 27.8 ). Both are instance-independent : they learn one prompt shared by every problem. Making the injected vectors instance-conditioned adds + 13.6 +13.6 (task-dedicated proposer alone, 42.0 42.0 ), and refining them recurrently adds a further + 14.7 +14.7 ( 56.7 56.7 ). The gain therefore requires conditioning on the instance and computing over that conditioning, not merely spending parameters at the input.
We report Countdown-4 only, as it is the benchmark that separates the methods most sharply; extending these controls to the remaining four benchmarks is straightforward but was not run.
#### Is the gain bought with 100k synthetic examples?
The symbolic training sets contain 100,000 100{,}000 generated instances, which raises the question of whether LRT’s symbolic result is a data-scale artifact. Table 8 varies the Countdown-4 training-set size while holding everything else fixed. The curve is close to flat: 52.7 52.7 at 1,000 1{,}000 examples against 56.7 56.7 at 100,000 100{,}000 – two orders of magnitude more data for 4.0 4.0 points. (The small non-monotonicity between 1 1 k and 5 5 k is within seed noise.) Two comparisons make the point sharper. First, SoftCoT and EBM-CoT are trained on the same 100,000 100{,}000 examples and reach 5.9 5.9 and 8.4 8.4 : the data is available to them and does not help. Second, LRT at 1,000 1{,}000 examples ( 52.7 52.7 ) already exceeds every other frozen-decoder method trained on 100,000 100{,}000 . The natural-language benchmarks are low-resource to begin with – MBPP has 374 374 training examples and StrategyQA 2,061 2{,}061 (Table 6 ) – and LRT is trained on those splits as given.
## Appendix D Refiner Design Ablations
Table 9 ablates the refiner’s design; each run holds the proposer, decoder, and training recipe fixed at the Table 1 setting and changes one component. Replacing the residual parameterization L ⋆ = L ( 0 ) + Δ L^{\star}{=}L^{(0)}{+}\Delta with direct regeneration of L ⋆ L^{\star} removes the anchor to the instance-conditioned proposal and costs 3.9 3.9 average points. Injecting the proposed base u u only at refiner initialization – rather than at every fast update – costs 3.2 3.2 points, as the fast state drifts away from the problem. Collapsing the two-timescale state ( z L , z H ) (z_{L},z_{H}) to a single state costs 2.6 2.6 points. Full backpropagation through all 45 45 passes, rather than truncating to the final cycle, leaves accuracy essentially unchanged ( − 0.1 -0.1 ) while raising training memory ≈ 6 × {\approx}6\times and step time ≈ 3 × {\approx}3\times relative to the truncated variant; truncated unrolling is therefore the better operating point.
Table 10 sweeps the residual penalty λ \lambda of § 3.5 . With λ = 0 \lambda{=}0 the residual Δ \Delta grows large and the refined latents drift toward a generic, instance-agnostic point; with λ = 1.0 \lambda{=}1.0 the penalty suppresses Δ \Delta almost entirely and LRT collapses toward the proposer-only result. The value λ = 0.01 \lambda{=}0.01 used throughout sits at the optimum.
## Appendix E Scaling and Depth
#### Test-time depth extrapolation.
Table 11 takes the refiner trained at S = H = 3 S{=}H{=}3 ( 9 9 cycles) and unrolls it for more cycles at inference only . Accuracy continues to improve, consistent with the learned transition approximating a convergent process; we did not train a refiner at those depths for comparison.
#### Non-recurrent control.
Table 12 replaces the recurrent refiner with a non-weight-tied feed-forward refiner of equal ( 7 7 M) parameter count. At matched capacity it trails the recurrent refiner by 6.1 6.1 average points, isolating recurrence – not parameters – as the source of the gain.
#### Latent count and width.
Table 13 sweeps the inference latent count K infer K_{\text{infer}} (with K = 32 K{=}32 fixed in training) and the working width d ′ d^{\prime} . K infer = 4 K_{\text{infer}}{=}4 is best, confirming on our setup the train/inference asymmetry adopted from EBM-CoT; accuracy saturates at d ′ = 256 d^{\prime}{=}256 , with d ′ = 512 d^{\prime}{=}512 adding refiner parameters for a negligible gain.
#### Why K = 32 K{=}32 in training but K infer = 4 K_{\text{infer}}{=}4 at test?
We characterize this asymmetry from both directions. The training count matters and the inference count does not, much. Training with K = 4 K{=}4 instead of 32 32 – matching train and test – collapses Countdown-4 to 9.5 9.5 . Inspection shows well-formed but incorrect expressions rather than truncated output, so this is a capacity failure, not a decoding artifact: the 32 32 query positions act as scratch space inside the proposer’s bidirectional pass, shaping the four that are read out even though the rest are discarded. In the other direction, keeping more latents at inference degrades the five-benchmark average mildly and monotonically – 54.1 54.1 , 53.8 53.8 , 52.9 52.9 , 51.8 51.8 at K infer = 4 , 8 , 16 , 32 K_{\text{infer}}{=}4,8,16,32 (Table 13 ) – and every setting stays far above SoftCoT ( 29.5 29.5 ) and EBM-CoT ( 33.5 33.5 ). The effect is therefore robust rather than a knife-edge tuned in LRT’s favour. The protocol is inherited, not chosen post hoc : the train/inference asymmetry and the value 4 4 come from EBM-CoT, whose latent-count ablation peaks near four, and we apply the identical protocol to SoftCoT, EBM-CoT, and LRT.
## Appendix F Training Protocol Ablations
Table 14 ablates the training protocol. Training the proposer and refiner jointly , rather than in the two stages of § 3.5 , costs 3.1 3.1 average points and is less stable, as the refiner adapts to a proposer that is still moving. Leaving the proposer unfrozen during Stage 2 changes little ( − 0.5 -0.5 ), so freezing it – which lets the base latents be cached – is essentially free. Removing the task instruction I I at inference, leaving the decoder to work from L ⋆ L^{\star} alone, costs 3.4 3.4 points: the latents carry instance-specific computation but not the decoder’s instruction-following prior.
## Appendix G Mechanistic Analyses
#### Refinement trajectory.
Figure 3 (a) plots the accuracy of the answer decoded from the latents after each high-level cycle. Accuracy rises smoothly from the proposed base L ( 0 ) L^{(0)} to the final L ⋆ L^{\star} rather than jumping at the last step, so intermediate latents are partially correct and the refiner improves them incrementally.
#### Latent convergence.
Figure 3 (b) tracks the residual ∥ L ( t ) − L ( t − 1 ) ∥ \lVert L^{(t)}-L^{(t-1)}\rVert , which contracts geometrically toward a small value: the unrolled refiner approaches a fixed point. This is the behavior expected when recurrent refinement is viewed as relaxation toward a task-conditioned attractor ( Bai et al., 2019 ; Geiping et al., 2025 ; Huang et al., 2026 ) .
#### What the latents encode.
Decoding each refined latent to its nearest decoder token embedding yields fragments aligned with intermediate reasoning – partial arithmetic for Countdown-4, candidate digits for Sudoku – rather than fluent text, consistent with L ⋆ L^{\star} carrying computation in a form the decoder consumes but does not verbalize.
#### Linear probe on the refined latents.
To test more directly whether the answer is already present in L ⋆ L^{\star} and the decoder merely transcribes it (§ 4.6 ), we cache the refined latents for the Countdown-4 test set and fit a logistic-regression probe on the flattened K infer × d K_{\text{infer}}{\times}d representation to predict a binary label: whether the frozen decoder subsequently solves the instance. The probe is fit on an 80 / 20 80/20 train/test split of the cached latents, with ℓ 2 \ell_{2} regularization selected by cross-validation on the training portion, and evaluated on the held-out 20 % 20\% . It reaches 57.5 % 57.5\% accuracy against a 52.5 % 52.5\% majority-class baseline; the same probe on the unrefined L ( 0 ) L^{(0)} is at chance. Refinement therefore writes some outcome-relevant information into the latent, but only weakly linearly readable information – and predicting whether the instance will be solved is a far easier target than the answer string itself. Because Countdown-4 test targets are held out (Appendix A ), the answer also cannot have been memorized into the latent from training. We report this as bounding evidence, not proof: a linear probe measures linear decodability only, and a non-linear reader might extract more.
#### Why generic latents collapse.
Table 15 reports the frozen decoder’s negative log-likelihood of the reference answer on Countdown-4 under different latent sources. Latents from a generic assistant proposer raise NLL above the no-latent baseline – they actively mislead the decoder – whereas task-dedicated and refined latents lower it. The mean cosine similarity of each latent to its nearest decoder input embedding shows why: generic latents land far off the decoder’s input manifold, while task-dedicated and refined latents stay close to it.
## Appendix H Generality Across Decoders
Table 16 swaps the frozen decoder. LRT remains far above the frozen-decoder baselines of Table 1 for every backbone, and a stronger decoder yields a stronger LRT, consistent with the division of labor in which the decoder supplies the sequence prior and the trained modules supply iterative computation.
## Appendix I Compute and Efficiency
Table 17 compares inference cost. LRT injects only K infer = 4 K_{\text{infer}}{=}4 latent vectors and decodes the answer directly, against the ≈ 210 {\approx}210 chain-of-thought tokens zero-shot CoT must generate; the 45 45 refiner passes act on a 7 7 M-parameter block and add little next to a single forward pass of the 8 8 B decoder. Net latency is below zero-shot CoT despite the added recursion.
#### Training cost.
Table 18 reports the training side, which the inference table above does not cover. Both stages run for 30 30 epochs at batch size 64 64 on a single 96 96 GB GPU. The dominant cost is the forward pass through the frozen 8 8 B decoder, not the 11 11 M trainable modules; because Stage 2 freezes the proposer, the base latents L ( 0 ) L^{(0)} are precomputed once and cached, so Stage 2 skips the proposer’s forward pass entirely. Truncated-gradient unrolling is what keeps the deep recursion affordable: full backpropagation through all 45 45 passes raises peak training memory ≈ 6 × {\approx}6\times and step time ≈ 3 × {\approx}3\times for a 0.1 0.1 -point accuracy difference (Appendix D ), so truncation is strictly the better operating point rather than an accuracy compromise.
## Appendix J Potential Risks
As with any research involving large language models, our work carries a risk of misuse: the techniques we present could, in principle, be adapted to generate malicious code or other harmful outputs. This concern is heightened for a project focused specifically on enhancing the reasoning capabilities of LLMs, since stronger reasoning may also amplify a model’s capacity to produce sophisticated harmful content.
## Appendix K Licenses, Parameters, and Intended Use of Artifacts
This work uses several existing scientific artifacts – pretrained language models, baseline code, software packages, and evaluation datasets – and produces new artifacts of its own. We record below the license or terms of use of each artifact, the implementation and parameter settings of the packages we rely on, and confirm that our use is consistent with the intended use under which each artifact was released. We also state the intended use of the artifacts we create and its compatibility with the original access conditions.
### K.1 Existing Artifacts
#### Pretrained models.
The frozen decoders are used under their respective open-weight licenses. Qwen3-8B and Qwen3-4B ( Yang et al., 2025 ) are released under the Apache License 2.0, and Llama-3.1-8B is used under the Llama 3.1 Community License Agreement. The TRM recurrent reasoner ( Jolicoeur-Martineau, 2025 ) , which we repurpose as the LRT refiner, is released by its authors under the MIT License. All of these licenses permit use, reproduction, modification, and redistribution for research purposes. These models were released as general-purpose research artifacts for studying and building language and reasoning systems, and our use of them – as a frozen decoder and a refiner module within an academic study of latent reasoning – is consistent with that stated intended use.
#### Baseline code.
We reproduce SoftCoT ( Xu et al., 2025b ) from the authors’ publicly released implementation, which is distributed under the NTUITIVE Dual License Agreement (a non-commercial license that permits use, reproduction, modification, and distribution of derivative works for academic research purposes). Our use is strictly for non-commercial academic research and is therefore consistent with these terms. EBM-CoT ( Chen et al., 2025b ) has no public implementation; our results come from an independent reimplementation written by us from the method description, and we release it under the same license as our own code.
#### Evaluation datasets.
HumanEval ( Chen et al., 2021 ) is released under the MIT License, and MBPP ( Austin et al., 2021 ) under the Creative Commons Attribution 4.0 International License (CC BY 4.0). StrategyQA ( Geva et al., 2021 ) is distributed for research use under the terms accompanying its public release; we use only its public split. Each of these datasets was created and released as a benchmark for evaluating the reasoning ability of NLP systems, and we use them solely for benchmark evaluation in an academic study – the use for which they were intended and distributed. We use each dataset through its standard public split and do not redistribute the underlying data.
#### Generated data.