Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs — part 3/5

Cleaned-text code points [30178, 45771). Bibliography omitted here; source.md includes it.

MGDM and TRM solve Sudoku almost perfectly ( 96.9 96.9 , 97.2 97.2 ) and EqR essentially completely ( 99.8 99.8 ), far above LRT’s 49.2 49.2 . We state this plainly: on Sudoku a specialized solver is much better than LRT, and the frozen-decoder route trades peak symbolic accuracy for one method that also handles language. Two observations qualify the comparison rather than rescue it. First, these models do not transfer: trained from scratch for symbolic decoding, they have no language interface at all. Second, their competence is uneven: TRM, the very reasoner LRT adopts as its refiner, scores 1.2 1.2 on Countdown-4 despite near-solving Sudoku. We attribute this partly to task structure: Countdown-4 admits an unusually sequential solution, a chain of arithmetic operations applied in order, which a from-scratch model with no sequence-modeling prior struggles to capture, whereas Sudoku suits TRM’s grid recurrence. The honest caveat is that TRM does not natively support sequence-to-sequence tasks; our adaptation (Appendix B ) is simple and stronger ones may exist. The central point survives: without some sequential prior, TRM struggles on a fundamentally sequential task, and LRT is built around exactly this division of labor.

#### Parameter-efficient prompt adaptation.

At a trainable budget matched to LRT’s ≈ 11 {\approx}11 M, prefix-tuning reaches 12.5 12.5 and P-tuning v2 28.4 28.4 on Countdown-4 (Appendix C ), the latter merely matching the no-latent Direct baseline ( 27.8 27.8 ): task supervision plus trainable soft prompts do not by themselves produce the gain. Making the injected vectors instance-conditioned adds + 13.6 +13.6 (to 42.0 42.0 ), and recurrent refinement a further + 14.7 +14.7 (to 56.7 56.7 ).

### 4.4 Scaling the Recurrent Refiner

In LRT the decoder is frozen, so all learned computation lives in the proposer and the refiner – and of the two, the refiner is where the iterative reasoning happens, making its capacity a natural lever. Table 1 uses a 7 7 M refiner, chosen for parity with EBM-CoT’s energy module (§ 3.3 ); we now enlarge it while everything else is held fixed .

Figure 2 scales the TRM refiner across 7 7 M, 14 14 M, and 28 28 M parameters and measures Countdown-4 and HumanEval. Both improve monotonically: Countdown-4 rises from 56.7 56.7 to 66.3 66.3 to 69.1 69.1 , and HumanEval from 37.8 37.8 to 44.0 44.0 to 46.1 46.1 . The gains are largest at the first doubling ( + 9.6 +9.6 and + 6.2 +6.2 points) and taper at 28 28 M ( + 2.8 +2.8 and + 2.1 +2.1 ). Crucially, the improvement is bought entirely with refiner capacity: the 8 8 B decoder never changes. Reasoning quality thus scales with the recurrent reasoner that carries the computation, separately from the LLM that supplies sequence modeling, and the 7 7 M setting used elsewhere is a conservative, parameter-matched operating point with clear headroom.

### 4.5 Mechanistic Ablations

The comparison in § 4.2 varies two things at once: relative to LRT, SoftCoT and EBM-CoT each swap both the proposer (task-dedicated → \rightarrow generic) and the refiner. We now isolate the two components.

#### Factorial decomposition.

LRT pairs a proposer and a refiner (§ 3.2 , § 3.3 ), so the natural ablation is the 2 × 3 2{\times}3 grid over proposer ∈ { \in\{ generic assistant, task-dedicated } \} and refiner ∈ { \in\{ none, energy, recurrent } \} . SoftCoT, EBM-CoT, and LRT are three of its six cells; Table 3 fills the rest, and two of the three new cells correspond to alternative pairings a reader might expect: task-dedicated + none is SoftCoT with our proposer substituted, and task-dedicated + energy is EBM-CoT with our proposer substituted. All six cells are reported. Holding the refiner at none , swapping the generic assistant for a task-dedicated proposer lifts the average from 12.3 12.3 to 37.2 37.2 ; holding the proposer task-dedicated, the recurrent refiner contributes roughly three times the energy refiner’s gain ( + 10.7 +10.7 vs. + 3.4 +3.4 ). The two also interact – the recurrent refiner adds + 10.7 +10.7 on a task-dedicated proposer but only + 9.3 +9.3 on a generic one, and on Countdown-4 every generic-proposer cell stays below the Direct baseline ( 27.8 27.8 ) whatever the refiner. Neither component alone accounts for LRT: the proposer must place the base latents on a usable manifold before the refiner can compute over them.

#### Depth versus capacity.

§ 4.4 scaled the refiner by parameters ; we now vary depth at fixed parameters (Appendix E ). A non-recurrent refiner with the same 7 7 M budget but a single feed-forward pass reaches only 47.5 / 33.0 47.5/33.0 on Countdown-4/HumanEval against LRT’s 56.7 / 37.8 56.7/37.8 ; and a refiner trained at S = H = 3 S{=}H{=}3 but unrolled deeper at test time only keeps improving, indicating a convergent iterative process rather than a fixed-length circuit.

#### Refinement is incremental.

Decoding the answer after each high-level cycle (Appendix G ) shows accuracy climbing monotonically across the unroll rather than appearing only at the final step, while the residual ∥ L ( t ) − L ( t − 1 ) ∥ \lVert L^{(t)}-L^{(t-1)}\rVert contracts toward a fixed point: the refiner performs incremental, convergent computation, not one-shot calibration. Appendix D confirms each design choice earns its place – regenerating L ⋆ L^{\star} instead of the residual Δ \Delta , or injecting the base only at initialization, each costs 3 3 – 4 4 average points.

### 4.6 Where Does the Computation Happen?

LRT combines a recurrent reasoner with a frozen LLM, inviting a natural worry: is the gain simply TRM, with the LLM reduced to a transcription device? Four measurements bound the answer from both sides.

TRM alone does not do the work. As a standalone solver it scores 1.2 1.2 on Countdown-4 against LRT’s 56.7 56.7 and cannot address the natural-language benchmarks at all (Table 2 ): on four of five benchmarks TRM alone is not merely worse but inapplicable, Sudoku being the exception noted in § 4.3 . The refiner needs on-manifold latents. The same refiner applied to latents from a generic proposer reaches only 14.8 14.8 on Countdown-4 (Table 3 ), below the no-latent Direct baseline of 27.8 27.8 : recurrence is not a free-standing engine, and anchoring it in the decoder’s representation space is what makes it useful. The decoder is load-bearing. Swapping it changes results meaningfully – 54.1 54.1 , 51.8 51.8 , 49.0 49.0 average for Qwen3-8B, Llama-3.1-8B, Qwen3-4B (Appendix H ) – and removing the instruction I I at inference costs 3.4 3.4 points (Appendix F ); neither should matter much if the decoder merely transcribed a finished answer, since any of these backbones can verbalize one. The answer is not sitting in the latent. A linear probe on cached L ⋆ L^{\star} predicting whether the decoder will solve the instance reaches 57.5 % 57.5\% against a 52.5 % 52.5\% majority baseline, with unrefined latents at chance (Appendix G ); even the outcome, let alone the answer string, is only weakly linearly readable. Nor can the answer have been memorized: Countdown-4 evaluation holds out targets.

This evidence is convergent but correlational: a linear probe bounds only linearly decodable information, and the decoder-swap argument presumes comparable verbalization ability across backbones. We claim the computation is distributed across refiner and decoder rather than concentrated in either, not that we have measured the split.

## 5 Conclusion

We presented Latent Recurrent Thoughts (LRT), which reasons with a frozen LLM by pairing a task-dedicated proposer with a TRM-based recurrent refiner, training fewer than 0.2 % 0.2\% of the parameters involved: the proposer maps a problem to base latents the frozen decoder can use, and the refiner computes over them, emitting a bounded residual correction. Across symbolic and natural-language reasoning LRT improves substantially over prior frozen-decoder latent-injection methods under an identical decoder, prompt, data, and budget, and over budget-matched zero-shot CoT; neither component suffices alone, and the computation is distributed across refiner and decoder rather than carried by either (§ 4.6 ). The result is scoped – modules are trained per task family, and large test-time budgets remain stronger on some tasks – but within it, a small recurrent module in front of a frozen LLM supplies much of the computation chain-of-thought otherwise externalizes.

## Limitations

#### Per-task modules, and what that costs the claim.

LRT keeps the decoder frozen but trains a proposer and a refiner against it, per task family. It is therefore not a single general model applied zero-shot: each new task requires training new modules, and the method is best described as per-task-trained latent augmentation of a frozen decoder rather than a general reasoning method. This bounds the fair comparison class: the controlled comparison in Table 1 is against other trained frozen-decoder latent modules, and comparisons to prompting are reported at matched inference budget, with thinking-mode prompting reported separately as an unmatched reference point (§ 4.3 ). Training additionally relies on answer supervision and, for the symbolic benchmarks, on the ability to generate synthetic training sets – though Appendix C shows 1,000 1{,}000 Countdown-4 examples already reach 52.7 52.7 , so the requirement is for some answer-labeled data rather than a great deal of it. Tasks where even that is unavailable remain out of reach.

#### Generalization is within-task, not across tasks.

Our Countdown-4 evaluation holds out targets, so the reported accuracy does reflect systematic generalization to unseen instances rather than memorization. But we have not tested transfer across tasks or difficulty levels: whether modules trained on Countdown-4 help on harder Countdown variants, or Sudoku modules on Kakuro, is untested and we make no claim about it. The per-task protocol matches SoftCoT and EBM-CoT, which makes the comparison controlled but leaves cross-task generality an open question for all three.

#### Scope of evaluation.

Our experiments use a single 8 8 B-parameter decoder and five benchmarks. Appendix H varies the decoder, but all backbones are of comparable scale; whether the division of labor between a frozen decoder and a small recurrent reasoner holds at substantially larger scales, or across broader task distributions, remains open. The parameter-efficient controls of Appendix C were run on Countdown-4 only.

#### Method caveats.

On Sudoku, symbolic-specialized solvers remain far ahead of LRT (Table 2 ); the frozen-decoder route trades peak symbolic accuracy for a single method that also handles natural language. The refiner’s recurrent unroll adds inference compute that grows with depth (Appendices E and I ). Our mechanistic analyses (§ 4.5 , § 4.6 , Appendix G ) are correlational: the monotone intermediate-decoding curve, the contracting latent residual, and the weakly-above-baseline linear probe are evidence about where computation happens, not proof, and the probe in particular bounds only linear decodability.

## Appendix A Implementation and Training Details

#### Architecture.

The proposer is a bidirectional encoder: an input projection P ↓ \mathrm{P}_{\!\downarrow} maps decoder embeddings of x x to working width d ′ = 256 d^{\prime}{=}256 , two pre-norm transformer blocks attend over K = 32 K{=}32 learned query vectors appended to the sequence, and an output projection P ↑ \mathrm{P}_{\!\uparrow} returns the K K base latents to decoder width d d . The refiner reuses one transition block of the same class (with independent weights) as f f , together with projections P ↓ ′ , P ↑ ′ \mathrm{P}^{\prime}_{\!\downarrow},\mathrm{P}^{\prime}_{\!\uparrow} and learned initial buffers z L 0 , z H 0 z_{L}^{0},z_{H}^{0} . Table 4 breaks down the trainable parameters; the Qwen3-8B decoder is frozen throughout, so LRT trains 11.2 11.2 M parameters, about 0.14 % 0.14\% of the model it reasons with.

#### Optimization.

Both stages use AdamW ( β 1 = 0.9 \beta_{1}{=}0.9 , β 2 = 0.999 \beta_{2}{=}0.999 , ϵ = 10 − 8 \epsilon{=}10^{-8} ) with a cosine learning-rate schedule, 5 % 5\% linear warmup, weight decay 0.01 0.01 , and gradient clipping at 1.0 1.0 , in bfloat16 . Stage 1 trains the proposer at peak learning rate 3 × 10 − 4 3{\times}10^{-4} ; Stage 2 trains the refiner at 2 × 10 − 4 2{\times}10^{-4} . Both run for 30 30 epochs at batch size 64 64 on a single 96 96 GB GPU, with gradient checkpointing. Table 5 collects the hyperparameters.

#### Datasets.

Table 6 lists sizes and splits. Countdown-4 instances are generated following Gandhi et al. (2024) and Sudoku puzzles following Ye et al. (2025) ; HumanEval, MBPP, and StrategyQA use their public splits. Countdown-4 evaluation holds out targets : no target value in the test set appears in training, so the reported accuracy reflects generalization to unseen targets rather than fitting the synthetic distribution.

#### What the supervision consists of.

The claim that LRT trains without reasoning traces is scoped to the symbolic benchmarks, and we state the full picture here. Countdown-4 and Sudoku are supervised strictly by the final answer; the search that produces it is never recorded and never used. StrategyQA is the one exception: in the main results its target text is the answer together with the dataset’s reference rationale, following prior work. Evaluation is unaffected – StrategyQA is scored by exact match on the normalized yes/no answer only, never on the rationale. To remove any ambiguity, we also trained StrategyQA with answer-only supervision: it reaches 76.9 76.9 , against 75.1 ± 2.3 75.1\pm 2.3 for the rationale-supervised setting reported in Table 1 . The rationale is therefore not necessary for LRT’s result, and no benchmark in this paper depends on trace supervision.

#### Inference.

At test time the proposer emits its K = 32 K{=}32 query latents and the first K infer = 4 K_{\text{infer}}{=}4 are kept (Table 13 ); the refiner runs the full S × H S{\times}H recursion with no stop-gradient, and the decoder greedy-decodes the answer from [ I ; x ; L ⋆ ] [I;x;L^{\star}] .

## Appendix B Reproduction and Harness Notes

#### Baseline reproduction.

SoftCoT is reproduced from the authors’ released code; EBM-CoT, which has no public implementation, is reproduced from the description of Chen et al. (2025b) . Both use the same frozen Qwen3-8B decoder, the same assistant model for their generic proposer, and the same stage-wise training budget as LRT, so differences reflect the latent module rather than the backbone or compute. To check the faithfulness of the EBM-CoT reproduction we compare on StrategyQA, the one benchmark shared with the original paper: 70.96 70.96 reported by Chen et al. (2025b) against 71.04 71.04 for our reproduction. Reproduced numbers may nonetheless deviate from the authors’ intended setup on the benchmarks they did not report.

#### Thinking versus non-thinking mode.

Qwen3 exposes a “thinking” mode that emits a long internal trace before answering. All rows of Table 1 use non-thinking mode so that the prompted baselines and the latent methods spend comparable inference compute; this departs substantially from the official technical-report numbers, especially on the coding tasks. To verify that the deviation reflects the mode rather than a harness bug, we re-ran with thinking mode enabled and reproduced the official MBPP figure. Those runs appear as reference points in Table 2 and are discussed in § 4.3 .