Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs — part 1/5
Cleaned-text code points [0, 15170). Bibliography omitted here; source.md includes it.
# Latent Recurrent Thoughts: Recurrent Refinement of Proposed Latents for Reasoning with Frozen LLMs
###### Abstract
Chain-of-thought reasoning unfolds in discrete token space: each step is committed as text, errors propagate, and eliciting good traces presupposes traces to imitate. Reasoning instead in a model’s continuous representation space – where intermediate states are vectors rather than words – sidesteps these constraints, but leaves open how those latent states should be computed. We approach this along two axes. First, we keep a large language model (LLM) frozen and use it for what it is already good at – modeling and decoding sequences – while a small auxiliary network supplies continuous latent thoughts as input. Second, we produce those latents by recurrence : a tiny recurrent reasoner refines them over many steps, decoupling the depth of computation from the size of the model, so that the latents are a product of iterative processing rather than a single forward pass. We instantiate this as Latent Recurrent Thoughts (LRT) 1 1 1 Code is available at https://github.com/czl-david/latent-recurrent-thoughts . : a task-dedicated proposer supplies base latents, a recurrent reasoner refines them through bounded residual corrections, and the frozen LLM decodes the answer. On symbolic reasoning with answer supervision but no reasoning traces (Countdown-4, Sudoku) and on natural-language reasoning (HumanEval, MBPP, StrategyQA), LRT substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.
## 1 Introduction
Large language models (LLMs) solve complex problems by externalizing intermediate steps as chain-of-thought (CoT) traces ( Wei et al., 2022 ; Kojima et al., 2022 ; Nye et al., 2021 ) , and recent systems push this further with sampling, search, and reinforcement learning ( Wang et al., 2023 ; Yao et al., 2023 ; Besta et al., 2024 ; Snell et al., 2025 ; Guo et al., 2025 ) . But CoT reasons in the discrete token space: each step must be verbalized from a fixed vocabulary, generation is autoregressive and error-propagating, and – most limiting – eliciting good CoT presupposes reasoning traces to imitate ( Zelikman et al., 2022 ; Brown et al., 2020 ) . This motivates reasoning that instead operates in a model’s continuous representation space.
Such continuous-space reasoning feeds latent vectors back into the model rather than decoded tokens ( Hao et al., 2025 ; Cheng and Van Durme, 2024 ; Su et al., 2025 ; Shen et al., 2025 ) , and existing methods realize it in many ways. One practical path – attractive when fine-tuning a capable LLM is too costly or risks catastrophic forgetting – keeps the decoder frozen : a small auxiliary model produces instance-conditioned latents that are injected into its input as soft tokens – the interface used by prefix and prompt tuning ( Li and Liang, 2021 ; Lester et al., 2021 ) . SoftCoT ( Xu et al., 2025b ) and EBM-CoT ( Chen et al., 2025b ) take this route, and so does our method.
Within this frozen-decoder approach, two components govern quality: the proposer that supplies the latents and the refiner that improves them. SoftCoT proposes with a frozen generic language assistant and performs no refinement at all; EBM-CoT keeps that proposer and adds a refiner that descends a learned scalar energy field, nudging latents toward lower-energy, “consistent” regions. We find both fragile. Off the natural-language substrate, a generic assistant’s proposed latents can be worse than no latents – they actively mislead the decoder – whereas a small proposer dedicated to the task family helps. And an energy refiner only follows the gradient of a fixed scalar field: it calibrates latents rather than computing with them, too shallow for multi-step search.
We introduce Latent Recurrent Thoughts (LRT) , which strengthens both links. A task-dedicated proposer, conditioned on the input x x , supplies base latents adapted to the task family; a recurrent reasoner then refines them – we use TRM ( Jolicoeur-Martineau, 2025 ) , a tiny recursive network that iteratively updates a latent state. Unlike an energy model, a recurrent reasoner applies a learned vector-valued transition that can implement constraint propagation and iterative correction, and can be unrolled to greater depth at no extra parameter cost. It predicts bounded residual corrections rather than regenerating the latent, keeping refinement anchored to the proposed base. Such recurrent reasoners – HRM, TRM, GRAM, EqR ( Wang et al., 2025 ; Jolicoeur-Martineau, 2025 ; Baek et al., 2026 ; Huang et al., 2026 ) – have proven to be powerful and parameter-efficient computation engines, but they have so far been built only as standalone solvers trained from scratch on a single task. LRT instead pairs one with a frozen, pretrained LLM: the recurrent reasoner carries the iterative computation, while the LLM contributes the broad sequence-modeling and language ability that a from-scratch solver lacks. To our knowledge, this pairing is new.
SoftCoT, EBM-CoT, and LRT all inject latents through the identical interface into the same frozen LLM (Qwen3-8B; Yang et al., 2025 ), so the proposer and refiner are the only variables and comparisons are controlled. We evaluate on two substrates. Symbolic tasks – Countdown-4 and Sudoku – have answer supervision but no reasoning traces : the search that solves an instance is never recorded, so it cannot be imitated and must instead be carried out as latent computation. Natural-language tasks – HumanEval, MBPP, and StrategyQA – are the home turf of soft-CoT methods. The split is stark. On the natural-language tasks SoftCoT and EBM-CoT improve modestly over a zero-shot CoT baseline and LRT improves further; but on Countdown-4 both baselines collapse – to 5.9 % 5.9\% and 8.4 % 8.4\% , far below the 30.0 % 30.0\% of zero-shot CoT – whereas LRT reaches 56.7 % 56.7\% . Off the natural-language substrate a generic proposer is not merely suboptimal but actively destructive, and only a task-dedicated proposer paired with a recurrent refiner converts latent injection back into a consistent gain.
Two scoping notes frame these numbers, and we state them here rather than in the limitations alone. First, LRT trains its proposer and refiner per task family : it is a per-task-trained latent augmentation of a frozen decoder, not a single zero-shot general model, so its comparison class is other trained frozen-decoder modules at a matched budget. Second, the CoT baselines are prompted and training-free, and run Qwen3-8B in non-thinking mode so that their inference budget is comparable. § 4.3 adds thinking mode as a labeled reference point: it beats LRT on three of the five benchmarks while spending roughly 5 5 – 450 × 450\times more inference compute. Our claim is about how far a small recurrent latent module carries a frozen decoder at a fixed, small inference cost – not that latent reasoning dominates test-time scaling.
## 2 Related Work
#### Chain-of-thought and test-time compute.
CoT prompting elicits multi-step reasoning by having the model emit intermediate steps ( Wei et al., 2022 ; Kojima et al., 2022 ; Nye et al., 2021 ) , and many methods improve it by spending inference compute: self-consistency aggregates sampled traces ( Wang et al., 2023 ) , tree- and graph-structured search organizes them ( Yao et al., 2023 ; Besta et al., 2024 ) , and compute-optimal or RL-trained reasoners scale the process further ( Snell et al., 2025 ; Guo et al., 2025 ) . All of these reason in the discrete token space and lean on reasoning traces for supervision ( Zelikman et al., 2022 ; Brown et al., 2020 ) – a dependence that is itself the motivation for moving reasoning off the token substrate.
#### Continuous-space reasoning.
A complementary line moves reasoning into a model’s latent space, feeding back vectors rather than decoded tokens. The most direct is hidden-state recurrence : Coconut feeds the decoder’s last hidden state back as the next input embedding, so that one continuous thought can hold a superposition of alternative continuations ( Hao et al., 2025 ) ; theory shows this lets a shallow transformer explore graph-reachability problems in parallel, whereas discrete CoT must proceed sequentially ( Zhu et al., 2025a ) . A second family compresses explicit CoT into dense vectors: contemplation tokens ( Cheng and Van Durme, 2024 ) , self-distillation into continuous space ( Shen et al., 2025 ) , interleaved latent and text tokens ( Su et al., 2025 ) , and a latent head whose compression rate is tuned by reinforcement learning ( Tan et al., 2025 ) ; a training-free variant decodes probability-weighted mixtures of token embeddings as “concept tokens” ( Zhang et al., 2025 ) . Related mechanisms add computation rather than verbalize it: implicit-CoT absorbs rationales by distillation or curricula ( Deng et al., 2023 ; Deng et al., 2024 ) , pause and filler tokens grant extra forward computation without emitting content ( Goyal et al., 2024 ; Pfau et al., 2024 ) , and recurrent-depth pretraining unrolls layers at inference ( Geiping et al., 2025 ) ; Chen et al. (2025a) and Zhu et al. (2025b) survey the area. Almost all of these pretrain or fine-tune the backbone, leaving open how to obtain useful latents when the decoder must stay fixed.
#### Latent conditioning with a frozen decoder.
Closest to our setting are methods that keep the decoder frozen and train only a small module supplying latents, sidestepping both the cost of fine-tuning a capable LLM and the catastrophic-forgetting risk it carries. SoftCoT runs a frozen generic assistant LM to produce soft thoughts injected as input embeddings ( Xu et al., 2025b ) , and SoftCoT++ adds test-time scaling by sampling and aggregating several such thoughts ( Xu et al., 2025a ) ; EBM-CoT keeps that proposer and calibrates its soft thoughts by descending a learned scalar energy field ( Chen et al., 2025b ) . Differentiable cache augmentation instead trains an offline “coprocessor” that writes latent embeddings into the frozen decoder’s key–value cache in one forward pass ( Liu et al., 2025 ) . All share LRT’s frozen-decoder interface but differ in how the latents arise: each obtains them from a single feed-forward proposal, at most nudged by an energy gradient, rather than from multi-step computation unfolding through a recurrent model. LRT departs on exactly this axis, replacing the generic proposer with a task-dedicated encoder and adding a recurrent reasoner that refines the latents through many bounded residual updates before decoding.
#### Recurrent and recursive reasoning.
Latent recurrence – applying a shared transition repeatedly to a persistent state – underlies Universal and Looped Transformers ( Dehghani et al., 2019 ; Giannou et al., 2023 ) and deep equilibrium models ( Bai et al., 2019 ) , decoupling computation depth from parameter count. Recent recursive reasoners revisit it for structured tasks: HRM couples two recurrent modules at different timescales ( Wang et al., 2025 ) , TRM simplifies recursion to a single tiny network ( Jolicoeur-Martineau, 2025 ) , GRAM makes it stochastic and multi-trajectory ( Baek et al., 2026 ) , and EqR frames reasoning as convergence to task-conditioned attractors ( Huang et al., 2026 ) . Each is a standalone solver trained from scratch on one task with no language ability of its own; LRT instead repurposes one (TRM) as a refiner conditioning a frozen pretrained LLM, so that iterative latent computation and broad sequence modeling come from separate components.
#### Iterative reasoning in specialized models.
A separate line builds models that reason by iterative refinement, but outside the language-model paradigm. Energy-based reasoners cast inference as optimization over a learned energy landscape, by iterative energy minimization ( Du et al., 2022 ) , by annealing a learned sequence of energy functions ( Du et al., 2024 ) , or by composing separately learned energy terms ( Du et al., 2020 ) . Discrete-diffusion reasoners such as MGDM ( Ye et al., 2025 ) refine an entire candidate solution in parallel rather than left to right. These models are strong on structured symbolic problems but each is trained from scratch for a single task family on a fixed symbolic interface; extending them to open-ended natural-language reasoning is a standing challenge. LRT keeps their iterative-refinement principle while inheriting language ability for free, by attaching a recurrent reasoner to a frozen pretrained LLM rather than building a solver from scratch.
## 3 Method
### 3.1 Problem Setup
We target reasoning with a frozen decoder. Let ℳ \mathcal{M} be a pretrained autoregressive LLM with parameters θ \theta and token embedding table E ∈ ℝ | V | × d E\in\mathbb{R}^{|V|\times d} ( d = 4096 d{=}4096 ; we use Qwen3-8B). A problem instance is a tokenized question x x with gold answer y y . SoftCoT, EBM-CoT, and LRT share a common structure: a small model produces K K continuous latent vectors L ∈ ℝ K × d L\in\mathbb{R}^{K\times d} from x x , places them in ℳ \mathcal{M} ’s input as soft tokens immediately after the question, and lets ℳ \mathcal{M} autoregressively decode the answer. The decoder is never updated; only the modules that produce L L are trained, and only through the cross-entropy of ℳ \mathcal{M} ’s output (§ 3.5 ).
The three methods differ solely in how L L is produced. SoftCoT uses (proposer, refiner) = = (frozen generic assistant LM, none); EBM-CoT keeps that proposer and adds an energy-based refiner; LRT replaces both, pairing a task-dedicated encoder with a recurrent reasoner. We write the LRT pipeline as L ( 0 ) \displaystyle L^{(0)} = g ψ ( x ) , L ⋆ = L ( 0 ) + r ϕ ( L ( 0 ) ) , \displaystyle=g_{\psi}(x),\qquad L^{\star}=L^{(0)}+r_{\phi}\big(L^{(0)}\big), y ^ \displaystyle\hat{y} = ℳ ( I , x , L ⋆ ) , \displaystyle=\mathcal{M}\big(I,\,x,\,L^{\star}\big), where g ψ g_{\psi} is the proposer (§ 3.2 ), r ϕ r_{\phi} the refiner (§ 3.3 ), and I I a task instruction; Figure 1 shows the full pipeline. Because the injection site and the frozen decoder are identical across all three methods, accuracy differences isolate the proposer and the refiner. All trainable modules operate in a small working dimension d ′ = 256 d^{\prime}{=}256 .
### 3.2 Task-Dedicated Proposer
#### Motivation.
SoftCoT obtains L L as the last-layer hidden states of a frozen generic assistant LM run on [ I assist ; x ; [UNK] × K ] [\,I_{\text{assist}};\,x;\,\texttt{[UNK]}^{\times K}\,] . A block of K K placeholder tokens appended to a prompt is off-distribution for a pretrained assistant, which therefore acts as little more than a fixed featurizer – most severely when x x is short, as with the handful of integers in a Countdown instance, where the placeholders dominate. We therefore discard the generic assistant and learn a small encoder purpose-built to emit latents for ℳ \mathcal{M} .
#### Architecture.