Enabling Agents to Communicate Entirely in Latent Space — part 1/4

Cleaned-text code points [0, 15755). Bibliography omitted here; source.md includes it.

# Enabling Agents to Communicate Entirely in Latent Space

###### Abstract

While natural language is the de facto communication medium for LLM-based agents, it presents a fundamental constraint. The process of downsampling rich, internal latent states into discrete tokens inherently limits the depth and nuance of information that can be transmitted, thereby hindering collaborative problem-solving. Inspired by telepathy, which bypasses symbolic language in communication, we propose Interlat ( Inter -agent Lat ent Space Communication ), a paradigm that leverages the continuous last hidden states of an LLM as a representation of its thought for direct communication (termed “latent communication” ). An additional learned compression process further compresses latent communication via latent space reasoning. Experiments demonstrate that Interlat outperforms both fine-tuned chain-of-thought (CoT) prompting and single-agent baselines, even across heterogeneous models, promoting more exploratory behavior and enabling genuine utilization of latent information. Further compression not only substantially accelerates inference by up to 24 × \times but also maintains competitive performance through an efficient information-preserving mechanism. We position this work as a feasibility study of entirely latent space inter-agent communication, and our results highlight its potential, offering valuable insights for future research. Our code is available at https://github.com/XiaoDu-flying/Interlat .

“The limits of my language mean the limits of my world.” — Ludwig Wittgenstein, Tractatus Logico-Philosophicus , §5.6.

## 1 Introduction

Large language model (LLM)-based agentic systems have emerged as a promising paradigm for solving complex tasks by orchestrating multiple agents through natural language communication Wang et al. (2025) ; Wang et al. (2024) ; Zhang et al. (2024b) ; Tran et al. (2025) . Despite its human readability, natural language imposes fundamental constraints on inter-agent communication. To communicate, an agent must compress its rich, high-dimensional internal states into a sequence of discrete tokens, typically exposing only a single linear message ( i.e., a chain of thought (CoT) Wei et al. (2022) plan). This downsampling not only discards alternative reasoning paths, but also incurs substantial redundancy, as much of the generated text serves linguistic coherence rather than task-relevant information Zhang et al. (2024a) . As a result, language-based communication can be ambiguous and lossy, which has been identified as a major source of coordination failures in multi-agent systems Chen et al. (2025) ; Cemri et al. (2025) .

To move beyond language space, we explore the direct transmission of internal representations for more precise and information-preserving communication. In multi-agent settings, we refer to this as latent communication . While direct sharing is challenging for humans, which is often depicted in fictions Liu (2008) , i.e., telepathy , LLM-based agents naturally perform most of their computation in latent space and produce rich hidden states throughout generation, which can be extracted to support direct, expressive communication. Previous hidden-state-based communication methods either rely on one-shot activation grafting Ramesh and Li (2025) or remain coupled to language trajectories Tang et al. (2025) , and typically require ad-hoc layer selection, adding tuning overhead.

In this work, we propose Interlat, a novel framework that realizes this vision by enabling inter-agent communication entirely in latent space. Rather than transmitting discrete tokens decoded by the language head, Interlat directly transmits the temporally aligned last-layer hidden states corresponding to an agent’s generated message, treating them as a representation of its thoughts. Under this formulation, we design a supervised objective that explicitly encourages the interpretation and utilization of task-relevant latent information, with a simple but effective stochastic token–latent mixing curriculum to stabilize training. To overcome the rigidity of full-trajectory message communication while preserving information integrity, we further train a separate reasoning process that autoregressively generates compact latent messages with a controllable number of generation steps directly in latent space, without decoding to language space tokens. This allows Interlat to compress long reasoning trajectories into concise latent prefixes, substantially improving efficiency while retaining task-critical information for downstream execution.

Experimentally, we focus on a two-agent sender-receiver scenario, which is the building block of various multi-agent systems. To reduce confounding factors, we intentionally exclude components such as tool use, retrieval, or multi-round debate. Analysis reveals that agents utilizing latent communication exhibit more exploratory behavior patterns that lead to higher success rates by leveraging task-relevant latent information rather than superficial pattern matching. Moreover, we demonstrate that latent messages can be compressed to as few as 8 tokens while maintaining competitive performance, achieving up to a 24 × 24\times reduction in communication latency. Further analysis of the output probability distribution after compression reveals how task-critical information is effectively preserved.

## 2 Related Work

#### Latent Reasoning in LLMs.

Recent research has begun shifting reasoning processes from the discrete language space to continuous latent representations, bypassing the bandwidth and efficiency limits of text ( ≈ 15 \approx 15 bits/token vs. ≈ 40 \approx 40 k bits/hidden-state) Zhu et al. (2025b) . To expand computation during inference, Goyal et al. (2023) introduces pause tokens, while Pfau et al. (2024) employs filler tokens to scaffold intermediate reasoning. Beyond token scheduling, Liu et al. (2024) proposes a latent coprocessor that modifies the transformer KV cache. Other work Hao et al. (2024) ; Shen et al. (2025) ; Cheng and Van Durme (2024) enables multi-path parallel reasoning by feeding the last hidden state back as the next input embedding. Modular frameworks Bae et al. (2024) ; Gao et al. (2024) ; Geiping et al. (2025) decouple encoding, latent reasoning, and decoding. Building upon these insights, we shift focus from single-model latent reasoning to inter-agent communication entirely in latent space.

#### Multi-agent Communication.

LLM-based agent systems typically orchestrate in natural language Zhu et al. (2025a) ; Wang et al. (2024) , which can introduce ambiguity and computational overhead Zhang et al. (2024a) ; Yu et al. (2024) ; Cemri et al. (2025) ; Chen et al. (2025) . Emergent communication studies Lazaridou et al. (2016) ; Lazaridou et al. (2018) ; Tucker et al. (2021) ; Tucker et al. (2022) show that non-linguistic protocols can emerge, yet depend on channels learned from scratch and disconnected from internal reasoning. Recent work explores richer forms: Pham et al. (2023) transmits probability-weighted tokenizer embeddings; Ramesh and Li (2025) blends hidden activations between agents; and Tang et al. (2025) communicates per-token latent deltas tied to language trajectories. Zheng et al. (2025) infers latent “thoughts” from hidden states via an autoencoder and injects them as prefixes using recovered dependencies. Unlike these works, our method directly transmits temporally aligned sequences of last hidden states and further compresses them to enable efficient, language-free communication. Interlat preserves agent autonomy by requiring no parameter sharing Christianos et al. (2021) , memory coupling Salemi et al. (2025) , or cache synchronization Fu et al. (2025) ; Zou et al. (2025) , while expanding the effective communication bandwidth.

## 3 Interlat

In this section, we formalize how to extract an agent’s states as the representation of its “thought” for inter-agent latent space communication. Let x = ( x 1 , … , x T ) x=(x_{1},\dots,x_{T}) denote a sequence consisting of a prompt x 1 : m x_{1:m} and a completion sequence y = ( y 1 , … , y L ) y=(y_{1},\dots,y_{L}) such that y ℓ = x m + ℓ y_{\ell}=x_{m+\ell} and L = T − m L=T-m . For each decoding step ℓ = 1 , … , L \ell=1,\dots,L , define: h ℓ \displaystyle h_{\ell} = Transformer ​ ( x ≤ m + ℓ − 1 ) m + ℓ − 1 , \displaystyle=\mathrm{Transformer}\!\left(x_{\leq m+\ell-1}\right)_{m+\ell-1}, H \displaystyle H = [ h 1 , h 2 , … , h L ] , \displaystyle=[\,h_{1},\,h_{2},\,\dots,\,h_{L}\,], where h ℓ ∈ ℝ d h_{\ell}\in\mathbb{R}^{d} is the last-layer hidden state immediately before predicting y ℓ y_{\ell} ( i.e., at position m + ℓ − 1 m+\ell-1 in the full sequence). H ∈ ℝ L × d H\in\mathbb{R}^{L\times d} collects these last hidden states for the completion region.

### 3.1 Latent Communication

Interlat removes natural language constraints by letting agents transmit their thoughts by directly passing the collected last hidden states, which we termed latent communication. As shown in Figure 1 , this transmission occurs at the end of an agent’s message generation process. Special tokens, x i = <bop> x_{i}=\texttt{<bop>} and x j = <eop> x_{j}=\texttt{<eop>} , are added to mark the beginning and the end of the latent communications. Consider an agent ℳ i \mathcal{M}_{i} solving a task 𝒯 = { x 1 , … , x m } \mathcal{T}=\{x_{1},\dots,x_{m}\} . Upon receiving a latent communication H = { h 1 , h 2 , … , h L } H=\{h_{1},h_{2},\dots,h_{L}\} from another agent, it forms its input embedding as: E = [ e ⁡ ( x 1 ) , … , e ⁡ ( x i ) , h 1 , h 2 , … , h L , e ⁡ ( x j ) ] , E\;=\;[\,e(x_{1}),\dots,e(x_{i}),h_{1},h_{2},\dots,h_{L},e(x_{j})\,], where e ⁡ ( ⋅ ) e(\cdot) is the token embedding. This inference process is analogous to language space multi-agent systems, except that it directly feeds hidden states between agents. The latent communications are processed by a trainable light-weight self-attention and a projection layer as a communication adapter for rescaling and interpretation. For brevity, we may refer to latent communication as latents where unambiguous.

### 3.2 Training Procedure

In this work, we consider two agents: a reasoning agent as a sender that produces a task-specific plan together with its last-layer hidden states, and an actor agent as a receiver that consumes this communication to generate actions to solve tasks. This two-agent setting can serve as a fundamental building block for more complex multi-agent systems.

Let Y t Y_{t} denote the next token at supervised position t ∈ S t\in S , where S S indexes the actor’s output tokens corresponding to the ground-truth response, and C t C_{t} the decoder prefix up to position t t . We encourage the actor to utilize H H by maximizing a supervised fine-tuning objective regularized by conditional distributional separation: ℒ total = ℒ task + λ S ​ ℒ sep + λ A ​ ℒ align , \mathcal{L}_{\mathrm{total}}\;=\;\mathcal{L}_{\mathrm{task}}\;+\;\lambda_{\mathrm{S}}\,\mathcal{L}_{\mathrm{sep}}\;+\;\lambda_{\mathrm{A}}\,\mathcal{L}_{\mathrm{align}}, where λ S , λ A > 0 \lambda_{\mathrm{S}},\lambda_{\mathrm{A}}>0 , and ℒ task \mathcal{L}_{\mathrm{task}} is the standard cross-entropy loss that ensures the model produces accurate and coherent responses for the given task.

#### Conditional thought separation.

We compare the conditional output distributions p θ p_{\theta} induced by matched latent communications H H and mismatched latents H ~ \tilde{H} ( i.e., latent communications sampled from a different task). Specifically, we minimize a weighted Jensen–Shannon divergence Lin (2002) : ℒ sep = − 1 | S | ∑ t ∈ S JS ( p θ ( ⋅ ∣ C t , H ) , p θ ( ⋅ ∣ C t , H ~ ) ) . \mathcal{L}_{\mathrm{sep}}=-\,\frac{1}{|S|}\sum_{t\in S}\mathrm{JS}\!\big(p_{\theta}(\cdot\!\mid\!C_{t},H),\,p_{\theta}(\cdot\!\mid\!C_{t},\tilde{H})\big). This objective separates matched from mismatched conditional distributions, providing a robust training signal that encourages the actor to attend to and leverage task-relevant latent information.

#### Plan-aligned regulation.

While maximizing separation encourages sensitivity to H H , it also introduces a failure mode where the model may exploit the objective by shifting probability mass toward idiosyncratic tokens that increase divergence while harming task utility. To mitigate this, we regularize predictions conditioned on H H using those conditioned on the corresponding language-space plan P P , generated by the same instruction-tuned model during the autoregressive generation of H H . Let p plan ( ⋅ ∣ C t , P ) p_{\mathrm{plan}}(\cdot\mid C_{t},P) denote the distribution when only the plan is provided. For brevity, we omit explicit conditioning on C t C_{t} in the following. ℒ align = β | S | ∑ t ∈ S KL ( p θ ( ⋅ ∣ H ) ∥ p plan ( ⋅ ∣ P ) ) \displaystyle\mathcal{L}_{\mathrm{align}}=\frac{\beta}{|S|}\sum_{t\in S}\mathrm{KL}\!\big(p_{\theta}(\cdot\mid H)\,\|\,p_{\mathrm{plan}}(\cdot\mid P)\big) + α | S | ∑ t ∈ S ( 1 − cos ( 𝐳 θ ( H ) , 𝐳 plan ( P ) ) ) , \displaystyle+\frac{\alpha}{|S|}\sum_{t\in S}\big(1-\cos\!\big(\mathbf{z}_{\theta}(H),\,\mathbf{z}_{\mathrm{plan}}(P)\big)\big), where 𝐳 θ \mathbf{z}_{\theta} and 𝐳 plan \mathbf{z}_{\mathrm{plan}} denote the corresponding normalized logit. All divergences and cosine similarities are computed at supervised positions, with probabilities obtained from the softmax of logits.

#### Curriculum Learning.

Learning to interpret latents from scratch is unstable. We thus adopt a token-to-latent curriculum that stochastically replaces early communication positions with their corresponding plan token embeddings. Concretely, we sample a replacement rate r ∈ { 0 , 0.1 , … , 1.0 } r\in\{0,0.1,\dots,1.0\} and form a mixed communication H ( r ) = e 1 , … , e ⌊ r ⋅ L ⌋ ⏟ token embeddings ⊕ h ⌊ r ⋅ L ⌋ + 1 , … , h L ⏟ latent states . H^{(r)}=\underbrace{e_{1},\dots,e_{\lfloor r\cdot L\rfloor}}_{\text{token embeddings}}\oplus\underbrace{h_{\lfloor r\cdot L\rfloor+1},\dots,h_{L}}_{\text{latent states}}. This method enhances training efficiency while achieving strong model performance.

## 4 Information Compression

While full-length latents H L ∈ ℝ L × d H_{L}\in\mathbb{R}^{L\times d} are highly expressive, their temporal length (often dozens to hundreds of steps) introduces substantial communication latency. Unlike natural-language tokens, whose semantics are discrete and inherently sequential, latent states are continuous and over-parameterized relative to task requirements, suggesting that much of the temporal structure is redundant. Our goal is therefore to learn an information-preserving bottleneck that compresses latents while retaining their utility for downstream agents.

#### Compression via Latent-Space Reasoning.

To this end, we train a separate reasoning model M ϕ M_{\phi} to generate compact latent messages H K ∈ ℝ K × d H_{K}\in\mathbb{R}^{K\times d} with K ≪ L K\ll L , while keeping the actor model and its communication adapter frozen. Rather than truncating or subsampling H L H_{L} , M ϕ M_{\phi} performs autoregressive reasoning entirely in latent space by feeding its last hidden state back as the next input embedding through a lightweight projection: ⟨ M ϕ ( E i ) → h i , E i + 1 = E i ⊕ Proj ( h i ) ⟩ . \langle M_{\phi}(E_{i})\rightarrow h_{i},\;E_{i+1}=E_{i}\oplus\mathrm{Proj}(h_{i})\rangle. This design enables an end-to-end differentiable latent generation loop without decoding to tokens, and isolates compression from changes in the actor’s behavior. During training, only the parameters of M ϕ M_{\phi} are updated, ensuring that compression is learned purely by adapting the latent message itself.

#### Training Objective.