Enabling Agents to Communicate Entirely in Latent Space — part 3/4

Cleaned-text code points [31379, 47254). Bibliography omitted here; source.md includes it.

As shown in Table 5 , both 3-agent latent configurations consistently outperform their natural language CoT counterparts with identical agent structures (e.g., 62.85% vs. 61.19% on seen tasks for the parallel tree). Moreover, both configurations yield solid improvements over the original 2-agent Interlat baseline (61.19%). These results demonstrate that the performance gains stem intrinsically from the rich expressive capacity and information preservation of the latent space, rather than a mere increase in the number of agents. Notably, the superior performance of the tree-3 configuration confirms that Interlat effectively supports parallel multi-agent reasoning, allowing downstream actors to synthesize heterogeneous latent perspectives without requiring pair-specific training.

## 6 Conclusion

In this work, we introduced Interlat, a paradigm that enables inter-agent communication entirely in latent space. Across experiments, our results show that directly transmitting and reasoning over latent states improves task performance and achieves substantially higher communication efficiency, even demonstrating compatibility across heterogeneous models. Beyond full-length latent exchange, we show that latent communication can be aggressively compressed through latent-space reasoning, forming a compact, task-preserving representation that retains parallel hypotheses while discarding redundant structure. Together, these findings suggest that communication need not be bound to language tokens, highlighting latent states as a viable, efficient, and generalizable medium for next-generation multi-agent systems.

## 7 Limitations

Our study has explored how inter-agent communication can be realized entirely in latent space and demonstrated its potential benefits in terms of performance and efficiency. However, several important limitations should be considered when interpreting our results and applying this paradigm more broadly.

First, our experiments mainly focus on a controlled two-agent setting, evaluated on one embodied interactive benchmark (ALFWorld) and one non-interactive symbolic reasoning benchmark (MATH). While these benchmarks jointly cover both interactive planning and single-turn reasoning scenarios, they do not yet fully capture the diversity of real-world multi-agent systems, such as settings with larger teams, dynamic role assignments, long-horizon collaboration, or richer environments involving tool use and external memory. Extending latent communication to larger-scale and more heterogeneous agent ecosystems is a natural and important direction for future work.

Second, our approach assumes access to internal model representations, specifically last-layer hidden states, in order to enable latent communication. This assumption may not hold for closed-source or API-only models, where hidden states are inaccessible. As a result, the current formulation of Interlat is primarily applicable to open or inspectable models, and extending latent communication to restricted-access settings is an open challenge.

Finally, latent communication trades human interpretability for efficiency and expressive capacity. Unlike language-based communication, latent messages are not directly human-readable, which complicates debugging, monitoring, and failure analysis in complex systems. While this work primarily focuses on task performance and efficiency and provides preliminary interpretability analysis via PCA (Figure 4 ), developing principled tools to improve both the interpretability and controllability of latent communication remains an important direction for broader deployment.

Despite these limitations, we view this work as an initial yet concrete step toward understanding and enabling inter-agent communication beyond language. Our results provide empirical evidence that latent space communication can support effective coordination and can be aggressively compressed while preserving task utility, highlighting its potential as a complementary paradigm to language-based interaction in LLM-powered agent systems.

## 8 Ethical Considerations

No human participants, crowdsourcing, or personally identifiable information (PII) were involved in this research. All experiments were conducted within a simulated environment using standard dataset splits.

Our study focuses on inter-agent communication in latent space, utilizing the last hidden states and their compressed variants. A potential theoretical risk is that such latent communication could be exploited to circumvent language-based safety mechanisms. To mitigate this concern to the greatest extent possible, we neither trained on nor evaluated any harmful instructions, and no harmful actions occurred during our experiments. Furthermore, to promote transparency, we provide PCA-based visualizations of latent communication in Figure 4 the analyze internal probability distribution of compressed latent communications in Figure 5 , offering a clearer understanding of the information being transmitted.

## 9 Acknowledgments

This research was partially supported by National Key R&D Program of China under Grant No. 2024YFF0907802, Zhejiang Provincial Natural Science Foundation of China under Grant No. LD24F020011, and Alibaba Group through Alibaba Research Intern Program.

## Appendix A LLM usage

ChatGPT 1 1 1 https://chat.openai.com/ was used purely with the language of the paper during the writing process, including spell-checking and paraphrasing the authors’ original content, without suggesting new content. Any content generated with the assistant underwent meticulous manual review and subsequently received final approval from the authors.

## Appendix B Compression Loss

#### Setup.

After training an actor M θ M_{\theta} to consume latent communications, we freeze M θ M_{\theta} and train a reasoning model M ϕ M_{\phi} to produce compact, information-dense latent communications of length K K that the frozen actor can still exploit. For an input instance with supervised token indices S S (the teacher-forced window after the first user turn), let H 1 : K gen = M ϕ ( x ) and H 1 : L full = M ins ( x ) , H^{\mathrm{gen}}_{1:K}\;=\;M_{\phi}(x)\quad\text{and}\quad H^{\mathrm{full}}_{1:L}\;=\;M_{\text{ins}}(x), (1) denote respectively the generated latent communication from the trainable reasoning model and the full-length latent communication extracted from a fixed instruction-tuned model M ins M_{\text{ins}} . A lightweight communication adapter g ⁡ ( ⋅ ) g(\cdot) (kept frozen) preprocesses the latent communication before concatenation with boundary tokens <bop> / <eop> . For brevity, we use H K ≡ H gen 1 : K H_{K}\equiv H^{\mathrm{gen}}_{1:K} and H L ≡ H full 1 : L H_{L}\equiv H^{\mathrm{full}}_{1:L} .

We define three actor-scored forward paths through the frozen actor M θ M_{\theta} given a prompt x x : (i) Path A (generated latents) : E ( A ) = [ e ⁡ ( x ) , e ⁡ ( <bop> ) , g ⁡ ( H K ) , e ⁡ ( <eop> ) ] E^{(A)}=[\,e(x),\,e(\texttt{<bop>}),\,g(H_{K}),\,e(\texttt{<eop>})\,] ; (ii) Path D (full-length latents) : E ( D ) = [ e ⁡ ( x ) , e ⁡ ( <bop> ) , g ⁡ ( H L ) , e ⁡ ( <eop> ) ] E^{(D)}=[\,e(x),\,e(\texttt{<bop>}),\,g(H_{L}),\,e(\texttt{<eop>})\,] ; (iii) Path B (no latents) : E ( B ) = [ e ⁡ ( x ) ] E^{(B)}=[\,e(x)\,] . Let 𝐳 t ( q ) \mathbf{z}^{(q)}_{t} be the frozen-actor logits at position t ∈ S t\!\in\!S under path q ∈ { A , D , B } q\!\in\!\{A,D,B\} , and p t ( q ) = softmax ⁡ ( 𝐳 t ( q ) / T ) p^{(q)}_{t}\;=\;\mathrm{softmax}\!\big(\mathbf{z}^{(q)}_{t}/T\big) (2) be the corresponding token distributions with temperature T ≥ 1 T\!\geq\!1 used for distillation. Unless stated otherwise, gradients do not flow into M θ M_{\theta} or g ⁡ ( ⋅ ) g(\cdot) .

#### (1) Actor cross-entropy utility.

We require the generated message to be useful for the frozen actor: ℒ task = 1 | S | ​ ∑ t ∈ S ( − log ⁡ p θ ​ ( y t ∣ C t , H K ) ) ⏟ (computed under Path A). \underbrace{\mathcal{L}_{\mathrm{task}}\;=\;\frac{1}{|S|}\sum_{t\in S}\big(-\log p_{\theta}(y_{t}\mid C_{t},H_{K})\big)}_{\text{(computed under Path A).}} (3) This term enforces that the compressed latents H K H_{K} still drive correct next-token predictions, directly penalizing information loss due to shortening ( K ≪ L K\!\ll\!L ). It prevents degenerate “over-compression” that would be efficient but useless to the actor. Practically, it anchors training on task utility, encouraging compression gain does not come at the cost of downstream performance.

#### (2) Uncertainty-weighted agreement.

We further encourage behavioral agreement between using full-length latent communication (Path D) and generated compressed latent communication (Path A), with per-token weights that reflect how much any latent reduces uncertainty relative to the no-latent baseline (Path B). Let the entropies be H ( q ) ( t ) = − ∑ v p ( q ) t ( v ) log p ( q ) t ( v ) ⏟ q ∈ { A , D , B } , . \underbrace{H^{(q)}(t)\;=\;-\sum_{v}p^{(q)}_{t}(v)\,\log p^{(q)}_{t}(v)}_{q\!\in\!\{A,D,B\}},. (4) Define raw weights w t ⋆ = max ⁡ ( H ( B ) ​ ( t ) − H ( D ) ​ ( t ) , 0 ) w_{t}^{\star}=\max\!\big(H^{(B)}(t)-H^{(D)}(t),\,0\big) and optionally clip w t ⋆ w_{t}^{\star} to [ 0 , τ ] [0,\tau] to suppress outliers. Normalize to unit mean: w t = w t ⋆ 1 | S | ​ ∑ u ∈ S w u ⋆ + ε . w_{t}\;=\;\frac{w_{t}^{\star}}{\frac{1}{|S|}\sum_{u\in S}w_{u}^{\star}+\varepsilon}. (5) The agreement term is a temperature-scaled KL: ℒ pref = 1 ∑ t ∈ S w t ∑ t ∈ S w t T 2 KL ( p ( D ) t ∥ p ( A ) t ) = T 2 ∑ t ∈ S w t ​ ∑ t ∈ S w t ​ ∑ v p t ( D ) ​ ( v ) ​ log ⁡ p t ( D ) ​ ( v ) p t ( A ) ​ ( v ) . \begin{split}\mathcal{L}_{\mathrm{pref}}&=\frac{1}{\sum_{t\in S}w_{t}}\sum_{t\in S}w_{t}\;T^{2}\,\mathrm{KL}\!\Big(p^{(D)}_{t}\,\big\|\,p^{(A)}_{t}\Big)\\ &=\frac{T^{2}}{\sum_{t\in S}w_{t}}\sum_{t\in S}w_{t}\sum_{v}p^{(D)}_{t}(v)\log\frac{p^{(D)}_{t}(v)}{p^{(A)}_{t}(v)}.\end{split} (6) By matching p ( A ) p^{(A)} to p ( D ) p^{(D)} where full latents actually reduce uncertainty (weights w t w_{t} ), this term teaches H K H_{K} to reproduce the informative behavioral effects of H L H_{L} while ignoring positions where latents are unhelpful. Unlike reconstruction-based or contrastive objectives, this formulation aligns compressed latents directly through the actor’s induced behavior, avoiding assumptions about latent invertibility or instance-level correspondence. This allows compressed communication to focus on functional equivalence rather than representational similarity. This is particularly important for reasoning latents, which are over-parameterized, temporally misaligned under compression, and lack a natural one-to-one mapping across steps. By aligning compressed and full communications through induced behavior, our formulation supports variable-length latent messages, enables abstraction across multiple reasoning steps, and yields more stable and transferable training signals.

#### (3) Latent direction alignment.

To stabilize compression, we align the global direction of actor-side latent features induced by generated vs. data latents. Let Z k ( q ) ∈ ℝ d z Z^{(q)}_{k}\in\mathbb{R}^{d_{z}} be the actor-side features (after g ⁡ ( ⋅ ) g(\cdot) and the actor’s input stack) at latent step k k under path q ∈ { A , D } q\in\{A,D\} . When H L H_{L} has length L ≠ K L\!\neq\!K , apply a fixed resampling operator ρ K \rho_{K} ( e.g., uniform down/up-sampling) and write Z 1 : K ( D ) = ρ K ( Z 1 : L ( D ) ) Z^{(D)}_{1:K}=\rho_{K}\!\big(Z^{(D)}_{1:L}\big) . Define step-averaged directions z ¯ ( q ) = 1 K ​ ∑ k = 1 K Z k ( q ) \bar{z}^{(q)}=\frac{1}{K}\sum_{k=1}^{K}Z^{(q)}_{k} and the cosine penalty ℒ geom = 1 − cos ⁡ ( z ¯ ( A ) , z ¯ ( D ) ) = 1 − ⟨ z ¯ ( A ) , z ¯ ( D ) ⟩ ‖ z ¯ ( A ) ‖ 2 ​ ‖ z ¯ ( D ) ‖ 2 . \begin{split}\mathcal{L}_{\mathrm{geom}}&=1-\cos\!\big(\bar{z}^{(A)},\,\bar{z}^{(D)}\big)\\ &=1-\frac{\langle\bar{z}^{(A)},\,\bar{z}^{(D)}\rangle}{\|\bar{z}^{(A)}\|_{2}\,\|\bar{z}^{(D)}\|_{2}}.\end{split} (7) This term preserves the geometry of the actor-side representations, preventing the compressed latents from drifting to directions that the actor interprets differently. Empirically, it improves stability and mitigates mode collapse when K K is small by retaining the global semantic orientation of H L H_{L} .

#### Overall objective.

The compression objective for M ϕ M_{\phi} (with M θ M_{\theta} frozen) is

ℒ compress = λ task ​ ℒ task + λ pref ​ ℒ pref + λ geom ​ ℒ geom . \begin{split}\mathcal{L}_{\mathrm{compress}}&=\lambda_{\mathrm{task}}\;\mathcal{L}_{\mathrm{task}}\\ &\quad+\lambda_{\mathrm{pref}}\;\mathcal{L}_{\mathrm{pref}}\\ &\quad+\lambda_{\mathrm{geom}}\;\mathcal{L}_{\mathrm{geom}}.\end{split} (8) In practice, all terms are computed over t ∈ S t\!\in\!S with teacher forcing; gradients propagate only to ϕ \phi .

### B.1 Baselines and settings in Interlat.

We consider two external baselines, which do not rely on latent communication at all. All baselines are trained using the same base models (Qwen2.5-7B/0.5B-Base, LLaMA3.1-8B-Base) as Interlat, differing only in whether and how inter-agent communication is provided.

1. CoT (full). We use complete Chain-of-Thought (CoT) traces produced by a related instruction-tuned model (Qwen2.5-7B-Instruct, Qwen2.5-0.5B-Instruct, and LLaMA3.1-8B-Instruct) to perform full-parameter supervised fine-tuning. In inference, the model receives a complete CoT plan before generating answers.

Rationale. This baseline serves as a strong upper bound for language-based communication: it evaluates whether latent communication can surpass explicit human-readable planning, and controls for the supervision quality provided by an instruction-tuned teacher.

2. No-CoT. The language model is trained to produce the final answer directly, without receiving any plan from other agents. Rationale. This baseline isolates the contribution of any communication signal. It tests whether inter-agent exchange, latent or linguistic, is necessary for solving multi-step tasks.

In addition, we evaluate controlled variants of Interlat to diagnose what information is encoded in the latents.

1. Text. Instead of latent communication, we feed the corresponding CoT plan (in language space) to the actor.

Rationale. This variant keeps the interaction protocol unchanged while varying only the communication channel. It enables a direct comparison between language-space and latent-space communication under matched training conditions, disentangling architectural factors from representational ones.

2. No-Comm. We remove any communication from the actor’s input. This variant quantifies the intrinsic benefit of communication in our framework and verifies that performance improvements do not arise solely from modifications to the underlying model parameters.

3. CrossTask. We replace the current task’s latent communication with one sampled from a different task. Rationale. This variant examines whether the actor is genuinely interpreting task-specific latent content. A substantial degradation indicates reliance on meaningful information encoded in the latents, rather than superficial distributional shortcuts.

4. Noised. We add perturbations to the latent communication H H : (a) CovNoise-0.5 × \times /1.0 × \times : covariance-shaped noise ε t ∼ 𝒩 ⁡ ( 0 , Σ ^ ) \varepsilon_{t}\sim\mathcal{N}(0,\hat{\Sigma}) with optional strength λ ∈ { 0.5 , 1.0 } \lambda\in\{0.5,1.0\} , where Σ ^ \hat{\Sigma} is the sample covariance of the original H H ; (b) WhiteNoise : a control drawn from 𝒩 ⁡ ( 0 , I ) \mathcal{N}(0,I) with the same length. Rationale. Noise-based perturbations interrogate the robustness and locality of latent-space semantics. Covariance-shaped noise preserves global second-order structure, whereas white noise does not, allowing us to assess whether the actor relies on fine-grained geometric relations within true latent trajectories.