XBridge: Entity-Grounded Latent Bridge for Heterogeneous LLM Communication — part 1/4

Cleaned-text code points [0, 15563). Bibliography omitted here; source.md includes it.
# XBridge : Entity-Grounded Latent Bridge for Heterogeneous LLM Communication

###### Abstract

Heterogeneous multi-agent LLM systems, where agents are powered by different model families, can outperform homogeneous configurations by reducing redundant reasoning patterns. Yet existing communication protocols either operate through text, discarding the sender’s internal representations, or require architectural homogeneity for latent-level transfer. We identify the entity grounding problem in cross-architecture communication: cross-attention bridges that transfer continuous representations across different LLM families suffer from rare-token compression collapse, where entity identity is lost in the continuous bottleneck (bridge-only F1 ∼ \sim 30%). We propose XBridge , a decode-free communication protocol that addresses this through two mechanisms. Lexical Anchor Mapping (LAM) maps the sender’s original context tokens to the receiver’s vocabulary, providing discrete entity anchors. A Latent Enrichment Bridge (LEB) lets the receiver query the sender’s hidden states for contextual enrichment. The entity anchors ground the bridge’s contextual signals to specific entities through the receiver’s own self-attention. Across three model families (Llama, Qwen, and Mistral), seven benchmarks, and both communication directions, XBridge outperforms text-based communication on all seven tasks for each model pair while achieving 11 × \times lower latency, and in a same-architecture setting it also exceeds a KV-sharing baseline on six of seven tasks. LEB requires only 264M trainable parameters (3.8% of the receiver), is trained on a small balanced sample set, and adds negligible inference overhead. Our implementation is available at https://github.com/WooseongYang/XBridge .

## 1 Introduction

Multi-agent systems (MAS) can improve the reasoning capabilities of large language models (LLMs) by distributing computation across multiple inference passes [ 1 ] .

Recent findings show that when interacting agents share identical parameters and inductive biases, their interactions may degenerate into redundant majority voting rather than surfacing complementary reasoning [ 2 , 3 ] . A natural way to mitigate this redundancy is to use heterogeneous ensembles: configurations comprising distinct model families with diverse tokenizers, training corpora, architectures, and alignment procedures. Yet this representational diversity introduces a severe communication bottleneck: agents must exchange information across structurally incompatible latent spaces.

Existing communication paradigms face an efficiency-fidelity trade-off. Text-based methods, such as natural language communication (NLComm) [ 4 ] , are architecture-agnostic but incur autoregressive decoding latency and compress the sender’s internal state into a textual summary. Latent-level methods avoid sender-side decoding, but existing approaches such as KV cache sharing [ 5 ] and hidden-state injection [ 6 ] typically assume architectural compatibility, including aligned hidden dimensions, attention layouts, positional encodings, or tokenizers. Learned continuous projections offer a possible route across model families [ 7 ] . However, we find that continuous projections alone degrade in a specific and practically important way: they can transfer contextual information without reliably preserving the entities to which that context refers.

We define this degradation as the entity grounding problem . In heterogeneous LLM communication, a continuous bridge can transfer contextual information from the sender, but it struggles to preserve the discrete entity identities, such as names, numbers, or rare tokens, to which that context refers. We use entity fidelity to refer to preserving the correct lexical identity, and entity grounding to refer to binding the sender’s contextual signal to the corresponding receiver-native lexical anchor. In bridge-only experiments, the receiver recovers aspects of the context but often loses the exact identities of the participating entities. Homogeneous communication settings are less exposed to this problem because shared tokenizers and representational spaces provide implicit entity anchors. Heterogeneous communication therefore creates a demanding setting in which continuous contextual transfer and exact discrete entity fidelity must be maintained simultaneously.

To address this bottleneck, we propose XBridge , a decode-free communication protocol that mitigates the entity grounding problem via dual channels. First, Lexical Anchor Mapping (LAM) deterministically translates the sender’s original context tokens into the receiver’s vocabulary, providing discrete entity anchors without autoregressive decoding. Second, a Latent Enrichment Bridge (LEB) uses gated cross-attention to let the receiver query the sender’s hidden states, supplying continuous contextual information that can be bound to these lexical anchors. By decoupling the preservation of discrete entity identity (the who and what ) from the transfer of continuous contextual knowledge (the how and why ), XBridge enables heterogeneous models to communicate without requiring textual decoding or direct latent-space compatibility. The bridge is lightweight, requiring only 264M trainable parameters (3.8% of the receiver), training on a small balanced sample set in under 10 minutes, and adding negligible inference overhead.

In summary, our core contributions are: • We formulate the entity grounding problem in heterogeneous LLM communication and demonstrate rare-token compression collapse, a failure mode in which continuous bridges preserve contextual signals but lose discrete entity identity.

• We propose XBridge , a decode-free dual-channel protocol that combines Lexical Anchor Mapping for receiver-native entity anchors with Latent Enrichment Bridge for sender-side contextual enrichment. To our knowledge, this is the first protocol to pair an explicit discrete-anchor channel with latent transfer for heterogeneous communication.

• We show that XBridge improves heterogeneous communication across three model families and seven benchmarks, achieving 11 × \times lower latency than text-based communication, and that in a same-architecture setting it exceeds KV-cache sharing on 6 of 7 tasks.

## 2 The Entity Grounding Problem

Heterogeneous communication requires satisfying three requirements. First, the receiver must recover receiver-native lexical identities for entities that appear in the sender context. Second, it must use contextual information encoded in the sender’s hidden states. Third, it must consume this information without treating sender activations as native receiver activations. We refer to these requirements as entity fidelity , entity grounding , and representation compatibility , respectively.

We formalize heterogeneous LLM communication as information transfer between models with incompatible tokenizers, vocabularies, and hidden spaces. Let M i = ( τ i , V i , E i , F i , G i ) , i ∈ { S , R } , M_{i}=(\tau_{i},V_{i},E_{i},F_{i},G_{i}),\qquad i\in\{S,R\}, (1) denote a sender M S M_{S} and receiver M R M_{R} , where τ i \tau_{i} is the tokenizer, V i V_{i} the vocabulary, E i E_{i} the embedding map, F i F_{i} the transformer, and G i G_{i} the language-model head. The models are heterogeneous when V S ≠ V R , d S ≠ d R , F S ≢ F R , V_{S}\neq V_{R},\qquad d_{S}\neq d_{R},\qquad F_{S}\not\equiv F_{R}, (2) where architectural mismatch may include differences in depth, attention layout, normalization, positional encoding, tokenizer design, or training distribution.

The sender observes a context C C , while the receiver observes a question Q Q and must generate an answer A A . The sender performs a single forward pass: c S = τ S  ( C ) = ( c 1 , … , c T C ) , H S = F S  ( E S  ( c S ) ) ∈ ℝ T C × d S . c_{S}=\tau_{S}(C)=(c_{1},\ldots,c_{T_{C}}),\qquad H_{S}=F_{S}(E_{S}(c_{S}))\in\mathbb{R}^{T_{C}\times d_{S}}. (3) Unless otherwise stated, H S H_{S} denotes the sender’s last-layer hidden states. A communication protocol produces a sender-side message m S = π S  ( c S , H S ) , m_{S}=\pi_{S}(c_{S},H_{S}), which the receiver consumes to generate p π  ( A ∣ C , Q ) = p R  ( A ∣ Q ; Γ θ  ( Q , m S ) ) . p_{\pi}(A\mid C,Q)=p_{R}(A\mid Q;\Gamma_{\theta}(Q,m_{S})). Here, Γ θ \Gamma_{\theta} denotes the receiver-side mechanism for consuming the message. Text protocols communicate strings or token sequences, latent protocols communicate continuous states, and hybrid protocols communicate both m S = ( m disc  ( C ) , m cont  ( C ) ) m_{S}=\bigl(m_{\mathrm{disc}}(C),m_{\mathrm{cont}}(C)\bigr) .

### 2.1 Entity Fidelity

Let ℰ ⁡ ( C ) \mathcal{E}(C) denote the set of entity mentions in C C , including names, numbers, dates, and other rare or semantically specific spans. For an entity e ∈ ℰ ⁡ ( C ) e\in\mathcal{E}(C) , define its receiver-native lexical realization as a R  ( e ) = τ R  ( e ) ∈ V R ∗ , a_{R}(e)=\tau_{R}(e)\in V_{R}^{\ast}, (4) where V R ∗ V_{R}^{\ast} is the set of finite token sequences over the receiver vocabulary. We call a R  ( e ) a_{R}(e) an entity anchor . Entity fidelity is the requirement that the receiver assigns high probability to the correct receiver-native anchor.

For a gold answer entity e ⋆ e^{\star} , we measure entity fidelity by the receiver-side rank rank π  ( e ⋆ , C , Q ) = rank v ∈ V R  [ ℓ R  ( v ∣ Q ; Γ θ  ( Q , m S ) ) ] v ∈ a R  ( e ⋆ ) , \mathrm{rank}_{\pi}(e^{\star};C,Q)=\mathrm{rank}_{v\in V_{R}}\left[\ell_{R}(v\mid Q;\Gamma_{\theta}(Q,m_{S}))\right]_{v\in a_{R}(e^{\star})}, (5) where ℓ R \ell_{R} is the receiver logit induced by G R G_{R} . For multi-token entities, we use the first answer token as a diagnostic proxy. A protocol has high entity fidelity when this rank is low.

Entity fidelity can also be characterized interventionally. Let C e → e ′ C^{e\rightarrow e^{\prime}} denote the context obtained by replacing every occurrence of e e with another entity e ′ e^{\prime} of the same semantic type. A protocol is entity-sensitive if A ^ π  ( C , Q ) = e ⟹ A ^ π  ( C e → e ′ , Q ) = e ′ . \hat{A}_{\pi}(C,Q)=e\quad\Longrightarrow\quad\hat{A}_{\pi}(C^{e\rightarrow e^{\prime}},Q)=e^{\prime}. (6) This condition separates entity identity from relational structure. If the relation is preserved but the answer does not follow the substituted entity, the protocol does not preserve entity fidelity.

### 2.2 Entity Grounding Failure

Entity fidelity concerns whether the receiver outputs the correct entity. Entity grounding concerns whether the sender’s contextual signal is bound to the correct receiver-native entity anchor. A protocol may transfer contextual information while still leaving that information ungrounded to the correct entity in V R V_{R} . A continuous bridge can be written abstractly as r θ ( ℓ ) = B θ ( ℓ )  ( H S , h R ( ℓ ) ) ∈ ℝ T R × d R , r_{\theta}^{(\ell)}=B_{\theta}^{(\ell)}(H_{S},h_{R}^{(\ell)})\in\mathbb{R}^{T_{R}\times d_{R}}, where h R ( ℓ ) ∈ ℝ T R × d R h_{R}^{(\ell)}\in\mathbb{R}^{T_{R}\times d_{R}} is the receiver hidden-state sequence at layer ℓ \ell with the receiver sequence length T R T_{R} . This abstraction covers projections, adapters, hidden-state fusion, and cross-attention modules. Such a bridge may transmit contextual information encoded in H S H_{S} , but by itself it does not expose receiver-native anchors a R  ( e ) a_{R}(e) .

The resulting failure is an identifiability problem. Consider two contexts C e C_{e} and C e ′ C_{e^{\prime}} that differ primarily by replacing entity e e with another entity e ′ e^{\prime} while preserving the same relational structure. A continuous-only protocol is vulnerable when d ⁡ ( m cont  ( C e ) , m cont  ( C e ′ ) ) ≤ ϵ , d\bigl(m_{\mathrm{cont}}(C_{e}),m_{\mathrm{cont}}(C_{e^{\prime}})\bigr)\leq\epsilon, while the required receiver-native anchors remain distinct: a R  ( e ) ≠ a R  ( e ′ ) . a_{R}(e)\neq a_{R}(e^{\prime}). In this regime, the receiver may recover the context but struggles to determine which discrete receiver-vocabulary entity instantiates the answer. We refer to this as rare-token compression collapse : continuous representations preserve contextual semantics but lose reliable access to rare or specific lexical identities.

### 2.3 Representation Mismatch

Even if entity identity is preserved, sender activations cannot be directly treated as receiver activations. Let μ R ( ℓ ) \mu_{R}^{(\ell)} denote the distribution of token-level hidden states encountered by receiver layer ℓ \ell . For a projection P : ℝ d S → ℝ d R P:\mathbb{R}^{d_{S}}\rightarrow\mathbb{R}^{d_{R}} , dimension matching only ensures that P  H S PH_{S} lies in ℝ d R \mathbb{R}^{d_{R}} . It does not imply that projected sender states match the receiver’s native hidden-state distribution: P  H S ≁ μ R ( ℓ ) in general , PH_{S}\not\sim\mu_{R}^{(\ell)}\quad\text{in general}, where μ R ( ℓ ) \mu_{R}^{(\ell)} denotes the distribution of hidden states at the ℓ \ell -th layer of the receiver model. Directly injecting P  H S PH_{S} into the receiver residual stream or key-value cache can therefore introduce out-of-distribution activations.

A representation-compatible protocol should instead let the receiver consume sender information through its own hidden state: h R ′ ( ℓ ) = h R ( ℓ ) + g θ ( ℓ )  ( h R ( ℓ ) , H S ) , h_{R}^{\prime(\ell)}=h_{R}^{(\ell)}+g_{\theta}^{(\ell)}(h_{R}^{(\ell)},H_{S}), (7) where g θ ( ℓ ) g_{\theta}^{(\ell)} is a learned receiver-conditioned retrieval operator. This formulation preserves the receiver’s native computation while allowing access to sender-side contextual information.

## 3 XBridge : Entity-Grounded Latent Communication

The analysis in Section 2 identifies two challenges for heterogeneous architecture communication: continuous bridges struggle to preserve discrete entity identity (Section 2.2 ), and sender activations lie outside the receiver’s representation space (Section 2.3 ). We therefore decompose heterogeneous communication into two complementary mechanisms: Lexical Anchor Mapping (LAM, Section 3.2 ) provides discrete entity anchors, and a Latent Enrichment Bridge (LEB, Section 3.3 ) provides contextual enrichment. Both signals are produced by a single sender forward pass with no autoregressive decoding; sender and receiver remain frozen throughout.

### 3.1 XBridge Overview

We adopt the asymmetric communication setting formalized in Section 2 , following the protocol established by Shi et al. [5] , in which the sender observes a context and the receiver observes a question: the sender M S M_{S} processes a context document C C through a single forward pass, producing last-layer hidden states H S ∈ ℝ T C × d S H_{S}\in\mathbb{R}^{T_{C}\times d_{S}} and context token sequence c S = ( c 1 , … , c T C ) c_{S}=(c_{1},\dots,c_{T_{C}}) . The receiver M R M_{R} holds only the question Q Q and must generate an answer A A from whatever signal the sender communicates. XBridge transmits both H S H_{S} and c S c_{S} : LAM (Section 3.2 ) uses c S c_{S} , and LEB (Section 3.3 ) uses H S H_{S} .

### 3.2 Lexical Anchor Mapping (LAM) for Discrete Grounding

LAM addresses the entity grounding failure (Section 2.2 ) by placing the sender’s original context tokens, including entity mentions, as receiver-native lexical inputs. These lexical anchors provide the discrete basis that LEB requires to ground its contextual enrichment to specific entities (Section 3.4 ).

#### Cross-vocabulary mapping.