# XBridge : Entity-Grounded Latent Bridge for Heterogeneous LLM Communication ###### Abstract Heterogeneous multi-agent LLM systems, where agents are powered by different model families, can outperform homogeneous configurations by reducing redundant reasoning patterns. Yet existing communication protocols either operate through text, discarding the sender’s internal representations, or require architectural homogeneity for latent-level transfer. We identify the entity grounding problem in cross-architecture communication: cross-attention bridges that transfer continuous representations across different LLM families suffer from rare-token compression collapse, where entity identity is lost in the continuous bottleneck (bridge-only F1 ∼ \sim 30%). We propose XBridge , a decode-free communication protocol that addresses this through two mechanisms. Lexical Anchor Mapping (LAM) maps the sender’s original context tokens to the receiver’s vocabulary, providing discrete entity anchors. A Latent Enrichment Bridge (LEB) lets the receiver query the sender’s hidden states for contextual enrichment. The entity anchors ground the bridge’s contextual signals to specific entities through the receiver’s own self-attention. Across three model families (Llama, Qwen, and Mistral), seven benchmarks, and both communication directions, XBridge outperforms text-based communication on all seven tasks for each model pair while achieving 11 × \times lower latency, and in a same-architecture setting it also exceeds a KV-sharing baseline on six of seven tasks. LEB requires only 264M trainable parameters (3.8% of the receiver), is trained on a small balanced sample set, and adds negligible inference overhead. Our implementation is available at https://github.com/WooseongYang/XBridge . ## 1 Introduction Multi-agent systems (MAS) can improve the reasoning capabilities of large language models (LLMs) by distributing computation across multiple inference passes [ 1 ] . Recent findings show that when interacting agents share identical parameters and inductive biases, their interactions may degenerate into redundant majority voting rather than surfacing complementary reasoning [ 2 , 3 ] . A natural way to mitigate this redundancy is to use heterogeneous ensembles: configurations comprising distinct model families with diverse tokenizers, training corpora, architectures, and alignment procedures. Yet this representational diversity introduces a severe communication bottleneck: agents must exchange information across structurally incompatible latent spaces. Existing communication paradigms face an efficiency-fidelity trade-off. Text-based methods, such as natural language communication (NLComm) [ 4 ] , are architecture-agnostic but incur autoregressive decoding latency and compress the sender’s internal state into a textual summary. Latent-level methods avoid sender-side decoding, but existing approaches such as KV cache sharing [ 5 ] and hidden-state injection [ 6 ] typically assume architectural compatibility, including aligned hidden dimensions, attention layouts, positional encodings, or tokenizers. Learned continuous projections offer a possible route across model families [ 7 ] . However, we find that continuous projections alone degrade in a specific and practically important way: they can transfer contextual information without reliably preserving the entities to which that context refers. We define this degradation as the entity grounding problem . In heterogeneous LLM communication, a continuous bridge can transfer contextual information from the sender, but it struggles to preserve the discrete entity identities, such as names, numbers, or rare tokens, to which that context refers. We use entity fidelity to refer to preserving the correct lexical identity, and entity grounding to refer to binding the sender’s contextual signal to the corresponding receiver-native lexical anchor. In bridge-only experiments, the receiver recovers aspects of the context but often loses the exact identities of the participating entities. Homogeneous communication settings are less exposed to this problem because shared tokenizers and representational spaces provide implicit entity anchors. Heterogeneous communication therefore creates a demanding setting in which continuous contextual transfer and exact discrete entity fidelity must be maintained simultaneously. To address this bottleneck, we propose XBridge , a decode-free communication protocol that mitigates the entity grounding problem via dual channels. First, Lexical Anchor Mapping (LAM) deterministically translates the sender’s original context tokens into the receiver’s vocabulary, providing discrete entity anchors without autoregressive decoding. Second, a Latent Enrichment Bridge (LEB) uses gated cross-attention to let the receiver query the sender’s hidden states, supplying continuous contextual information that can be bound to these lexical anchors. By decoupling the preservation of discrete entity identity (the who and what ) from the transfer of continuous contextual knowledge (the how and why ), XBridge enables heterogeneous models to communicate without requiring textual decoding or direct latent-space compatibility. The bridge is lightweight, requiring only 264M trainable parameters (3.8% of the receiver), training on a small balanced sample set in under 10 minutes, and adding negligible inference overhead. In summary, our core contributions are: • We formulate the entity grounding problem in heterogeneous LLM communication and demonstrate rare-token compression collapse, a failure mode in which continuous bridges preserve contextual signals but lose discrete entity identity. • We propose XBridge , a decode-free dual-channel protocol that combines Lexical Anchor Mapping for receiver-native entity anchors with Latent Enrichment Bridge for sender-side contextual enrichment. To our knowledge, this is the first protocol to pair an explicit discrete-anchor channel with latent transfer for heterogeneous communication. • We show that XBridge improves heterogeneous communication across three model families and seven benchmarks, achieving 11 × \times lower latency than text-based communication, and that in a same-architecture setting it exceeds KV-cache sharing on 6 of 7 tasks. ## 2 The Entity Grounding Problem Heterogeneous communication requires satisfying three requirements. First, the receiver must recover receiver-native lexical identities for entities that appear in the sender context. Second, it must use contextual information encoded in the sender’s hidden states. Third, it must consume this information without treating sender activations as native receiver activations. We refer to these requirements as entity fidelity , entity grounding , and representation compatibility , respectively. We formalize heterogeneous LLM communication as information transfer between models with incompatible tokenizers, vocabularies, and hidden spaces. Let M i = ( τ i , V i , E i , F i , G i ) , i ∈ { S , R } , M_{i}=(\tau_{i},V_{i},E_{i},F_{i},G_{i}),\qquad i\in\{S,R\}, (1) denote a sender M S M_{S} and receiver M R M_{R} , where τ i \tau_{i} is the tokenizer, V i V_{i} the vocabulary, E i E_{i} the embedding map, F i F_{i} the transformer, and G i G_{i} the language-model head. The models are heterogeneous when V S ≠ V R , d S ≠ d R , F S ≢ F R , V_{S}\neq V_{R},\qquad d_{S}\neq d_{R},\qquad F_{S}\not\equiv F_{R}, (2) where architectural mismatch may include differences in depth, attention layout, normalization, positional encoding, tokenizer design, or training distribution. The sender observes a context C C , while the receiver observes a question Q Q and must generate an answer A A . The sender performs a single forward pass: c S = τ S ​ ( C ) = ( c 1 , … , c T C ) , H S = F S ​ ( E S ​ ( c S ) ) ∈ ℝ T C × d S . c_{S}=\tau_{S}(C)=(c_{1},\ldots,c_{T_{C}}),\qquad H_{S}=F_{S}(E_{S}(c_{S}))\in\mathbb{R}^{T_{C}\times d_{S}}. (3) Unless otherwise stated, H S H_{S} denotes the sender’s last-layer hidden states. A communication protocol produces a sender-side message m S = π S ​ ( c S , H S ) , m_{S}=\pi_{S}(c_{S},H_{S}), which the receiver consumes to generate p π ​ ( A ∣ C , Q ) = p R ​ ( A ∣ Q ; Γ θ ​ ( Q , m S ) ) . p_{\pi}(A\mid C,Q)=p_{R}(A\mid Q;\Gamma_{\theta}(Q,m_{S})). Here, Γ θ \Gamma_{\theta} denotes the receiver-side mechanism for consuming the message. Text protocols communicate strings or token sequences, latent protocols communicate continuous states, and hybrid protocols communicate both m S = ( m disc ​ ( C ) , m cont ​ ( C ) ) m_{S}=\bigl(m_{\mathrm{disc}}(C),m_{\mathrm{cont}}(C)\bigr) . ### 2.1 Entity Fidelity Let ℰ ⁡ ( C ) \mathcal{E}(C) denote the set of entity mentions in C C , including names, numbers, dates, and other rare or semantically specific spans. For an entity e ∈ ℰ ⁡ ( C ) e\in\mathcal{E}(C) , define its receiver-native lexical realization as a R ​ ( e ) = τ R ​ ( e ) ∈ V R ∗ , a_{R}(e)=\tau_{R}(e)\in V_{R}^{\ast}, (4) where V R ∗ V_{R}^{\ast} is the set of finite token sequences over the receiver vocabulary. We call a R ​ ( e ) a_{R}(e) an entity anchor . Entity fidelity is the requirement that the receiver assigns high probability to the correct receiver-native anchor. For a gold answer entity e ⋆ e^{\star} , we measure entity fidelity by the receiver-side rank rank π ​ ( e ⋆ , C , Q ) = rank v ∈ V R ​ [ ℓ R ​ ( v ∣ Q ; Γ θ ​ ( Q , m S ) ) ] v ∈ a R ​ ( e ⋆ ) , \mathrm{rank}_{\pi}(e^{\star};C,Q)=\mathrm{rank}_{v\in V_{R}}\left[\ell_{R}(v\mid Q;\Gamma_{\theta}(Q,m_{S}))\right]_{v\in a_{R}(e^{\star})}, (5) where ℓ R \ell_{R} is the receiver logit induced by G R G_{R} . For multi-token entities, we use the first answer token as a diagnostic proxy. A protocol has high entity fidelity when this rank is low. Entity fidelity can also be characterized interventionally. Let C e → e ′ C^{e\rightarrow e^{\prime}} denote the context obtained by replacing every occurrence of e e with another entity e ′ e^{\prime} of the same semantic type. A protocol is entity-sensitive if A ^ π ​ ( C , Q ) = e ⟹ A ^ π ​ ( C e → e ′ , Q ) = e ′ . \hat{A}_{\pi}(C,Q)=e\quad\Longrightarrow\quad\hat{A}_{\pi}(C^{e\rightarrow e^{\prime}},Q)=e^{\prime}. (6) This condition separates entity identity from relational structure. If the relation is preserved but the answer does not follow the substituted entity, the protocol does not preserve entity fidelity. ### 2.2 Entity Grounding Failure Entity fidelity concerns whether the receiver outputs the correct entity. Entity grounding concerns whether the sender’s contextual signal is bound to the correct receiver-native entity anchor. A protocol may transfer contextual information while still leaving that information ungrounded to the correct entity in V R V_{R} . A continuous bridge can be written abstractly as r θ ( ℓ ) = B θ ( ℓ ) ​ ( H S , h R ( ℓ ) ) ∈ ℝ T R × d R , r_{\theta}^{(\ell)}=B_{\theta}^{(\ell)}(H_{S},h_{R}^{(\ell)})\in\mathbb{R}^{T_{R}\times d_{R}}, where h R ( ℓ ) ∈ ℝ T R × d R h_{R}^{(\ell)}\in\mathbb{R}^{T_{R}\times d_{R}} is the receiver hidden-state sequence at layer ℓ \ell with the receiver sequence length T R T_{R} . This abstraction covers projections, adapters, hidden-state fusion, and cross-attention modules. Such a bridge may transmit contextual information encoded in H S H_{S} , but by itself it does not expose receiver-native anchors a R ​ ( e ) a_{R}(e) . The resulting failure is an identifiability problem. Consider two contexts C e C_{e} and C e ′ C_{e^{\prime}} that differ primarily by replacing entity e e with another entity e ′ e^{\prime} while preserving the same relational structure. A continuous-only protocol is vulnerable when d ⁡ ( m cont ​ ( C e ) , m cont ​ ( C e ′ ) ) ≤ ϵ , d\bigl(m_{\mathrm{cont}}(C_{e}),m_{\mathrm{cont}}(C_{e^{\prime}})\bigr)\leq\epsilon, while the required receiver-native anchors remain distinct: a R ​ ( e ) ≠ a R ​ ( e ′ ) . a_{R}(e)\neq a_{R}(e^{\prime}). In this regime, the receiver may recover the context but struggles to determine which discrete receiver-vocabulary entity instantiates the answer. We refer to this as rare-token compression collapse : continuous representations preserve contextual semantics but lose reliable access to rare or specific lexical identities. ### 2.3 Representation Mismatch Even if entity identity is preserved, sender activations cannot be directly treated as receiver activations. Let μ R ( ℓ ) \mu_{R}^{(\ell)} denote the distribution of token-level hidden states encountered by receiver layer ℓ \ell . For a projection P : ℝ d S → ℝ d R P:\mathbb{R}^{d_{S}}\rightarrow\mathbb{R}^{d_{R}} , dimension matching only ensures that P ​ H S PH_{S} lies in ℝ d R \mathbb{R}^{d_{R}} . It does not imply that projected sender states match the receiver’s native hidden-state distribution: P ​ H S ≁ μ R ( ℓ ) in general , PH_{S}\not\sim\mu_{R}^{(\ell)}\quad\text{in general}, where μ R ( ℓ ) \mu_{R}^{(\ell)} denotes the distribution of hidden states at the ℓ \ell -th layer of the receiver model. Directly injecting P ​ H S PH_{S} into the receiver residual stream or key-value cache can therefore introduce out-of-distribution activations. A representation-compatible protocol should instead let the receiver consume sender information through its own hidden state: h R ′ ( ℓ ) = h R ( ℓ ) + g θ ( ℓ ) ​ ( h R ( ℓ ) , H S ) , h_{R}^{\prime(\ell)}=h_{R}^{(\ell)}+g_{\theta}^{(\ell)}(h_{R}^{(\ell)},H_{S}), (7) where g θ ( ℓ ) g_{\theta}^{(\ell)} is a learned receiver-conditioned retrieval operator. This formulation preserves the receiver’s native computation while allowing access to sender-side contextual information. ## 3 XBridge : Entity-Grounded Latent Communication The analysis in Section 2 identifies two challenges for heterogeneous architecture communication: continuous bridges struggle to preserve discrete entity identity (Section 2.2 ), and sender activations lie outside the receiver’s representation space (Section 2.3 ). We therefore decompose heterogeneous communication into two complementary mechanisms: Lexical Anchor Mapping (LAM, Section 3.2 ) provides discrete entity anchors, and a Latent Enrichment Bridge (LEB, Section 3.3 ) provides contextual enrichment. Both signals are produced by a single sender forward pass with no autoregressive decoding; sender and receiver remain frozen throughout. ### 3.1 XBridge Overview We adopt the asymmetric communication setting formalized in Section 2 , following the protocol established by Shi et al. [5] , in which the sender observes a context and the receiver observes a question: the sender M S M_{S} processes a context document C C through a single forward pass, producing last-layer hidden states H S ∈ ℝ T C × d S H_{S}\in\mathbb{R}^{T_{C}\times d_{S}} and context token sequence c S = ( c 1 , … , c T C ) c_{S}=(c_{1},\dots,c_{T_{C}}) . The receiver M R M_{R} holds only the question Q Q and must generate an answer A A from whatever signal the sender communicates. XBridge transmits both H S H_{S} and c S c_{S} : LAM (Section 3.2 ) uses c S c_{S} , and LEB (Section 3.3 ) uses H S H_{S} . ### 3.2 Lexical Anchor Mapping (LAM) for Discrete Grounding LAM addresses the entity grounding failure (Section 2.2 ) by placing the sender’s original context tokens, including entity mentions, as receiver-native lexical inputs. These lexical anchors provide the discrete basis that LEB requires to ground its contextual enrichment to specific entities (Section 3.4 ). #### Cross-vocabulary mapping. Given the sender’s context tokens c S = ( c 1 , … , c T C ) c_{S}=(c_{1},\dots,c_{T_{C}}) obtained by tokenizing C C with M S M_{S} ’s tokenizer, each token c i ∈ 𝒱 S c_{i}\in\mathcal{V}_{S} is mapped to the receiver’s vocabulary 𝒱 R \mathcal{V}_{R} via a deterministic, training-free mapping function ϕ : 𝒱 S → 𝒱 R ∗ \phi:\mathcal{V}_{S}\to\mathcal{V}_{R}^{*} , precomputed offline once per sender–receiver tokenizer pair. For tokens whose surface string appears in both vocabularies, ϕ \phi performs a direct ID-to-ID lookup. For the remainder, a string fallback decodes the token to its surface string and re-tokenizes it with the receiver’s tokenizer, producing one or more receiver tokens expanded in place. The mapping is lossless and adds negligible latency ( < < 1 ms); detailed statistics and examples are in Appendix A.1 . #### Receiver-native lexical embeddings. The mapped token IDs are converted to receiver embeddings e ctx = ( E R ​ [ ϕ ⁡ ( c 1 ) ] , … , E R ​ [ ϕ ⁡ ( c T C ) ] ) e_{\text{ctx}}=(E_{R}[\phi(c_{1})],\dots,E_{R}[\phi(c_{T_{C}})]) via the receiver’s frozen embedding matrix E R E_{R} , omitting expansion from re-tokenization for clarity. These embeddings are prepended to the receiver’s question input, following the input prepend convention used in soft-token communication [ 8 ] : x R = [ e ctx ; E R ​ ( Q ) ] x_{R}=[e_{\text{ctx}}\;;\;E_{R}(Q)] (8) By using E R E_{R} as a lookup table rather than training a separate projection, transmitted tokens are guaranteed to lie in the same space as the receiver’s own input tokens, ensuring that pretrained self-attention patterns apply directly to the transferred context. Unlike prior methods that require the sender to autoregressively decode a summary [ 4 , 8 ] , LAM transfers the full context without decoding, eliminating both latency and information loss from fixed-length compression. ### 3.3 Latent Enrichment Bridge (LEB) for Context Adaptation LAM preserves entity identity but transmits only the sender’s raw tokens, not the sender’s contextual understanding of those tokens. The sender’s last-layer hidden states H S H_{S} encode information integrated across all context positions through the sender’s full transformer stack, information that raw token embeddings cannot convey. We use the last layer because it produces the most globally-integrated representations, as confirmed by a layer sweep in Appendix B.1 . However, H S H_{S} lies in the sender’s representation space, which the receiver has never encountered during training (Section 2.3 ). Directly injecting these representations, whether by prepending hidden states or replacing KV caches, places out-of-distribution vectors into the receiver’s processing pipeline and degrades performance. Cross-attention provides a natural solution: rather than injecting sender representations into the receiver, the receiver queries the sender in its own native space. This receiver-driven mechanism has three stages, illustrated in the bridge panel of Figure 3 . #### Projections. We insert M = 4 M{=}4 gated cross-attention modules into the receiver, one at each of layers ℓ ∈ { 6 , 13 , 20 , 27 } \ell\in\{6,13,20,27\} out of 28, following the insertion density of Flamingo [ 9 ] . We find four modules optimal; fewer provide insufficient capacity while more overfit with limited training data (Appendix B.1 ). Each module has its own learnable parameters ( ∼ {\sim} 66M each, ∼ {\sim} 264M total, 3.8% of the receiver). Both models remain frozen; only the bridge modules are trained. At each insertion layer ℓ \ell , the receiver’s hidden state h R ( ℓ ) ∈ ℝ d R h_{R}^{(\ell)}\in\mathbb{R}^{d_{R}} is projected into a query, while the sender’s hidden states H S H_{S} are projected into keys and values: Q ( ℓ ) \displaystyle Q^{(\ell)} = W Q ( ℓ ) ⋅ LN R ( ℓ ) ​ ( h R ( ℓ ) ) , K ( ℓ ) = W K ( ℓ ) ⋅ LN S ( ℓ ) ​ ( H S ) , V ( ℓ ) = W V ( ℓ ) ⋅ LN S ( ℓ ) ​ ( H S ) \displaystyle=W_{Q}^{(\ell)}\cdot\mathrm{LN}_{R}^{(\ell)}(h_{R}^{(\ell)}),\quad K^{(\ell)}=W_{K}^{(\ell)}\cdot\mathrm{LN}_{S}^{(\ell)}(H_{S}),\quad V^{(\ell)}=W_{V}^{(\ell)}\cdot\mathrm{LN}_{S}^{(\ell)}(H_{S}) (9) where W Q ( ℓ ) ∈ ℝ d R × d R W_{Q}^{(\ell)}\in\mathbb{R}^{d_{R}\times d_{R}} , W K ( ℓ ) , W V ( ℓ ) ∈ ℝ d R × d S W_{K}^{(\ell)},W_{V}^{(\ell)}\in\mathbb{R}^{d_{R}\times d_{S}} , and LN R ( ℓ ) \mathrm{LN}_{R}^{(\ell)} , LN S ( ℓ ) \mathrm{LN}_{S}^{(\ell)} are separate layer normalizations for the receiver and sender. The projection matrices simultaneously handle the dimensional mismatch ( d S → d R d_{S}\to d_{R} ), serving as the learned translation between representation spaces. All four modules query the same H S H_{S} , but each has its own parameters, enabling different layers to extract different aspects of the sender’s representation. #### Cross-attention. The receiver’s query selects relevant sender positions through scaled dot-product attention: A ( ℓ ) = softmax ⁡ ( Q ( ℓ ) ​ K ( ℓ ) ⊤ / d k ) ⋅ V ( ℓ ) A^{(\ell)}=\mathrm{softmax}\!\left(Q^{(\ell)}K^{(\ell)\top}/\sqrt{d_{k}}\right)\cdot V^{(\ell)} (10) This is the key advantage over injection-based alternatives: the receiver decides what information to retrieve based on its current processing state at layer ℓ \ell , rather than receiving a fixed representation regardless of what it needs. #### Gated residual. The cross-attention output is added to the receiver’s hidden state via a gated residual connection, preserving the receiver’s pretrained representations while incorporating bridge information: h R ′ ( ℓ ) = h R ( ℓ ) + tanh ⁡ ( α ( ℓ ) ) ⋅ A ( ℓ ) h_{R}^{\prime(\ell)}=h_{R}^{(\ell)}+\tanh(\alpha^{(\ell)})\cdot A^{(\ell)} (11) where h R ′ ( ℓ ) h_{R}^{\prime(\ell)} is the updated hidden state that continues through the receiver’s subsequent layers. We initialize α ( ℓ ) = 1.0 \alpha^{(\ell)}=1.0 ( tanh ⁡ ( 1.0 ) ≈ 0.76 \tanh(1.0)\approx 0.76 ) so that the bridge contributes immediately, unlike Flamingo’s zero initialization which requires the model to discover the bridge signal during training. With our limited training budget, warm initialization converges faster (Appendix B.1 ). The bridge is trained with standard next-token prediction loss on the answer tokens: ℒ ( θ ) = − ∑ t = 1 | A | log P R ( a t ∣ a < t , e ctx , Q ; θ bridge ) \mathcal{L}(\theta)=-\sum_{t=1}^{|A|}\log P_{R}(a_{t}\mid a_{