XBridge: Entity-Grounded Latent Bridge for Heterogeneous LLM Communication — part 2/4

Cleaned-text code points [15563, 31223). Bibliography omitted here; source.md includes it.

Given the sender’s context tokens c S = ( c 1 , … , c T C ) c_{S}=(c_{1},\dots,c_{T_{C}}) obtained by tokenizing C C with M S M_{S} ’s tokenizer, each token c i ∈ 𝒱 S c_{i}\in\mathcal{V}_{S} is mapped to the receiver’s vocabulary 𝒱 R \mathcal{V}_{R} via a deterministic, training-free mapping function ϕ : 𝒱 S → 𝒱 R ∗ \phi:\mathcal{V}_{S}\to\mathcal{V}_{R}^{*} , precomputed offline once per sender–receiver tokenizer pair. For tokens whose surface string appears in both vocabularies, ϕ \phi performs a direct ID-to-ID lookup. For the remainder, a string fallback decodes the token to its surface string and re-tokenizes it with the receiver’s tokenizer, producing one or more receiver tokens expanded in place. The mapping is lossless and adds negligible latency ( < < 1 ms); detailed statistics and examples are in Appendix A.1 .

#### Receiver-native lexical embeddings.

The mapped token IDs are converted to receiver embeddings e ctx = ( E R ​ [ ϕ ⁡ ( c 1 ) ] , … , E R ​ [ ϕ ⁡ ( c T C ) ] ) e_{\text{ctx}}=(E_{R}[\phi(c_{1})],\dots,E_{R}[\phi(c_{T_{C}})]) via the receiver’s frozen embedding matrix E R E_{R} , omitting expansion from re-tokenization for clarity. These embeddings are prepended to the receiver’s question input, following the input prepend convention used in soft-token communication [ 8 ] : x R = [ e ctx ; E R ​ ( Q ) ] x_{R}=[e_{\text{ctx}}\;;\;E_{R}(Q)] (8) By using E R E_{R} as a lookup table rather than training a separate projection, transmitted tokens are guaranteed to lie in the same space as the receiver’s own input tokens, ensuring that pretrained self-attention patterns apply directly to the transferred context. Unlike prior methods that require the sender to autoregressively decode a summary [ 4 , 8 ] , LAM transfers the full context without decoding, eliminating both latency and information loss from fixed-length compression.

### 3.3 Latent Enrichment Bridge (LEB) for Context Adaptation

LAM preserves entity identity but transmits only the sender’s raw tokens, not the sender’s contextual understanding of those tokens. The sender’s last-layer hidden states H S H_{S} encode information integrated across all context positions through the sender’s full transformer stack, information that raw token embeddings cannot convey. We use the last layer because it produces the most globally-integrated representations, as confirmed by a layer sweep in Appendix B.1 . However, H S H_{S} lies in the sender’s representation space, which the receiver has never encountered during training (Section 2.3 ). Directly injecting these representations, whether by prepending hidden states or replacing KV caches, places out-of-distribution vectors into the receiver’s processing pipeline and degrades performance. Cross-attention provides a natural solution: rather than injecting sender representations into the receiver, the receiver queries the sender in its own native space. This receiver-driven mechanism has three stages, illustrated in the bridge panel of Figure 3 .

#### Projections.

We insert M = 4 M{=}4 gated cross-attention modules into the receiver, one at each of layers ℓ ∈ { 6 , 13 , 20 , 27 } \ell\in\{6,13,20,27\} out of 28, following the insertion density of Flamingo [ 9 ] . We find four modules optimal; fewer provide insufficient capacity while more overfit with limited training data (Appendix B.1 ). Each module has its own learnable parameters ( ∼ {\sim} 66M each, ∼ {\sim} 264M total, 3.8% of the receiver). Both models remain frozen; only the bridge modules are trained. At each insertion layer ℓ \ell , the receiver’s hidden state h R ( ℓ ) ∈ ℝ d R h_{R}^{(\ell)}\in\mathbb{R}^{d_{R}} is projected into a query, while the sender’s hidden states H S H_{S} are projected into keys and values: Q ( ℓ ) \displaystyle Q^{(\ell)} = W Q ( ℓ ) ⋅ LN R ( ℓ ) ​ ( h R ( ℓ ) ) , K ( ℓ ) = W K ( ℓ ) ⋅ LN S ( ℓ ) ​ ( H S ) , V ( ℓ ) = W V ( ℓ ) ⋅ LN S ( ℓ ) ​ ( H S ) \displaystyle=W_{Q}^{(\ell)}\cdot\mathrm{LN}_{R}^{(\ell)}(h_{R}^{(\ell)}),\quad K^{(\ell)}=W_{K}^{(\ell)}\cdot\mathrm{LN}_{S}^{(\ell)}(H_{S}),\quad V^{(\ell)}=W_{V}^{(\ell)}\cdot\mathrm{LN}_{S}^{(\ell)}(H_{S}) (9) where W Q ( ℓ ) ∈ ℝ d R × d R W_{Q}^{(\ell)}\in\mathbb{R}^{d_{R}\times d_{R}} , W K ( ℓ ) , W V ( ℓ ) ∈ ℝ d R × d S W_{K}^{(\ell)},W_{V}^{(\ell)}\in\mathbb{R}^{d_{R}\times d_{S}} , and LN R ( ℓ ) \mathrm{LN}_{R}^{(\ell)} , LN S ( ℓ ) \mathrm{LN}_{S}^{(\ell)} are separate layer normalizations for the receiver and sender. The projection matrices simultaneously handle the dimensional mismatch ( d S → d R d_{S}\to d_{R} ), serving as the learned translation between representation spaces. All four modules query the same H S H_{S} , but each has its own parameters, enabling different layers to extract different aspects of the sender’s representation.

#### Cross-attention.

The receiver’s query selects relevant sender positions through scaled dot-product attention: A ( ℓ ) = softmax ⁡ ( Q ( ℓ ) ​ K ( ℓ ) ⊤ / d k ) ⋅ V ( ℓ ) A^{(\ell)}=\mathrm{softmax}\!\left(Q^{(\ell)}K^{(\ell)\top}/\sqrt{d_{k}}\right)\cdot V^{(\ell)} (10) This is the key advantage over injection-based alternatives: the receiver decides what information to retrieve based on its current processing state at layer ℓ \ell , rather than receiving a fixed representation regardless of what it needs.

#### Gated residual.

The cross-attention output is added to the receiver’s hidden state via a gated residual connection, preserving the receiver’s pretrained representations while incorporating bridge information: h R ′ ( ℓ ) = h R ( ℓ ) + tanh ⁡ ( α ( ℓ ) ) ⋅ A ( ℓ ) h_{R}^{\prime(\ell)}=h_{R}^{(\ell)}+\tanh(\alpha^{(\ell)})\cdot A^{(\ell)} (11) where h R ′ ( ℓ ) h_{R}^{\prime(\ell)} is the updated hidden state that continues through the receiver’s subsequent layers. We initialize α ( ℓ ) = 1.0 \alpha^{(\ell)}=1.0 ( tanh ⁡ ( 1.0 ) ≈ 0.76 \tanh(1.0)\approx 0.76 ) so that the bridge contributes immediately, unlike Flamingo’s zero initialization which requires the model to discover the bridge signal during training. With our limited training budget, warm initialization converges faster (Appendix B.1 ).

The bridge is trained with standard next-token prediction loss on the answer tokens: ℒ ( θ ) = − ∑ t = 1 | A | log P R ( a t ∣ a < t , e ctx , Q ; θ bridge ) \mathcal{L}(\theta)=-\sum_{t=1}^{|A|}\log P_{R}(a_{t}\mid a_{<t},e_{\text{ctx}},Q;\theta_{\text{bridge}}) (12) where e ctx e_{\text{ctx}} is computed via LAM (Section 3.2 ). Since the sender is frozen, its outputs ( H S H_{S} and context token IDs) are precomputed and cached, reducing training to under 10 minutes on a single GPU on a balanced set of 587 samples drawn across the seven task domains. Once trained, the bridge is fixed and applied to all tasks at inference time without further adaptation (Section 4 ). Training data composition is analyzed in Appendix B.2 . The bridge adapts to different model pairs by adjusting only the projection dimensions; we train separate instances for each pair using the same protocol.

### 3.4 Entity-Grounded Bridge Integration

The two mechanisms are integrated through the receiver’s own processing, without any explicit fusion layer. LAM places entity mentions (e.g., “Christopher Nolan”) in the receiver’s input as discrete embeddings. As these propagate through the receiver’s layers via self-attention, they shape the receiver’s hidden states h R ( ℓ ) h_{R}^{(\ell)} at every depth, embedding entity information into the representations that LEB modules use as queries. This is the grounding mechanism: when LEB at layer ℓ \ell computes Q ( ℓ ) = W Q ( ℓ ) ⋅ LN R ( ℓ ) ​ ( h R ( ℓ ) ) Q^{(\ell)}=W_{Q}^{(\ell)}\cdot\mathrm{LN}_{R}^{(\ell)}(h_{R}^{(\ell)}) , the query already encodes which entities are present and where they appear, enabling the cross-attention to focus on the sender positions that carry relevant contextual information about those entities. Without LAM, h R ( ℓ ) h_{R}^{(\ell)} reflects only the question, producing queries too vague to extract entity-specific signals from the sender, so the LEB conveys contextual signals while entity identity is lost in the continuous bottleneck, which is why LEB-only performance drops to ∼ \sim 30% F1 (Section 4.3 ). Without LEB, the receiver has the sender’s context tokens but lacks the sender’s contextual understanding of their relationships. Together, LEB’s contextual enrichment is anchored to specific entities rather than floating in continuous space. We verify this complementarity quantitatively in Section 4.3 .

## 4 Experiments

### 4.1 Experimental Setup

#### Models.

We evaluate communication across three model families to test cross-architecture generality: Llama-3.1-8B-Instruct [ 10 ] (Llama), Qwen2.5-7B-Instruct [ 11 ] (Qwen), and Mistral-7B-Instruct-v0.3 (Mistral). These families differ in architecture (hidden dimension, layer count, attention structure), tokenizer design, and training data, ensuring that improvements are not an artifact of a single architecture pairing. We evaluate three heterogeneous pairs: Llama → \to Qwen, Qwen → \to Llama, and Mistral → \to Qwen. We also verify that the framework generalizes to same-architecture settings (Appendix C.1 ). To test whether larger senders produce richer representations for LEB, we additionally evaluate sender size scaling with Qwen2.5-{1.5B, 7B, 14B}-Instruct → \to Llama.

#### Tasks and protocol.

We evaluate on seven benchmarks: HotpotQA [ 12 ] , MuSiQue [ 13 ] , QASPER [ 14 ] , 2WikiMultihopQA (2Wiki) and MultiFieldQA-en (MFldQA) from LongBench [ 15 ] , Countries and Tipsheets [ 5 ] . The sender receives the context document only and the receiver receives the question only, the asymmetric protocol established by Shi et al. [5] , which we apply identically to every method compared here. For NLComm baselines, the sender generates a 128-token summary via greedy decoding. We report F1 score. Task details are in Appendix E .

#### Compared methods.

We compare against natural language communication (NLComm) [ 4 ] , where the sender generates a text summary and relays it to the receiver. We distinguish NLComm hetero (cross-family) and NLComm homo (same family). We also report two single-model reference points measured with the receiver alone. NoComm, where the receiver sees only the question, is a lower reference indicating what the receiver achieves without communication. FullComm, where the receiver reads the full context directly, removes the communication problem rather than solving it, and serves as an upper reference in the sense of the Skyline used by Shi et al. [5] . Neither is a competing communication method, and both are excluded from the best-per-task comparison. Same-architecture methods, including NLComm homo and KVComm [ 5 ] which shares KV caches between models of the same family, are compared in Appendix C.1 .

### 4.2 Main Results

Table 1 presents heterogeneous communication results across all three model pairs and seven benchmarks. XBridge outperforms NLComm hetero on all 7 tasks across all three model pairs, with average gains of +21.4 pp (Llama → \to Qwen), +14.3 pp (Qwen → \to Llama), and +20.9 pp (Mistral → \to Qwen). The improvements are largest on tasks requiring multi-hop reasoning or world knowledge (Countries, 2WikiMQA, MuSiQue), where LEB provides contextual enrichment that text summaries cannot convey. On average, XBridge surpasses even FullComm across all three model pairs (63.2 vs. 55.2 for Llama → \rightarrow Qwen, 61.9 vs. 61.2 for Qwen → \rightarrow Llama, 65.0 vs. 55.2 for Mistral → \rightarrow Qwen). Since LAM places the full mapped context in the receiver’s input, XBridge sees what FullComm sees and adds the sender’s hidden states through LEB, so the comparison is not one of indirect against direct access. The same-architecture setting isolates this: with Qwen2.5-7B sending to itself, LAM reduces to an identity mapping, so the receiver’s input matches FullComm’s and only LEB differs, and XBridge homo {}_{\text{homo}} still leads by 7.9 pp on average (Appendix C.1 ). With identical models and identical input, the gain comes from LEB carrying the sender’s integrated representation that the receiver would otherwise recompute. The optimal sender varies by task: Mistral → \to Qwen achieves the highest scores on HotpotQA, QASPER, and MuSiQue, while Llama → \to Qwen leads on MFldQA and 2WikiMQA. This complementarity suggests that selecting the sender based on task characteristics can further improve performance. In the same-architecture setting, XBridge also exceeds KVComm [ 5 ] on 6 of 7 tasks (Appendix C.1 ).

### 4.3 Ablation: Both Anchors and Enrichment Are Necessary

The gains reported above rely on both LAM and LEB working together. To understand how much each contributes, we remove them individually on HotpotQA (Table 5 ). XBridge without LAM ( XBridge -LAM {}_{\text{-LAM}} ) drops to 30.3% F1, close to NoComm: LEB receives continuous signals but without entity anchors these signals are not bound to specific entities, confirming the entity grounding failure identified in Section 2.2 . XBridge without LEB ( XBridge -LEB {}_{\text{-LEB}} ) reaches 56.5%, providing entity grounding but no contextual enrichment beyond what the tokens themselves contain. The full XBridge achieves 78.8%, exceeding the FullComm. The +22.3 pp gain from adding LEB to LAM contrasts with the modest +5.7 pp from LEB-only over NoComm, demonstrating that grounding is the enabling condition for LEB effectiveness. LEB contribution scales with task reasoning complexity: from +35.8 pp on factual inference requiring world knowledge (Countries) to +2.0 pp on structured extraction (Tipsheets), with per-task breakdown in Appendix C.2 .

### 4.4 Bridge Mechanism and Role Separation

The ablation confirms that both mechanisms are necessary. A deeper question follows: does each carry a distinct type of information, or do they redundantly encode the same signal? To test this, we run an entity perturbation experiment on 100 HotpotQA samples, replacing every occurrence of the answer entity in the context with an unrelated entity of the same type (e.g., “Christopher Nolan” → \to “Bong Joon Ho”), then independently control which version each mechanism receives (Table 2 ).

Swapping LEB alone does not change the output entity: conditions A and D produce identical outputs (45 original, 0 swapped, 55 neither). The “neither” category contains outputs that mention neither the original nor the substituted entity, such as a date, another entity, or an unrelated span; we therefore interpret the experiment by comparing the original-versus-swapped direction among target-entity outputs. Under this interpretation, swapping LAM shifts the output toward the substituted entity regardless of which LEB version is used; conditions B and C produce nearly identical counts (37 swapped, 4–5 original, 58–59 neither). These results indicate a clear role separation: LAM supplies which entities appear in the output, while LEB supplies how the receiver reasons about them.

Figure 4 examines LEB’s effect on the receiver’s internal state across five communication conditions on 196 HotpotQA samples. NLComm methods attend most heavily to answer positions (2.0–2.7 × \times above average) yet achieve the worst token rank because paraphrased text replaces exact entity mentions. XBridge is the only method that produces positive cosine alignment between generation and answer entity representations, translating to correct-token probability of 0.752 vs. 0.635–0.662 for all other conditions (94% of individual samples, Appendix D.1 ).

### 4.5 Beyond Accuracy: Latency, Composability, and Sender Scaling