Cache-to-Cache: Direct Semantic Communication Between Large Language Models — part 3/4
Cleaned-text code points [31799, 47725). Bibliography omitted here; source.md includes it.
C2C Choice . In our design, we adopt terminal alignment , as it provides a simpler and more direct layer mapping strategy that empirically performs slightly better in our experiments.
#### A.1.2 Tokenization Alignment
For dialogue inputs, we first apply the chat template of each tokenizer, which produces a sequence consisting of alternating sections of (1) template tokens and (2) message tokens . These two types of sections are handled differently during alignment.
Template sections . Template tokens are structural markers (e.g., role delimiters, formatting tokens) that differ across tokenizers and carry no semantic content. To preserve sequence consistency without introducing unnecessary distortions, these sections are aligned by simple length padding: the shorter side is padded with <pad> tokens until both tokenizers’ sequences are of equal length.
Message sections . Message tokens correspond to the actual textual content of user or assistant dialogs. Each target model token in a message section is decoded into its string form and re-encoded using the source model tokenizer. Special tokens (e.g., <pad> , <eos> ) are mapped directly if possible; otherwise, the source model’s unknown token is used. For regular tokens, if the re-encoding produces a single source model token, a direct one-to-one mapping is established. If multiple source model tokens are produced (a one-to-many case), one of the two selection strategies is applied: (1) first-occurrence selection : choose the first source model token from the candidate set, yielding a deterministic and computationally efficient mapping. (2) Maximal-coverage selection : decode each candidate token, compute its string length, and select the longest; this heuristic aims to preserve maximal surface correspondence with the original target model token.
C2C choice We observed that the two selection strategies generally produce very similar results, with more than 80% of sequences yielding identical alignments across strategies. Based on this observation, we empirically adopt Maximal-coverage selection as the default strategy to reduce the risk of losing information in one-to-many tokenization cases.
Through this design, template sections are aligned structurally via padding, while message sections are aligned semantically at the token level, ensuring robust correspondences between target model and source model representations in chat-formatted inputs.
#### A.1.3 Fuser Architecture
Beyond the C2C fuser, we also examined a more complex yet potentially more powerful variant, which we denote as C2C-C (Complex) . The main complexity comes from the introduction of an additional projection stage: instead of directly concatenating Sharer and Receiver caches as in C2C, Sharer cache is first projected into the receiver’s dimensionality through a 3-layer MLP. The concatenated representation is then processed along two familiar routes—feature fusion and dynamic weighting—to yield the final S&R cache.
The main experiment results are presented in Table 9 . Note that we fix the maximum response length to 8 tokens and the maximum communication length to 256 tokens in this experiment to reduce evaluation cost. C2C-C attains stronger performance than the default C2C, suggesting that increasing the architectural sophistication of fuser can further amplify the benefits of C2C communication. In this table, we also report Performance Gap Recovered (PGR) ( Ong et al., 2024 ) metric, which quantifies how much of the performance gap between a weak and a strong model is recovered. Nevertheless, the focus of this work is on introducing the C2C paradigm itself. For this purpose, we adopt a simple yet effective fuser design, leaving systematic investigation of more elaborate architectures to future work.
### A.2 Additional Experimental Results
#### A.2.1 Cache Enrichment Detail
In Table 10 we show the effect of single-layer cache enrichment. Layer 4 and 16 benefit from the cache enrichment approach by replacing the KV-Cache with the few-shot one, while cache enrichment on other layers shows performance degradation.
#### A.2.2 Strong-to-Weak Communication
Table 11 reports the results on LongBenchV1 when pairing the weak receiver Qwen3-0.6B with a much stronger sharer, Qwen3-4B, under different input lengths. Across all length regimes, C2C consistently outperforms both the receiver alone and the T2T baseline. On average, C2C achieves a 51.01% PGR over the weak-to-strong gap. These results demonstrate that in strong-to-weak settings, C2C can effectively transfer the stronger model’s contextual understanding, yielding notable gains for the weaker receiver.
We additionally evaluated the strong-to-weak setting (Qwen3-0.6B as receiver and Qwen3-4B as sharer) on other benchmarks beyond LongBenchV1. The detailed results are provided in Section A.1.3 , Table 9 .
#### A.2.3 Accuracy Breakdown
We analyze where the accuracy gains of C2C come from by using Venn diagrams on the MMLU-Redux benchmark, as illustrated in Figure 7 . For this analysis, we use the C2C-C variant introduced in Section A.1.3 , as it has the potential to achieve stronger performance and provides a clearer breakdown of where C2C ’s accuracy originates.
Models with comparable capacity . When the Receiver (Qwen3-0.6B) and the Sharer (Qwen2.5-Math-1.5B-Instruct, denoted as Qwen2.5-Math-1.5B) have comparable overall capacity but complementary strengths, C2C not only inherits part of the Sharer’s ability but also solves additional questions by integrating understanding from both models.
Models with disparate capacity . When the Sharer (Qwen3-4B) is substantially stronger than the Receiver (Qwen3-0.6B), C2C tends to integrate more of the stronger model’s understanding. Quantitatively, in the disparate-capacity case (Figure 7(b) ), among the questions that the Sharer can answer correctly, C2C also answers 72.11% correctly. In contrast, in the comparable-capacity case (Figure 7(a) ), C2C succeeds on only 50.97%.
Subcategory breakdown . We analyze the accuracy gains on different subcategories using the radar plot and histogram plot shown in Figure 8 , Figure 9 , and Figure 10 . The result shows that for the Qwen2.5-0.5B and Qwen3-0.6B model pair, C2C outperforms T2T on 12 out of 17 total categories. Using C2C in history, law, and chemistry can bring an additional 7, 7.6, and 7.5 accuracy improvement compared to T2T. For the Qwen3-4B and Qwen3-0.6B model pair, C2C outperforms T2T on all 17 categories. Using T2T in engineering results in a 1% performance drop, showing the case where text communication between models fails to help the receiver models achieve better performance. At the same time, C2C better utilizes the 4B model’s contextual understanding of the problems and achieves a 24% increase in accuracy.
#### A.2.4 Progressive Behavior
To investigate the impact of fused KV-Cache proportion on the accuracy of the receiver model, we gradually added the proportion of fused KV-Cache derived from the sharer to the receiver model before generating outputs. Specifically, former and latter refer to progressively replacing the receiver’s KV-Cache with the fused KV-Cache from front to back and back to front, respectively. We observe that the overall accuracy first decreases and then increases as the replacement ratio grows. The performance reduction may stem from the gap between training and testing, where only the full receiver KV-Cache is used during training. When the fused proportion goes up to over 50%, the performance of C2C continues to increase with respect to the proportion, reflecting the progressive benefits of C2C. Note that projecting using the latter cache generally has a larger impact than projecting the former, since it is closer to the final response.
### A.3 Additional Experiment Setup
#### A.3.1 Cache Enrichment
We conducted Oracle experiments with Qwen3-0.6B and Qwen3-4B to examine how KV-Cache enrichment influences model performance. The evaluation was performed on MMLU-Redux. The few-shot examples are selected from MMLU while excluding overlaps with MMLU-Redux to ensure fairness. To probe different ways of applying cache enrichment, we compared four cache enrichment strategies: All-layer Cache Enrichment (apply cache enrichment on all layers), Single-Layer Cache Enrichment (apply cache enrichment only on single layers), Selective Cache Enrichment - Best (select n layers that have the highest accuracy according to Single-Layer Cache Enrichment), Selective Cache Enrichment - Worst (select n layers that have the lowest accuracy according to Single-Layer Cache Enrichment). All methods utilized Few-Shot–optimized KV-Caches while maintaining the same cache length as Zero-Shot, enabling a controlled evaluation of cache enrichment and its layer-specific effects.
#### A.3.2 Cache Transformation
We employed the MMLU-Redux dataset to train a 3-layer MLP that maps the last layer KV-Cache in the source LLM (Qwen3-4B) to the last layer of the target LLM (Qwen3-0.6B). In this oracle experiment, we adopt the MSE loss of the projected KV-Cache and the target KV-Cache instead of the next-token prediction loss, as we aim to test whether the representation space can be transformed using a neural network. The Key Cache and Value Cache are first concatenated on the hidden space dimension, then put for MSE calculation. A detailed calculation of loss is shown in the following code: source_prefilled = source_model.forward(prefill_input_id) target_prefilled = target_model.forward(prefill_intput_id) source_k, source_v = source_prefilled.past_key_values[-1] target_k, target_v = target_prefilled.past_key_values[-1] project_k = k_projector(source_k) project_v = v_projector(source_v) mseloss(torch.dstack([project_k, project_v]), torch.dstack([target_k, target_v]))
For visualization, 300 samples were randomly selected from the dataset. The source, target, and transformed KV-Cache were all projected into two-dimensional space using t-SNE, allowing us to examine the alignment of representations between the two models. For t-SNE generation, we set perplexity to 50 and max iterations to 1000.
#### A.3.3 Query-level routing
Query-level routing aims to improve the performance–efficiency trade-off by dynamically assigning harder queries to a stronger LLM. Following prior work, we adopt a matrix factorization framework. Query embeddings are obtained from the OpenAI text-embedding-3-small encoder, while model embeddings are taken from pretrained vectors of gpt-4-1106-preview and mixtral-8x7b-instruct-v0.1. These embeddings are used to compute a strong win rate score for each query, which reflects its relative difficulty. Queries are then ranked by this score. For each evaluated model pair, we define the strong model as the one achieving higher standalone benchmark accuracy and the weak model as the lower-performing one. Queries in the upper half of the ranking are routed to the strong model, while those in the lower half are routed to the weak model.
#### A.3.4 Evaluation Method
Main evaluation . We evaluate C2C on four multiple-choice benchmarks: OpenBookQA, MMLU-Redux, ARC-Challenge, and C-Eval. For MMLU-Redux, we exclude questions annotated with the error type no correct answer . For all evaluations, we adopt a deterministic generation configuration without sampling, using greedy decoding to ensure reproducibility. Specifically, we use Non-CoT prompts, following the unified format described in Section A.3.6 . Model outputs are then matched to the correct option labels to compute accuracy. To control evaluation cost, we set the maximum response length to 64 tokens unless otherwise specified, where the response refers to the final answer generated by the Receiver, since the base models do not always follow instructions, and longer limits would substantially increase inference time. For the T2T setting, we additionally set the maximum communication length to 256 tokens, where the communication refers to the messages passed from the Sharer to the Receiver.
LongBench evaluation . We evaluate C2C on the LongBench-E dataset, which comprises a total of 13 individual datasets. The prompts and evaluation procedures use the official LongBench settings, with a maximum output length of 2,048 tokens.
#### A.3.5 C2C Training
Training data . (1) Performance experiment. The fuser was trained on the OpenHermes-2.5 Dataset with a maximum sequence length of 2,048 tokens. Training used 500,000 samples for one epoch with a macro batch size of 256, corresponding to 1,929 total training steps.
(2) Scaling sequence lengths experiment. The fuser was trained on the LongBench-E benchmark with a maximum sequence length of 12,000 tokens. The data was randomly split into 3/4 for training and 1/4 for evaluation by data index, ensuring independence between training and evaluation. Training used 1,896 samples for one epoch with a macro batch size of 16, corresponding to 118 total training steps.
(3) Scaling model sizes and different model combinations experiment. The fuser was trained on the auxiliary_train split of the MMLU dataset with a maximum sequence length of 1,024 tokens. Training used 15,000 samples for one epoch with a macro batch size of 128, corresponding to 116 total training steps.
Training scheme . All experiments were conducted with a fixed random seed of 42 to ensure reproducibility. Unless otherwise noted, the training configuration was as follows: optimization employed a learning rate of 1 × 10 − 4 1\times 10^{-4} with a linear scheduler and a 10% warmup ratio, a weight decay of 0.01, and a maximum gradient norm of 1. The temperature was linearly annealed from 1.0 to 0.001 across the total number of training steps. Layer alignment was configured with the last aligned scheme across all experiments. The tokenization alignment was applied only when the paired models employed different tokenizers, in which case the longest strategy was used. For data preparation, each dataset was partitioned into a training split (99%) and a small held-out validation split (1%). The validation split was not used for model updates but was monitored during training to report evaluation loss.
#### A.3.6 Evaluation Prompts
Text 1 presents the prompts used for the main evaluation on multiple-choice datasets. Text 2 is the alternative non-CoT version used for response diagnosis in Appendix A.4.4 . Text 3 provides the prompt for the Sharer model in the T2T evaluation; the Receiver model uses the same prompt as in the C2C setting. Text 4 shows the prompt used in the cache enrichment experiment. For the zero-shot method, no shots are included in the prompt. The few-shot method uses exactly the same prompt as Text 4. For Oracle methods, we adopt the few-shot prompt but remove the KV-Cache associated with the shots after the forward pass. The prompt for LongBench evaluation follows its official configuration, which varies across the different sub-datasets.
### A.4 Additional Analysis
#### A.4.1 Effective Rank
We list the definition of effective rank that was proposed by Roy and Vetterli (2007) here as a reference. For a matrix W that has size M × N M\times N , the singular value decomposition of it can be expressed as W = U Σ V W=U\Sigma V and the singular values σ = ( σ 1 , σ 2 , … , σ m i n ( M , N ) ) T \sigma=(\sigma_{1},\sigma_{2},...,\sigma_{min(M,N)})^{T} are the non-negative diagonal entries of the matrix Σ \Sigma . The singular value distribution is denoted as: p i = σ i ‖ σ ‖ 1 p_{i}=\frac{\sigma_{i}}{\|\sigma\|_{1}} (6) Denote the Shannon Entropy as: H ( p 1 , p 2 , … , p m i n ( M , N ) ) = − ∑ i = 1 m i n ( M , N ) p i log p i H(p_{1},p_{2},...,p_{min(M,N)})=-\sum_{i=1}^{min(M,N)}p_{i}\log p_{i} (7) The effective rank is defined as: e r a n k ( W ) = e − ∑ i = 1 m i n ( M , N ) p i l o g p i erank(W)=e^{-\sum_{i=1}^{min(M,N)}p_{i}logp_{i}} (8)