Communicating Activations Between Language Model Agents β€” part 3/3

Cleaned-text code points [29698, 45158). Bibliography omitted here; source.md includes it.

Table 3 presents results on each of the seven reasoning benchmarks across various baselines and activation communication. Notably, while NLD consistently outperforms π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} - 3.2 \mathtt{3.2} - 𝟹 ​ 𝙱 \mathtt{3B} , it does not always display a performance improvement over π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} - 3.1 \mathtt{3.1} - 𝟾 ​ 𝙱 \mathtt{8B} ; but remarkably, AC consistently outperforms both single-model baselines . In fact, AC offers an up to 27.0 % 27.0\% improvement over NLD across six of the seven reasoning datasets. When applying 𝑾 \bm{W} to A A ’s activation before performing the replacement function, we see even further gains of 2.6 βˆ’ 50.0 % 2.6-50.0\% over vanilla AC for four of the seven datasets. We hypothesize that the benefits from the learned linear layer are less consistent across datasets because the subset of C4 data used to train 𝑾 \bm{W} likely contains text more semantically similar to some datasets than others, hence some datasets provide 𝑾 \bm{W} with out-of-distribution inputs which reduces performance compared to vanilla AC.

While we fix A A as the smaller model and B B as the larger model in Table 3 (so as to ensure decoding happens with the presumably more capable model), this need not be the case; swapping A A and B B yields results of 81.5 Β± 0.0 81.5\pm 0.0 and 61.0 Β± 4.8 61.0\pm 4.8 on Biographies and GSM8k respectively (without the linear layer). While these accuracies are lower than their non-swapped counterparts, notably they still are higher than both single-model baselines (and higher than NLD for Biographies); plus this is much more compute-efficient as the smaller model is now the one requiring the full instead of partial forward pass.

Note that we find AC outperforms NLD on 48 of the 57 datasets in the full MMLU benchmark ; complete MMLU results, as well as a suite of additional experiments, are shown in Appendix B .

#### Performance-compute tradeoff and generalization to different model scales

Thus far, we have been considering the absolute performance of AC with respect to NLD, for which our method attains state-of-the-art results; however the superiority of activations as a language for inter-LLM communication is further illustrated by AC’s larger ratio of performance improvement to added inference-time compute over individual LMs. Figure 3 displays the results of single models, AC, and NLD across model scales and suites within the π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} family on the Biographies dataset. Incoming arrows to AC and NLD nodes denote the base models between which communication occurred. Not only does AC consistently outperform both single-model baselines unlike NLD, but also notice that the slope of each black line is far greater than the slope of each gray line, indicating that AC consistently achieves greater increases in accuracy per additional unit of inference-time compute (normalized by the compute of a single forward pass through π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} - 3.2 \mathtt{3.2} - 𝟷 ​ 𝙱 \mathtt{1B} on the given prompt) compared to NLD.

#### Communication across model families

Table 4 displays results for AC between models from the πš€πš πšŽπš— \mathtt{Qwen} - 2.5 \mathtt{2.5} , π™ΆπšŽπš–πš–πšŠ \mathtt{Gemma} - 𝟸 \mathtt{2} , and π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} - 𝟹 \mathtt{3} families. We see that AC beats NLD across the board , and beats both individual models for 4 / 5 4/5 of the 6 6 model pairs on Biographies/GSM8k respectively β€”demonstrating the efficacy of AC irrespective of model architecture, size, tokenizer, and training data. Moreover, these results are obtained without training 𝑾 \bm{W} , meaning we do not need a separate projection layer between activation spaces to attain SOTA results, even for extremely distinct models! (We hypothesize this is because we are only replacing B B ’s last-token activation, hence B B can learn from A A without an extreme alteration to its activation distribution. An alternative explanation is to see this result as proof of the platonic representation hypothesis ( Huh et al., 2024 ) , which historical deep learning works have oft alluded to, including in the context of cross-model representation stitching ( Moschella et al., 2023 ; Kornblith et al., 2019 ) .)

## 5 Conclusion

We present a simple approach to enable effective and computationally efficient communication between language models by injecting information from the activations of one model into the activations of another during the forward pass. Salient features of this approach include: (i) Scales up LLMs on new tasks by leveraging existing, frozen LLMs along with zero additional task-specific parameters and data, (ii) Applies to diverse domains and settings, and (iii) Saves a substantial amount of compute .

There are some limitations to this method. First, when not using the learned model-specific mapping discussed in Section 3.1 , our method requires both models to have aligned embedding spaces, such that the activation of one model roughly retains its meaning in the other’s activation space (note that unlike past works such as Pham et al. (2024) we do not require shared tokenizers or aligned vocabularies, only aligned embeddings). While less restrictive than past works ( Pham et al., 2024 ) , this assumption is somewhat limiting, but can be relaxed when we let f f be the learned model-specific mapping; and in practice we find that even amongst different models in the π™»π™»πšŠπ™Όπ™° \mathtt{LLaMA} family, no such mapping is required for state-of-the-art results.

Second, this method requires access to embeddings and will not work with black-box API access; however exploring API-only approaches is highly limiting, and recent releases of powerful open-source models ( Dubey et al., 2024 ) merit the development of embedding-based techniques.

Third, while a concern might be the limited interpretability of communicating activations as opposed to natural language, we note the following. First, there is a fundamental tradeoff between interpretability and information preservation (as activations, by virtue of being much higher-dimensional than the space of natural language, allow proportionally higher-entropy communication) ( Pham et al., 2024 ) , which merits discussion beyond the scope of this work. But second, we actually posit that our method suggests a new avenue towards interpreting LM activations: β€œtranslating” activations based on the beliefs they induce as messages in listening agents, similar to the method put forward in Andreas et al. (2018) . We recognize this as a promising avenue for future research.

Additional directions of future work include using AC to allow large LMs to leverage small, tunable LMs as β€œknowledge bases” during decoding ( Lee et al., 2024 ) , as in collaborative decoding ( Shen et al., 2024 ) setups; and testing our approach on more complex coordination games (e.g., Lewis-style negotiation games ( Lewis et al., 2017 ) , Diplomacy).

## Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

## Acknowledgements

The authors are grateful to Jacob Andreas, Yoon Kim, and Sham Kakade for their valuable discussions and feedback.

## Appendix A Qualitative Results

## Appendix B Additional Experiments

### B.1 Modifying Activations of All Tokens

Recall that AC grafts the last-token layer- k k activation of A A into B B ’s last-token layer- j j activation. But is modifying just the last token activation enough to communicate information from A A to B B ?

Note that after applying masked attention in each of the previous Transformer layers, the last token activation of A A attends to all tokens before it, hence incorporating information from the entire sequence. Indeed, this must be the case for activation communication to recover the gap between the zero-communication and skyline setups on both coordination games, which (for Tip Sheets in particular) require information starting at the first few tokens of A A ’s prompt to be communicated.

To verify this empirically, we experiment with summing the activations of all tokens in the sequence rather than just the last (we cannot replace all tokens as this would just replace B B ’s layer- j j activation with A A ’s layer k k -activation). Results are shown in Table 5 .

Indeed, applying f f to all tokens decreases performance relative to applying f f to just the last token. Note that the fact performance generally decreases from f = f= πš›πšŽπš™πš•πšŠπšŒπšŽ \mathtt{replace} to f = f= πšœπšžπš– \mathtt{sum} , and further with all tokens, is expected. The high performance of AC with f = f= πš›πšŽπš™πš•πšŠπšŒπšŽ \mathtt{replace} means that the edited last-token activation 𝒃 \bm{b} retains some meaning in B B ’s activation space; it is less likely for this to be the case when f = f= πšœπšžπš– \mathtt{sum} (at the very least 𝒃 \bm{b} has norm roughly 2 Γ— 2\times that of B B ’s original last-token activation), and when doing this for all tokens we’d expect performance to decrease even further as now all activation vectors, not just the last, are out-of-distribution with respect to B B ’s activation space.

### B.2 Incorporating Chain-of-Thought Prompting

How does AC perform in relation to NLD in cases where A A might incur a long response (possibly with chain-of-thought for intermediate answer computation)? I.e., does AC lose out on the benefits of CoT?

First, note that we still reap the benefits of CoT when we sample a completion from B B after AC ( where B B gets all the information encoding A A ’s β€œbeliefs” about the prompt via AC, hence CoT on A A ’s side is not needed ). To verify this, we experiment with prompting A A with CoT, generating a full response, and then passing the layer- k k last-token activation of the CoT response to B B as part of AC. Results are shown in Table 6 .

Indeed, we empirically find our above intuition (in orange ) to hold, as there is no significant improvement over vanilla AC when generating from A A using CoT.

### B.3 Learning 𝑾 \bm{W} In-Distribution

Recall our reasoning about the AC ( 𝑾 ) (\bm{W}) results from Section 4.2 : β€œWe hypothesize that the benefits from the learned linear layer are less consistent across datasets because the subset of C4 data used to train 𝑾 \bm{W} likely contains text more semantically similar to some datasets than others, hence some datasets provide 𝑾 \bm{W} with out-of-distribution inputs which reduces performance compared to vanilla AC.”

Indeed, we verify this hypothesis by training 𝑾 \bm{W} on the GSM8k train set (to produce 𝑾 in dist \bm{W}_{\textrm{in dist}} ) and then evaluating with this task-specific linear layer on the GSM8k test set. Results are shown in Table 7 .

Indeed, learning 𝑾 \bm{W} in-distribution significantly boosts performance, confirming our hypothesis. Unfortunately we cannot run this experiment for the other datasets, as there is no in-distribution training data available for MMLU (we use all public data for testing).

Hence, this suggests that AC ( 𝑾 \bm{W} ) should unilaterally improve over vanilla AC if we choose a training set with good coverage across many tasks and distributions, such that there are sentences semantically similar to prompts across the span of downstream task datasets.

### B.4 Activation Space Similarity ∝ \propto AC Performance Gain

We conduct the following experiment: for each of the six pairs of models A , B A,B in the above experiment (see Table 4 ), we compute the increase in Biographies performance with AC relative to the average individual performance of A A and B B . We also compute the matrix analogue of the squared cosine similarity between the models’ activation spaces, βˆ₯ Y ⊀ ​ X βˆ₯ F 2 βˆ₯ X βˆ₯ F 2 ​ βˆ₯ Y βˆ₯ F 2 , \frac{\lVert Y^{\top}X\rVert_{F}^{2}}{\lVert X\rVert_{F}^{2}\,\lVert Y\rVert_{F}^{2}}, where X X is the matrix of A A ’s activations on 3072 3072 sentences from C4 (the same dataset used to train 𝑾 \bm{W} ), Y Y is the corresponding matrix for B B , and βˆ₯ β‹… βˆ₯ F \lVert\cdot\rVert_{F} denotes the Frobenius norm. This yields the plot in Figure 11 .

There is a clear positive correlation between the similarity of the activation distributions and the AC performance gain, as expected; the more aligned A A and B B ’s activation spaces are, the more semantically meaningful and useful the embedding we graft from A A to B B becomes.

### B.5 Communicating Activations Between Identical Models

Note that AC as described in Section 3.1 only supports communication between distinct models. We can extend AC to work for communication between identical models as follows: let A A and B B be instances of the same model. We can sample a completion from A A with temperature and graft the last-token layer- k k activation of the completion into B B at layer j j as part of the AC procedure. This still saves a substantial amount of compute over NLD between 2 model instances, showing our technique can apply to this setting. Table 8 shows the results of this experiment.

Indeed, while communication between multiple model instances doesn’t always show improvement over the single model itself (a well-known result from ( Du et al., 2023 ) ), AC matches/outperforms NLD on five of the seven datasets .

The intuition behind debate between multiple identical model instances is that sampling multiple completions (with temperature) from the same model yields diverse reasoning paths that can be recombined into a stronger final answer. The above experiment shows that the same intuition holds for ACβ€”we are sampling multiple times from the same model, but passing responses between agents via AC rather than as NL messages.

### B.6 Additional Rounds of Natural Language Debate

In Section 4.2 we fix NLD to 2 2 agents and 2 2 rounds, however we find in additional experiments that AC outperforms NLD even with additional rounds, highlighting the superiority and robustness of activations as an alternative β€œlanguage” for inter-LM communication. Results are shown in Table 9 ; we see that for 5 of the 7 reasoning benchmarks, AC beats NLD even with 3 3 - 4 4 rounds while using substantially less compute.

βˆ— Runs required too much compute

### B.7 Full MMLU Benchmark Results

Table 10 below displays complete results of both AC and NLD on the full MMLU benchmark. Notably, AC matches/outperforms NLD on 48/57 datasets, with substantially less compute used , indicating its superiority and robustness as an alternative β€œlanguage” for inter-LLM communication.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .