Communicating Activations Between Language Model Agents

arxiv:2501.14082

Original source · Paper JSON

LLMs: architectures, context, training · 2025-01-23

Cites (6) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Communicating Activations Between Language Model Agents
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Work
• Multi-agent communication
• Activation engineering
• Model composition and grafting
• 3 Communicating Activations Between Language Models
• 3.1 Method
• 3.2 Compute Analysis
• 3.3 Why should this work?
• 4 Experiments
• 4.1 Multi-player coordination games
• 4.2 Reasoning Benchmarks
• Baselines
• Models
• Datasets
• Metrics
• Comprehensive evaluation with the 𝙻𝙻𝚊𝙼𝙰 \mathtt{LLaMA} family
• Performance-compute tradeoff and generalization to different model scales
• Communication across model families
• 5 Conclusion
• Impact Statement
• Acknowledgements
• Appendix A Qualitative Results
• Appendix B Additional Experiments
• B.1 Modifying Activations of All Tokens
• B.2 Incorporating Chain-of-Thought Prompting
• B.3 Learning 𝑾 \bm{W} In-Distribution
• B.4 Activation Space Similarity ∝ \propto AC Performance Gain
• B.5 Communicating Activations Between Identical Models
• B.6 Additional Rounds of Natural Language Debate
• B.7 Full MMLU Benchmark Results
• Instructions for reporting errors