Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

arxiv:2404.07143

Original source · Paper JSON

LLMs: architectures, context, training · 2024-04-10

Cites (6) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
• Abstract
• 1 Introduction
• 2 Background
• 3 Method
• 3.1 Infini-attention
• 3.1.1 Scaled Dot-product Attention
• 3.1.2 Compressive Memory
• 3.2 Memory and Effective Context Window
• 4 Experiments
• 4.1 Implementation details
• 4.2 Long-context Language Modeling
• 4.3 LLM Continual Pre-training
• 5 Related Work
• 6 Conclusion
• Acknowledgments
• Appendix A Additional Training Details
• Appendix B Passkey Retrieval Task
• Instructions for reporting errors