Unlimiformer: Long-Range Transformers with Unlimited Length Input

arxiv:2305.01625

Original source · Paper JSON

LLMs: architectures, context, training · 2023-05-02

Cites (3) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• Unlimiformer: Long-Range Transformers with Unlimited Length Input
• Abstract
• 1 Introduction
• 2 Unlimiformer
• 2.1 Encoding
• 2.2 Retrieval-augmented Cross-Attention
• 2.3 Attention reformulation
• 3 Training Unlimiformer
• 3.1 Low (additional-) Cost Training Methods: Applying Unlimiformer at validation or test-time only
• 3.2 Long-range Training Methods: Applying Unlimiformer at training time
• 4 Experimental Settings
• 4.1 Datasets
• Metrics
• 4.2 Baselines
• 5 Results
• 5.1 Long Document Summarization
• Low-cost training
• Long-range training
• 5.2 Book Summarization
• 6 Analysis
• Is the long input really needed?
• 7 Related Work
• 8 Conclusions
• 9 Limitations
• Acknowledgments
• Appendix A Training details
• Appendix B WikiSum scraping
• Appendix C Evaluation details
• Appendix D Computational Cost
• GPU-time
• Memory usage
• Appendix E Validation Results
• Appendix F Sample Outputs
• Appendix G Additional Related Work
• Instructions for reporting errors