Memorizing Transformers

arxiv:2203.08913

Original source · Paper JSON

LLMs: architectures, context, training · 2022-03-16

Cites (0) · Cited by (6)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Memorizing Transformers
• Abstract
• 1 Introduction
• 2 Related Work
• 3 Method
• 3.1 k k NN-augmented Attention Layer
• 3.2 Distributional Shift
• 3.3 Approximate k k NN
• 4 Experiments
• 4.1 Datasets
• arXiv Math
• Github
• Formal Math – Isabelle
• C4(4K + + )
• PG-19
• 4.2 Experimental Method
• 4.3 Effect of external memory
• Adding external memory results in substantial gains across datasets and architectures,
• Increasing the size of the memory increases the benefit of the memory.
• 4.4 Scaling to larger models
• 4.5 Finetuning on Larger Memories
• Finetuning on a larger memory.
• Finetuning a non-memory model to use memory
• 4.6 Information Retrieval Patterns
• Which tokens show a benefit from memory?
• What information is being looked up?
• 5 Conclusion
• Acknowledgments
• Ethics
• Reproducibility
• Appendix A Length of inputs
• A.1 Ablation studies
• Multiple k k NN layers.
• k k NN layer index
• Number of neighbors
• Random seeds
• Appendix B What does the model retrieve from memory?
• Retrieving citation names
• Retrieving function names from the codebase
• B.1 More Retrieving examples in formal theorem proving corpus
• Example 1
• Example 2
• Example 3
• Example 4
• Example 5
• Example 6
• Instructions for reporting errors