Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

arxiv:2006.16236

Original source · Paper JSON

LLMs: architectures, context, training · 2020-06-29

Cites (1) · Cited by (15)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Abstract
• 1 Introduction
• 2 Related Work
• 2.1 Efficient Transformers
• 2.2 Understanding Self-Attention
• 2.3 Linearized softmax
• 3 Linear Transformers
• 3.1 Transformers
• 3.2 Linearized Attention
• 3.2.1 Feature Maps and Computational Cost
• 3.3 Causal Masking
• 3.3.1 Gradient Computation
• 3.3.2 Training and Inference
• 3.4 Transformers are RNNs
• 4 Experiments
• 4.1 Synthetic Tasks
• 4.1.1 Convergence Analysis
• 4.1.2 Memory and Computational Requirements
• 4.2 Image Generation
• 4.2.1 MNIST
• 4.2.2 CIFAR-10
• 4.3 Automatic Speech Recognition
• 5 Conclusions
• Acknowledgements
• Appendix A Gradient Derivation
• Appendix B Training Evolution
• Appendix C Image Generation Throughput Discussion
• C.1 Stateful softmax attention
• C.2 Equalizing the batch size
• Appendix D Qualitative Results on Image Generation
• Instructions for reporting errors