Your Transformer is Secretly Linear

arxiv:2405.12250

Original source · Paper JSON

LLMs: architectures, context, training · 2024-05-19

Cites (2) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Your Transformer is Secretly Linear
• Abstract
• 1 Introduction
• 2 Related Work
• Structure-based pruning
• 3 Analysis of Pretrained Architectures
• 3.1 Linearity Score
• 3.2 Linearity Dynamics at Pretraining and Fine-tuning
• 4 Improving Linearity with Regularized Pretraining
• 5 Exploiting Linearity for Pruning
• 6 Conclusion
• 7 Limitations
• 8 Ethics Statement
• Appendix A Error Distribution by Layers
• Instructions for reporting errors