xLSTM: Extended Long Short-Term Memory

arxiv:2405.04517

Original source · Paper JSON

LLMs: architectures, context, training · 2024-05-07

Cites (5) · Cited by (2)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12
13. Part 13
14. Part 14
15. Part 15
16. Part 16
17. Part 17
18. Part 18
19. Part 19
20. Part 20
21. Part 21
22. Part 22
23. Part 23
24. Part 24
25. Part 25
26. Part 26
27. Part 27
28. Part 28
29. Part 29
30. Part 30

Contents

• xLSTM: Extended Long Short-Term Memory
• Abstract
• 1 Introduction
• 2 Extended Long Short-Term Memory
• 2.1 Review of the Long Short-Term Memory
• 2.2 sLSTM
• New Memory Mixing.
• 2.3 mLSTM
• 2.4 xLSTM Architecture
• xLSTM Blocks.
• xLSTM Architecture.
• 2.5 Memory and Speed Considerations
• 3 Related Work
• Linear Attention.
• State Space Models.
• Recurrent Neural Networks.
• Gating.
• Covariance Update Rule.
• Most Related.
• Residually Stacking Architectures.
• 4 Experiments
• 4.1 Synthetic Tasks and Long Range Arena
• Test of xLSTM’s Exponential Gating with Memory Mixing.
• Test of xLSTM’s Memory Capacities on Associative Recall Tasks.
• Test of xLSTM’s Long Context Capabilities on Long Range Arena.
• 4.2 Method Comparison and Ablation Study
• Comparing xLSTM to Other Methods.
• Ablation Studies.
• 4.3 xLSTM as Large Language Model
• Sequence Length Extrapolation.
• Validation Perplexity and Downstream Tasks.
• Performance on PALOMA Language Tasks.
• Scaling Laws.
• 5 Limitations
• 6 Conclusion
• Acknowledgements
• Appendix A Extended Long Short-Term Memory
• A.1 Vanilla Long Short-Term Memory Formulation: Vector Notation
• A.2 sLSTM
• Proof of Equivalence for sLSTM Stabilized Version.
• A.3 mLSTM
• Recurrent mLSTM Backward Pass.
• Parallel mLSTM Forward Pass.
• Parallel mLSTM Backward Pass.
• A.4 Detailed Block Structure
• Appendix B Experiments
• Training Setup.
• Nearest Neighbor Search Task.
• Wikitext-103 Rare Token Prediction.
• B.1 Synthetic Tasks and Long Range Arena
• B.1.1 Test of xLSTM’s Exponential Gating with Memory Mixing.
• Experiment Setup.
• Additional Formal Language Results.
• Individual Task Description.
• B.1.2 Test of xLSTM’s Memory Capacities on Associative Recall Tasks.
• Why Multi-Query Associative Recall for Memory Tests of LLM Architectures.
• Experiment Setup.
• Extended Results.
• B.1.3 Test of xLSTM’s Long Range Capabilities on the Long Range Arena.
• Experiment Setup.
• Results.
• B.2 Method Comparison and Ablation Study on SlimPajama (15B)
• General Training Procedure.
• Details on Comparison to Other Methods.
• Details on Training Precision for Baselines.
• General Details on Ablation Studies.
• Additional Ablation Study on Matrix Memory.
• Details on new xLSTM Components Ablation Study.
• Details on Gating Technique Ablation Study.
• Details on Scaling Experiments.
• B.3 xLSTM Large Language Models – SlimPajama300B
• General Training Procedure.
• Details on Downstream Evaluation.
• Details on PALOMA.
• Appendix C Detailed Results on PALOMA Language Model Evaluation
• Instructions for reporting errors