TransformerFAM: Feedback attention is working memory

arxiv:2404.09173

Original source · Paper JSON

LLMs: architectures, context, training · 2024-04-14

Cites (7) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4

Contents

• TransformerFAM: Feedback attention is working memory
• Abstract
• 1 Introduction
• Assumption 1 .
• 2 TransformerFAM
• 2.1 Block Sliding Window Attention (BSWA)
• 2.2 Feedback Attention Memory
• 3 Experiments
• 3.1 Training
• 3.1.1 Data
• 3.2 PassKey Retrieval
• 3.3 Long Context Tasks
• 4 Related Work
• 5 Conclusion
• Appendix A Architecture details
• A.1 Flan-PaLM architecture
• A.2 FAM hyperparameters
• Appendix B Additional details of TransformerFAM
• B.1 FAM initialization
• B.2 Input Length Extrapolation
• B.2.1 FAM Position Encoding
• B.2.2 Random Position Offset
• B.2.3 Random State Passing
• B.3 FAM illustrated
• B.4 Ablation Studies
• B.4.1 FAM length
• B.4.2 The number of previous FAM blocks
• B.4.3 FAM Position Encoding
• B.4.4 Random Position Offset
• B.4.5 Random State Passing
• B.4.6 Prompt vs. Prefix
• B.4.7 TransformerXL vs. TransformerBSWA
• B.4.8 Comparison with other methods
• Appendix C Don’t
• C.1 Beyond Block-Recurrent Transformers
• C.2 Feedback Memory Segment
• C.3 Static Summary Tokens
• C.4 Diversity Loss
• C.5 Reconstruction Loss
• Appendix D Experiments details
• D.1 PassKey Retrieval
• D.2 Long Context Tasks
• D.3 GPT-3 Tasks
• D.4 Complexity
• Appendix E Related Work
• Appendix F Attention Visualization
• Appendix G Limitations
• Appendix H Broader Impacts
• Instructions for reporting errors