Hyena Hierarchy: Towards Larger Convolutional Language Models

arxiv:2302.10866

Original source · Paper JSON

LLMs: architectures, context, training · 2023-02-21

Cites (0) · Cited by (3)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5

Contents

• Hyena Hierarchy: Towards Larger Convolutional Language Models
• Abstract
• 1 Introduction
• The 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} hierarchy
• Narrowing the capabilities gap
• Scaling in language and vision
• Toward much longer context
• 2 Preliminaries and Related Work
• 2.1 Explicit and Implicit Convolutions
• Fast Methods for Convolutions
• 2.2 The Self-Attention Operator
• Remark 2.1 .
• Subquadratic Operators
• 3 Hyena: Definition and Properties
• 3.1 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} Recurrences
• Remark 3.1 .
• 3.2 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} Matrices
• Remark 3.2 ( 𝖧𝗒𝖾𝗇𝖺 \sf Hyena generalizes 𝖧𝟥 \sf H3 and 𝖦𝖲𝖲 \sf GSS .) .
• Remark 3.3 .
• 3.3 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} Filters
• Specializing filters in Hyena
• Preserving causality
• Proposition 3.1 (Causal Hyenas) .
• Efficiency
• 3.4 Hyena Algorithm
• Proposition 3.2 (Computational Complexity) .
• 4 Experiments
• 4.1 Shrinking the gap on in-context learning
• How to parametrize long convolutions
• Pushing sequence length to the limit
• Comparing operators
• Generality of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operators and filters
• 4.2 Language Modeling
• 4.3 Downstream Evaluation
• 4.4 Benchmarking
• 4.5 Large-Scale Image Classification
• 5 Discussion and Conclusion
• Acknowledgments
• Appendix A Experimental Details
• A.1 Mechanistic Design Synthetic Benchmarks
• Long convolution comparisons:
• Operator comparisons:
• A note on Transformer performance
• A.2 Language Modeling
• WikiText103:
• The Pile:
• PG-19
• Architectures
• FLOP computation
• A.3 Downstream Evaluation
• SuperGLUE:
• Models
• LAMBADA:
• A.4 Image Classification
• a
• CIFAR-10:
• Appendix B Theoretical Results and Details
• B.1 Proofs
• Proof of Proposition 3.1
• Proof.
• B.2 Analysis of Data-Controlled Mechanisms
• Isolating the surrogate attention matrix
• Operator decomposition of the surrogate attention matrix
• Appendix C Discussion and Additional Results
• Vocabulary size scaling
• Single layer recall
• C.1 Learning Arithmetic
• Appendix D Samples and Visualizations
• D.1 Hyena Matrices
• D.2 Hyena Filters
• D.3 Positional Encoding and Filters Initialization
• D.4 Downstream Examples
• MultiRC
• Instructions for reporting errors