Hyena Hierarchy: Towards Larger Convolutional Language Models β€” part 3/5

Cleaned-text code points [31126, 46922). Bibliography omitted here; source.md includes it.

We repeat our associative recall experiment, this time benchmarking different 2 2 layer models rather than changing the convolution parametrization: an order 2 2 Hyena, GSS ( Mehta et al., 2022 ) , H3 ( Dao et al., 2022c ) , AFT-conv ( Zhai et al., 2021 ) , RWKV ( Peng, 2021 ) , and a standard GPT ( Brown et al., 2020 ) using FlashAttention ( Dao et al., 2022b ) . As shown in Table 4.2 , 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} is the only operator able to solve the task. Our results challenge the observation that only Transformers are capable of challenging in-context learning.

Surprisingly, rankings of model performance at a fixed sequence length on The Pile are consistent with rankings on aggregate scores on our synthetics (Appendix C ).

#### Generality of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operators and filters

𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operators and filters can also applied successfully beyond language tasks. We experiment on sequential CIFAR, where pixels are flattened as a sequence, and use the same operator defined for language. We reach the accuracy of standard S4 ( Gu et al., 2021 ) with same model size ( 91 % 91\% ). In Section 4.5 and Appendix A , we discuss larger-scale image classification experiments with Hyena.

### 4.2 Language Modeling

Next, we verify the scaling of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} on autoregressive language modeling. We evaluate the perplexity on WikiText103 (Table 4.4 ) and The Pile (Table 4.4 ). On the The Pile , we train different models for 5 , 10 , 15 5,10,15 billion tokens (different runs), adjusting the learning rate scheduler. 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} is the first attention-free, convolution architecture to match GPT quality with a 20 % ~20\% 10 10 10 The FLOP reduction consists in the non-parametric FLOPs of π–²π–Ύπ—…π–Ώπ– π—π—π–Ύπ—‡π—π—‚π—ˆπ—‡ \sf SelfAttention devoted to attention matrix computation. The ratio of parametric to non-parametric FLOPs (and hence the gains) depend on the ratio of model width D D and sequence length L L used in training. reduction in total FLOPs. Preliminary scaling laws are shown in Figure 4.2 , collecting the training runs at 5 , 10 , 15 5,10,15 billion tokens. Each curve represents a different training run. In Appendix A , we provide results on the PG-19 long-range benchmark ( Rae et al., 2019 ) .

### 4.3 Downstream Evaluation

We perform a downstream evaluation on SuperGLUE ( Wang et al., 2019 ) tasks. We compare 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} (trained for 137 137 billion tokens) with the best available pre-trained attention-free model, RWKV ( Peng, 2021 ) (trained for 332 332 billion tokens), and a reference GPTNeo ( Black et al., 2021 ) (trained for 300 300 billion tokens) of the same size. Tables 4.5 and 4.6 summarize the results. 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} performs similarly to other models despite having been trained on less than half the number of total tokens. We observe 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} to display characteristic few-shot capabilities of standard Transformers, with some tasks e.g., MultiRC seeing a lift of more than 20 % 20\% accuracy over zero-shot when the model is provided additional prompts as context. The improvements are more noticeable in generation tasks, where the additional prompts can instruct the model on how it should be responding to the questions. We report an additional downstream evaluation on the LAMBADA task ( Paperno et al., 2016 ) in Appendix A .

### 4.4 Benchmarking

We benchmark runtime of an order 2 2 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operator compared to attention and FlashAttention layers ( Dao et al., 2022b ) . 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} uses a fused CUDA kernel to perform π–₯π–₯π–³π–’π—ˆπ—‡π— {\sf FFTConv} ( Dao et al., 2022c ) . We set batch size to 64 64 and measure runtime (in milliseconds). Results are provided in Figure 4.3 . 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} speedups reach 100 Γ— 100\times at sequence length 64 64 K. Crossover points for 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} and attention is at length 2048 2048 , and for 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} and FlashAttention is between 4096 4096 and 8196 8196 . Despite the absolute reduction in FLOPs, speedups are achieved only on longer sequences when the gap grows sufficiently large. This occurs because hardware utilization of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} is lower than FlashAttention. We expect the gap between theoretical maximum speedup to shrink with improved implementations of π–₯π–₯π–³π–’π—ˆπ—‡π— {\sf FFTConv} and specialized hardware.

### 4.5 Large-Scale Image Classification

Finally, we demonstrate the potential of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} as a general deep learning operator by applying it to image classification. On ImageNet , we drop-in replace attention layers in the Vision Transformer (ViT) ( Dosovitskiy et al., 2020 ) with the 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operator (without changes from its language counterpart) and match performance with ViT. We also show that using smaller image patches boosts performance in both attention and 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} . Since this results in longer sequence lengths, we expect 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} to outperform in speed as patches get more fine-grained approaching pixel-level. On CIFAR-2D, we test a 2D version of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} long convolution filters in a standard convolutional architecture, which improves on the 2D long convolutional model S4ND ( Nguyen et al., 2022 ) in accuracy with a 8 % 8\% speedup and 25% fewer parameters. See Appendix A.4 for additional vision architectures and training procedure details.

## 5 Discussion and Conclusion

In this work, we introduced an attention-free drop-in replacement to the core building block of many large-scale language models. 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operators are a recurrence of gating and implicitly parametrized long convolutions, can be evaluated efficiently in subquadratic time, and can learn in-context on very long sequences. On The Pile , deep stacks of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operators constitute one of the first attention-free, convolutional architectures to match perplexity and downstream performance of Transformers with a significant reduction in training compute. Our promising results at the sub-billion parameter scale suggest that attention may not be all we need, and that simpler subquadratic designs such as 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} , informed by a set of simple guiding principles and evaluation on mechanistic interpretability benchmarks, may form the basis for efficient large models. We are excited about what new capabilities Hyena opens up as we scale and optimize the inference speed of these models.

## Acknowledgments

We would like to thank Karan Goel, Albert Gu, Avanika Narayan, Khaled Saab, Michael Zhang, Elliot Epstein and Sabri Eyuboglu for helpful discussion and feedback on earlier drafts, and Together Computer and Crusoe for providing the compute used to train models in this paper. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); US DEVCOM ARL under No. W911NF-21-2-0251 (Interactive Human-AI Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1-2480: Understanding and Applying Non-Euclidean Geometry in Machine Learning; N000142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), Department of Defense (DoD) through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program, and members of the Stanford DAWN project: Facebook, Google, and VMWare. This work is supported by NSF (1651565), AFOSR (FA95501910024), ARO (W911NF-21-1-0125), ONR, DOE (DE-SC0022222), CZ Biohub, and Sloan Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.

## Appendix A Experimental Details

An implementation of 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} can be found at this link .

### A.1 Mechanistic Design Synthetic Benchmarks

Our synthetic reasoning are inspired by mechanistic interpretability ( Elhage et al., 2021 ) , in-context learning (ICL) ( Garg et al., 2022 ) and language model benchmarking ( Liang et al., 2022 ) research. The evaluation revolves around 4 4 main tasks: β€’ Associative recall: Each string is produced by concatenating key-value tuples from a different random dictionary. This test verifies whether a model is able to extract right value given a key as prompt, effectively applying a data-controlled shift (delay).

β€’ Majority voting and counting: Testing if a model can densely activate its data-controlled matrix i.e., through many non-zero entries (consider the string ’ a a a a a a a a a a b β†’ \rightarrow a ’).

β€’ ICL of linear functions: Verifying whether a model can perform ICL on real-valued inputs. Prompts are generated as x 1 , w k ​ x 1 , … , x n β†’ w k ​ x n x_{1},w^{k}x_{1},\dots,x_{n}\rightarrow w^{k}x_{n} , where both x k x_{k} and w k ∈ R n o w^{k}\in R^{n_{o}} are sampled from a normal distribution.

β€’ Arithmetic: Basic capability check.

For each task, we train models using the hyperparameters shown in Table A.1 . We consider increasing settings of difficulty controlled by sequence length, spanning values 1024 , 2048 , 4098 , 8196 , 16392 , 32784 , 65568 , 131136 1024,2048,4098,8196,16392,32784,65568,131136 and vocabulary sizes 10 , 20 , 30 , 40 10,20,30,40 . For ICL of functions, we vary instead the dimension n o n_{o} .

Note that for associative recall on longer sequences, multiple copies of key-value tuples appear in the prompt. To see this, consider how likely it is to sample multiple copies of a particular key-value pair with a vocabulary size of 40 40 , in order to form a sequence of 100 100 k characters. Models capable of looking further back in the sequence effectively see more data, and can solve challenging versions of the in-context learning task. Increasing the vocabulary size has the increasing the average distance between instances of the same key-value pair in each prompt, highlighting performance gaps between different approaches.

#### Long convolution comparisons:

We compare different convolution parametrizations, embedding them in an order 2 2 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} operator. All convolutions are applied separately to input channels (referred to as single-input single-output (SISO) in signal processing, or depthwise in other machine learning contexts).

β€’ Conv1d: Explicit convolutions (regular convolution layers with fixed filter size). We use a fixed filter size of 64 64 , to match parameters of the other approaches.

β€’ FNO: Filters parametrized explicitly in the frequency-domain ( Li et al., 2020 ) . We set the number of modes to 64 64 .

β€’ H3: Implicit parametrization using state-space models (SSMs), and in particular the standard S4 ( Gu et al., 2021 ) . We set the state dimension to 64 64 .

β€’ TransferFunc: Implicit parametrization via transfer functions, a classical system-theoretic generalization of SSMs. Transfer functions are defined by a ratio of polynomials (we parametrize the coefficients, and evaluate the polynomials efficiently via FFTs). We set the order to 64 64 .

β€’ CKConv: Implicit parametrization using π–₯π–₯𝖭 \sf FFN s ( Romero et al., 2021b ) .

β€’ item 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} : Combination of implicit parametrizations via π–₯π–₯𝖭 \sf FFN s (with exponential decay modulation as shown in Figure 3.1 ), and short explicit filters.

CKConv and 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} use the same size of π–₯π–₯π–­π—Œ {\sf FFNs} (width 32 32 to match in parameters).

In Table A.2 , we report additional results on the challenging setting of sequence length 131072 131072 and vocabulary size 30 30 . Implicit parametrizations of convolutions outperform explicit parametrizations on associative recall, with CKConv and 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} greatly improving on the ability to extract the right key, value relations from different inputs. In Appendix C , we discuss how results on our synthetic tasks can be indicative of performance at a larger scale.

#### Operator comparisons:

We compare different models on the same associative recall task, using hyperparameters in Table A.1 . 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} uses our filter parametrization with decay windowing for long convolutions, and short explicit convolutions of size 3 3 after the dense input projections. All other models use defaults from their largest scale experiment, while keeping the size to 2 2 layers and width 64 64 .

#### A note on Transformer performance

Transformers can solve associative recall tasks with longer sequences, provided the length does not prevent them from fitting in memory, and enough examples are present in the training data. In all our experiments, we keep the number of samples fixed ( 2000 2000 ), a regime where Transformers struggle to find the generalizing solution (see Table A.2 ).

For shorter sequences (see Appendix C ), Transformers solve the task easily even with limited data, comparably to Hyena .

More broadly, these different properties of attention and attention-free token-mixing layers may explain improved performance when they are combined in hybrid architectures ( Dao et al., 2022c ) . The focus on this work has been identifying an architecture capable of performing without attention, which is necessary to tackle domains where long sequences are common. However, when training with shorter sequences (up to 8 8 k), if final downstream performance is the only metric of interest, improved results can be obtained by hybridizing our models similarly to H3 ( Dao et al., 2022c ) .

### A.2 Language Modeling

#### WikiText103:

We train 125 125 M parameter models on WikiText103 and compare perplexity to Transformers, hybrid models such as H3 ( Dao et al., 2022c ) , and other variants of subquadratic attention. All models use the same GPT2 tokenizer with vocabulary size 50257 50257 . We test order 3 3 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} with our proposed filter parametrization for two long convolutions, and a shorter explicit convolution on the third. We also consider 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} (slim) that are 1.5 1.5 x deeper than Transformers ( 12 12 versus 18 18 layers), with width multiplier of the FFNs set to 2 2 . We find trading-off width for depth to be generally favourable. These modifications are made possible by the reduction in overall FLOPs of Hyena operators compared to self-attention, in particular non-parametric FLOPs which include materialization of the attention matrix, application of softmax, and matrix-value reduction.

#### The Pile:

We follow a same procedure and train 125 125 M and 355 355 M-sized models on The Pile ( Gao et al., 2020 ) . Hyperparameters are reported in Table A.3 . Hyperparameters for 355 355 M are the same beyond a reduction in peak learning rate to 4 β‹… 10 βˆ’ 4 4\cdot 10^{-4} . For larger models ( 1.3 1.3 B), we set a learning rate of 2.2 β‹… 10 βˆ’ 4 2.2\cdot 10^{-4} .

We perform three experiments for each model type and size, and train for 5 , 10 , 15 5,10,15 billion tokens at a sequence length 2024 2024 and global batch size 256 256 . All models are trained on a single node of 8 8 A ​ 100 A100 80 80 GB GPUs. We use order 2 2 𝖧𝗒𝖾𝗇𝖺 {\sf Hyena} s, with the same architectural considerations described above for WikiText103 . In addition to our data scaling experiments at 5 5 , 10 10 and 15 15 billion tokens, we provide preliminary results for models at the 1.3 1.3 B parameter scale ( 10.8 10.8 perplexity after 5 5 billion tokens), and train a 153 153 M model ( 130 130 billion tokens), reaching a perplexity of 9.8 9.8 . The 153 153 M is the same used in our downstream evaluation on SuperGLUE.