Hyena Hierarchy: Towards Larger Convolutional Language Models β part 5/5
Cleaned-text code points [62896, 66901). Bibliography omitted here; source.md includes it.
The positional encoding chosen for the π§ππΎππΊ \sf Hyena filters is a truncated complex exponential basis. Specifically, with Ο k β ( t ) = e i β 2 β Ο β k β t / L \rho_{k}(t)=e^{i2\pi kt/L} for k = 0 , β¦ β K β 1 k=0,\dots K-1 , the positional encoding is defined as a map from β \mathbb{R} to β 2 β K + 1 \mathbb{R}^{2K+1} such that π―ππππππππΊπ
π€ππΌππ½πππ β‘ ( t ) = [ t β β [ Ο 0 ] β ( t ) β― β β [ Ο K β 1 ] β ( t ) β β [ Ο 0 ] β ( t ) β― β β [ Ο K β 1 ] β ( t ) ] {\sf PositionalEncoding}(t)=\begin{bmatrix}t&\mathfrak{R}[\rho_{0}](t)&\cdots&\mathfrak{R}[\rho_{K-1}](t)&\mathfrak{I}[\rho_{0}](t)&\cdots&\mathfrak{I}[\rho_{K-1}](t)\end{bmatrix} where β β‘ [ β
] \mathfrak{R}[\cdot] , β β‘ [ β
] \mathfrak{I}[\cdot] denote the real and imaginary part of their argument, respectively. In the main text, we use D e = 2 β K + 1 D_{e}=2K+1 to denote the size of a positional encoding with K K features. The number of features of the positional encoding has an impact on the filter initialization and training performances. In particular, we show how K K leads to a preconditioning of the spectrum of the filter at initialization. Figures D.6 , D.7 , D.8 display the initialized filters (with no πΆπππ½ππ \sf Window function) for different values of K K ( { 8 , 32 , 64 } \{8,32,64\} ) for L = 128 L=128 and frequency Ο a \omega_{a} of sinusoidal activation Ο β‘ ( β
) = sin ( Ο a β
) \sigma(\cdot)=\sin(\omega_a \cdot) set to 1. We notice how the choice of K K induces a bias in the modeled frequencies at initialization. Specifically the filters resemble low-pass filters with a cut-off frequency of approximatively 2 β K + 1 2K+1 .
This cut-off frequency is strongly related to the smoothness of the filter; as previously mentioned, we empirically observe better training dynamics of filters initialized to be non-smooth, i.e. with a rich high-frequency content. While we can achieve good initializations by increasing K K , this results in larger π₯π₯π \sf FFN s (its input dimension is 2 β K + 1 2K+1 , i.e. the number of positional encoding features) which come with a higher parameter count. A more efficient solution is to increase the frequency Ο a \omega_{a} of the sinusoidal activation. Figure D.9 show how with K = 8 K=8 we can cover the full spectrum simply by setting Ο a = 10 \omega_{a}=10 .
### D.4 Downstream Examples
#### MultiRC
We report examples of downstream evaluation of small models on the MultiRC question-answering task. We report answers of small Hyena (153M, trained for 130 130 B tokens on The Pile ) and the public checkpoint RWKV-v4 ( Peng, 2021 ) ( 169 169 M, trained for 332 332 B tokens on The Pile ). We select randomized examples with indices being powers of 2 2 . Alignment of Hyena βs responses to the task format is greatly improved by providing few-shot examples in the prompt, which may be a promising sign for larger models based on the proposed attention-free architecture. Each example shows responses for specific examples in the validation set (example indices are listed at the beginning of each example).
## Instructions for reporting errors
We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:
Click the "Report Issue" ( ) button, located in the page header.
Tip: You can select the relevant text first, to include it in your report.
Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.
Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .