xLSTM: Extended Long Short-Term Memory — part 12/30

Cleaned-text code points [169152, 177404). Bibliography omitted here; source.md includes it.
{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin=1} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{ {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-16.94081pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 -23.44 0)} \pgfsys@hbox{65}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope \hbox to0.0pt{}{{ {}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}-\mathchoice{\hbox to44.48pt{\vbox to5.81pt{\pgfpicture\makeatletter\hbox{\hskip 22.23741pt\lower-1.5pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin=1} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{ {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-22.23741pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 -30.77 0)} \pgfsys@hbox{65}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope \hbox to0.0pt{}{{ {}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{\hbox to44.48pt{\vbox to5.81pt{\pgfpicture\makeatletter\hbox{\hskip 22.23741pt\lower-1.5pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin=1} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{ {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-22.23741pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 -30.77 0)} \pgfsys@hbox{65}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope \hbox to0.0pt{}{{ {}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{\hbox to29.63pt{\vbox to4.01pt{\pgfpicture\makeatletter\hbox{\hskip 14.8171pt\lower-1.0pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin=1} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{ {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-14.8171pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 -20.5 0)} \pgfsys@hbox{65}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope \hbox to0.0pt{}{{ {}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}{\hbox to23.88pt{\vbox to2.9pt{\pgfpicture\makeatletter\hbox{\hskip 11.94083pt\lower-0.75pt\hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} \lxSVG@begingroup@{stroke=#000000} \lxSVG@begingroup@{fill=#000000} \lxSVG@setlinewidth{\the\pgflinewidth}\lxSVG@begingroup@{stroke-width=0.4pt} \lx@inpgf@ignorespaces\nullfont\lxSVG@begingroup@{_scopebegin=1} \lxSVG@closescope \hbox to0.0pt{\lxSVG@begingroup@{_scopebegin=1} {}{ {{}}\lx@inpgf@ignorespaces\hbox{\hbox{{\lxSVG@begingroup@{_scopebegin=1} {{}{}{{ {}{}}}{ {}{}} {{}{{\lx@inpgf@ignorespaces}}}{{}{\lx@inpgf@ignorespaces}}{}{{}{\lx@inpgf@ignorespaces}} {\lx@inpgf@ignorespaces }{{{{\lx@inpgf@ignorespaces}}\lxSVG@begingroup@{_scopebegin=1} \lxSVG@transformcm{1.0}{0.0}{0.0}{1.0}{-11.94083pt}{0.0pt}\lxSVG@begingroup@{transform=matrix(1.0 0.0 0.0 1.0 -16.52 0)} \pgfsys@hbox{65}\lxSVG@closescope }}} \lxSVG@closescope }}} } \lxSVG@closescope \hbox to0.0pt{}{{ {}{}{}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}}}{\lx@inpgf@ignorespaces}{\lx@inpgf@ignorespaces}\hss}\lxSVG@discardpath\lxSVG@closescope \hss}}\lxSVG@closescope\endpgfpicture}}}\right)\ \, stabil. forget gate (17)

We show in Appendix A.2 , that replacing f t \mathrm{f}_{t} by f t ′ \mathrm{f}^{\prime}_{t} and i t \mathrm{i}_{t} by i t ′ \mathrm{i}^{\prime}_{t} in the forward pass does neither change the output of the whole network nor the derivatives of the loss with respect to the parameters.

##### New Memory Mixing.

sLSTM can have multiple memory cells like the original LSTM (see Appendix A.2 ). Multiple memory cells enable memory mixing via recurrent connections 𝑹 𝒛 \bm{R}_{\bm{z}} , 𝑹 𝐢 \bm{R}_{\mathbf{i}} , 𝑹 𝐟 \bm{R}_{\mathbf{f}} , 𝑹 𝐨 \bm{R}_{\mathbf{o}} from hidden state vector 𝒉 \bm{h} to memory cell input 𝒛 \bm{z} and the gates 𝐢 \mathbf{i} , 𝐟 \mathbf{f} , 𝐨 \mathbf{o} , respectively. A new aspect in memory mixing is the effect of exponential gating. The new sLSTM can have multiple heads with memory mixing within each head but not across heads. The introduction of heads for sLSTM together with exponential gating establishes a new way of memory mixing.

### 2.3 mLSTM

To enhance storage capacities of LSTMs, we increase the LSTM memory cell from a scalar c ∈ ℝ c\in\mathbb{R} to a matrix 𝑪 ∈ ℝ d × d \bm{C}\in\mathbb{R}^{d\times d} . Hence, retrieval is performed via a matrix multiplication. At time t t , we want to store a pair of vectors, the key 𝒌 t ∈ ℝ d \bm{k}_{t}\in\mathbb{R}^{d} and the value 𝒗 t ∈ ℝ d \bm{v}_{t}\in\mathbb{R}^{d} (we use the Transformer terminology). Later at time t + τ t+\tau , the value 𝒗 t \bm{v}_{t} should be retrieved by a query vector 𝒒 t + τ ∈ ℝ d \bm{q}_{t+\tau}\in\mathbb{R}^{d} . This is the setting of Bidirectional Associative Memories (BAMs) ( Kohonen, 1972 ; Anderson, 1972 ; Nakano, 1972 ; Anderson et al., 1977 ) . The covariance update rule ( Sejnowski, 1977 ; Dayan & Willshaw, 1991 ) for storing a key-value pair is 𝑪 t \displaystyle\bm{C}_{t}\ = 𝑪 t − 1 + 𝒗 t  𝒌 t ⊤ . \displaystyle=\ \bm{C}_{t-1}\ +\ \bm{v}_{t}\ \bm{k}_{t}^{\top}\ . (18) We assume a layer-norm before projecting inputs to keys and values, therefore they have zero mean. The covariance update rule is optimal ( Dayan & Willshaw, 1991 ) for a maximal separability of retrieved binary vectors, which is equivalent to a maximal signal/noise ratio. Higher separability is possible when limiting retrieval to pairwise interactions and conceding quadratic complexity like attention ( Krotov & Hopfield, 2016 ; Krotov & Hopfield, 2017 ; Ramsauer et al., 2021 ) . The covariance update rule is equivalent to Fast Weight Programmers ( Schmidhuber, 1992 ; Schlag et al., 2021 ) , which have later been equipped with a constant decay rate multiplied to 𝑪 t − 1 \bm{C}_{t-1} and a constant learning rate multiplied to 𝒗 t  𝒌 t ⊤ \bm{v}_{t}\bm{k}_{t}^{\top} ( Ba et al., 2016a ) . In this spirit, we integrate the covariance update rule into the LSTM framework, where the forget gate corresponds to decay rate and the input gate to the learning rate, while the output gate scales the retrieved vector.