Nested Learning: The Illusion of Deep Learning Architectures β€” part 8/14

Cleaned-text code points [107836, 123336). Bibliography omitted here; source.md includes it.

From now on, we assume that keys { π’Œ i } i = 1 L \{\bm{k}_{i}\}_{i=1}^{L} , values { 𝒗 i } i = 1 L \{\bm{v}_{i}\}_{i=1}^{L} , and queries { 𝒒 i } i = 1 L \{\bm{q}_{i}\}_{i=1}^{L} are given: they often are defined as the projections of the input, i.e., π’Œ t = 𝒙 t ​ W π’Œ , 𝒗 t = 𝒙 t ​ W 𝒗 , 𝒒 t = 𝒙 t ​ W 𝒒 . \displaystyle\bm{k}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{k}}},\qquad\bm{v}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{v}}},\qquad\bm{q}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}}. (61) In this design, since projection parameters (i.e., W π’Œ {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{k}}} , W 𝒗 {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{v}}} , and W 𝒒 {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}} ) are optimized in a lower frequency level, the sequence model component (e.g., self-attention) has a higher frequency and so the learning process of the associative memory happens in a lower level. Accordingly, for the sake of clarity, we only discuss the higher frequency level (i.e., the internal learning process of the associative memory).

Softmax Attention. From the associative memory viewpoint: given keys { π’Œ i } i = 1 L \{\bm{k}_{i}\}_{i=1}^{L} , values { 𝒗 i } i = 1 L \{\bm{v}_{i}\}_{i=1}^{L} , and queries { 𝒒 i } i = 1 L \{\bm{q}_{i}\}_{i=1}^{L} , Softmax attention block ( Bahdanau, 2014 ; Vaswani et al., 2017 ) can be reformulated as a non-parametric solution to the β„“ 2 ​ ( β‹… ) \ell_{2}(\cdot) regression objective with Nadaraya-Watson estimators ( Zhang et al., 2022 ; Fan, 2018 ) : β„³ βˆ— = arg ⁑ min ⁑ βˆ‘ i = 1 L β„³ ⁑ 𝐬 ⁑ ( π’Œ i , 𝒒 ) ​ β€– 𝒗 i βˆ’ β„³ β€– 2 2 = βˆ‘ i = 1 L 𝐬 ⁑ ( π’Œ i , 𝒒 ) βˆ‘ j = 1 L 𝐬 ⁑ ( π’Œ j , 𝒒 ) ​ 𝒗 i , \displaystyle\mathcal{M}^{*}=\arg\min_{\mathcal{M}}\sum_{i=1}^{L}\mathbf{s}(\bm{k}_{i},\bm{q})\|\bm{v}_{i}-\mathcal{M}\|^{2}_{2}=\sum_{i=1}^{L}\frac{\mathbf{s}(\bm{k}_{i},\bm{q})}{\sum_{j=1}^{L}\mathbf{s}(\bm{k}_{j},\bm{q})}\bm{v}_{i}, (62) where L L is the sequence length ( Sun et al., 2024 ) . This formulation optimizes the memory β„³ ⁑ ( β‹… ) \mathcal{M}(\cdot) with respect to the entire context; however, one design choice can be to limit the optimization process to the past c c tokens, resulting in: β„³ βˆ— = arg ⁑ min ⁑ βˆ‘ i = t βˆ’ c + 1 t β„³ ⁑ 𝐬 ⁑ ( π’Œ i , 𝒒 i ) ​ β€– 𝒗 i βˆ’ β„³ β€– 2 2 = βˆ‘ i = t βˆ’ c + 1 t 𝐬 ⁑ ( π’Œ i , 𝒒 ) βˆ‘ j = t βˆ’ c + 1 t 𝐬 ⁑ ( π’Œ j , 𝒒 ) ​ 𝒗 i , \displaystyle\mathcal{M}^{*}=\arg\min_{\mathcal{M}}\sum_{i=t-c+1}^{t}\mathbf{s}(\bm{k}_{i},\bm{q}_{i})\|\bm{v}_{i}-\mathcal{M}\|^{2}_{2}=\sum_{i=t-c+1}^{t}\frac{\mathbf{s}(\bm{k}_{i},\bm{q})}{\sum_{j=t-c+1}^{t}\mathbf{s}(\bm{k}_{j},\bm{q})}\bm{v}_{i}, (63) which is equivalent to the sliding window attention (SWA). Therefore, attention and its more expressive variants ( Wang et al., 2025 ) also are instances of Definition 1 , when instead of gradient descent or other parametric methods, we find the optimal non-parametric solution to the mapping.

RNNs with Hebbian Rule. The first generation of modern recurrent architectures (e.g., Linear attention ( Katharopoulos et al., 2020 ) , RetNet ( Sun et al., 2023 ) , RWKV ( Peng et al., 2023 ) , lightening attention ( Li et al., 2025 ) ) are based on Hebbian-like learning rules ( Hebb, 2005 ) . For this class of models, the inner objective to measure the quality of mapping between keys and values is the dot-product similarity. That is, given a matrix-valued memory β„³ ∈ ℝ d Γ— n \mathcal{M}\in\mathbb{R}^{d\times n} , keys and values π’Œ , 𝒗 ∈ ℝ d \bm{k},\bm{v}\in\mathbb{R}^{d} , objective β„’ ~ ​ ( β„³ , π’Œ t , 𝒗 t ) := βˆ’ 2 ​ ⟨ β„³ ​ π’Œ t , 𝒗 t ⟩ \tilde{\mathcal{L}}(\mathcal{M};\bm{k}_{t},\bm{v}_{t}):=-2\langle\mathcal{M}\bm{k}_{t},\bm{v}_{t}\rangle , and a kernel Ο• ⁑ ( β‹… ) \phi(\cdot) , we optimize the equivalent associative memory optimization problem (see Definition 1 ) with gradient descent and weight decay, resulting in: β„³ t = Ξ± t ​ β„³ t βˆ’ 1 βˆ’ Ξ· ​ βˆ‡ β„³ t βˆ’ 1 β„’ ~ ​ ( β„³ t βˆ’ 1 , Ο• ⁑ ( π’Œ t ) , 𝒗 t ) ⏟ βˆ’ 𝒗 t ​ Ο• ​ ( π’Œ t ⊀ ) = Ξ± t ​ β„³ t βˆ’ 1 + Ξ· t ​ 𝒗 t ​ Ο• ​ ( π’Œ t ⊀ ) , \displaystyle\mathcal{M}_{t}=\alpha_{t}\mathcal{M}_{t-1}-\eta\>\underset{-\bm{v}_{t}\phi\left(\bm{k}_{t}^{\top}\right)}{\underbrace{\nabla_{\mathcal{M}_{t-1}}\tilde{\mathcal{L}}\left(\mathcal{M}_{t-1};\phi(\bm{k}_{t}),\bm{v}_{t}\right)}}\ \>=\alpha_{t}\mathcal{M}_{t-1}+\eta_{t}\>\bm{v}_{t}\phi\left(\bm{k}_{t}^{\top}\right), (64) which recovers the original linear attention recurrence ( Katharopoulos et al., 2020 ) . Given different settings for Ξ± t \alpha_{t} (i.e., either is 1 1 , learnable, channel-wise, and/or input-dependent) and also Ο• ⁑ ( β‹… ) \phi(\cdot) (i.e., identity, polynomial kernels, etc.), the above recurrence recovers different variants of linear attention with Hebbian rule ( Katharopoulos et al., 2020 ; Kacham et al., 2024 ; Arora et al., 2024 ; Sun et al., 2023 ; Peng et al., 2024 ; Beck et al., 2024 ) . Therefore, the variants of linear attention with Hebbian rule can be reformulated as the process of an optimization problem, in which the memory aims to learn the mapping between keys and values based on dot-product similarity objective, with gradient descent.

RNNs with Delta Rule. To improve the memory management and to enhance the memory capacity of the above group, several studies suggest replacing Hebbian rule with Delta rule as the learning algorithm in recurrent neural networks ( Schlag et al., 2021 ) , resulting in models such as DeltaNet ( Schlag et al., 2021 ) , Longhorn ( Liu et al., 2024b ) , and RWKV7 ( Peng et al., 2025a ) . When letting β„³ ∈ ℝ d Γ— n \mathcal{M}\in\mathbb{R}^{d\times n} , delta rule is equivalent to optimizing MSE objective β„’ ~ t = β€– β„³ t ​ π’Œ t βˆ’ 𝒗 t β€– 2 2 \tilde{\mathcal{L}}_{t}=\|\mathcal{M}_{t}\bm{k}_{t}-\bm{v}_{t}\|^{2}_{2} with Ret t ​ ( β„³ , β„³ t βˆ’ 1 ) = β€– β„³ t βˆ’ β„³ t βˆ’ 1 β€– F 2 \text{Ret}_{t}(\mathcal{M},\mathcal{M}_{t-1})=\|\mathcal{M}_{t}-\mathcal{M}_{t-1}\|_{F}^{2} as local retention, and stochastic gradient descent as the optimizer: β„³ t = β„³ t βˆ’ 1 βˆ’ Ξ· t ​ βˆ‡ β„³ t βˆ’ 1 β„’ ~ ​ ( β„³ t βˆ’ 1 , Ο• ⁑ ( π’Œ t ) , 𝒗 t ) ⏟ ( β„³ t βˆ’ 1 ​ π’Œ t βˆ’ 𝒗 t ) ​ π’Œ t ⊀ = ( 𝐈 βˆ’ Ξ· t ​ π’Œ t ​ π’Œ t ⊀ ) ​ β„³ t βˆ’ 1 + Ξ· t ​ 𝒗 t ​ π’Œ t ⊀ . \displaystyle\mathcal{M}_{t}=\mathcal{M}_{t-1}-\eta_{t}\>\underset{\left(\mathcal{M}_{t-1}\bm{k}_{t}-\bm{v}_{t}\right)\bm{k}_{t}^{\top}}{\underbrace{\nabla_{\mathcal{M}_{t-1}}\>\tilde{\mathcal{L}}\left(\mathcal{M}_{t-1};\phi\left(\bm{k}_{t}\right),\bm{v}_{t}\right)}}=\left(\mathbf{I}-\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)\mathcal{M}_{t-1}+\eta_{t}\>\bm{v}_{t}\bm{k}_{t}^{\top}. (65) Using other forms of retention gates (e.g., Ret t ​ ( β„³ , β„³ t βˆ’ 1 ) = β€– β„³ t βˆ’ Ξ± t ​ β„³ t βˆ’ 1 β€– F 2 \text{Ret}_{t}(\mathcal{M},\mathcal{M}_{t-1})=\|\mathcal{M}_{t}-\alpha_{t}\>\mathcal{M}_{t-1}\|_{F}^{2} ), optimization algorithms with weight decay (e.g., regularizing with β€– β„³ t β€– q q \|\mathcal{M}_{t}\|_{q}^{q} for a given q > 0 q>0 ), multiple steps of gradient descent, and/or different formulations of learnable parameters such as Ξ· t \eta_{t} and Ξ± t \alpha_{t} can result in diverse variants of delta rule ( Peng et al., 2025a ; Sun et al., 2024 ; Hu et al., 2025 ; Wang et al., 2025 ; Siems et al., 2025 ; Irie et al., 2021 ; Liu et al., 2024b ; Behrouz et al., 2025b ) . Therefore, Delta rule and its variants are all instances of an optimization problem, in which the model aims to learn a mapping between keys and values based on the L 2 L_{2} -regression objective.

Beyond Conventional Learning Rules: Omega, Oja’s, and Non-Euclidean Learning Rules. More recently, there have been growing interests in designing architectures from the associative memory perspective (see Definition 1 ) and use more complex internal objectives, and/or optimization algorithms, resulting in learning algorithms beyond Delta and Hebbian rules ( Behrouz et al., 2025b ; Irie et al., 2022a ; Behrouz et al., 2025a ; Von Oswald et al., 2023 ; Zhang et al., 2025 ) . More specifically, to enhance the stability of Hebbian rule (discussed in Equation 64 ), Irie et al. (2022a) introduced OjaNet based on Oja’s rule ( Oja, 1982 ) with the following recurrence: β„³ t = Ξ± t ​ β„³ t βˆ’ 1 + Ξ· t ​ 𝒗 t ​ ( Ο• ​ ( π’Œ t ) ⊀ βˆ’ β„³ t βˆ’ 1 ⊀ ​ 𝒗 t ) . \displaystyle\mathcal{M}_{t}=\alpha_{t}\mathcal{M}_{t-1}+\eta_{t}\>\bm{v}_{t}\left(\phi(\bm{k}_{t})^{\top}-\mathcal{M}_{t-1}^{\top}\bm{v}_{t}\right). (66) In the associative memory formulation (as in Definition 1 ), this recurrence can simply be reformulated as one step of gradient descent as: β„³ t = β„³ t βˆ’ 1 βˆ’ Ξ· t ​ βˆ‡ β„³ t βˆ’ 1 β„’ ~ ​ ( β„³ t βˆ’ 1 , Ο• ⁑ ( π’Œ t ) , 𝒗 t ) ⏟ β„³ t βˆ’ 1 ⊀ ​ 𝒗 t βˆ’ 𝒗 t ​ Ο• ​ ( π’Œ t ) ⊀ , \displaystyle\mathcal{M}_{t}=\mathcal{M}_{t-1}-\eta_{t}\>\underset{\mathcal{M}_{t-1}^{\top}\bm{v}_{t}-\bm{v}_{t}\phi(\bm{k}_{t})^{\top}}{\underbrace{\nabla_{\mathcal{M}_{t-1}}\>\tilde{\mathcal{L}}\left(\mathcal{M}_{t-1};\phi\left(\bm{k}_{t}\right),\bm{v}_{t}\right)}}, (67) where β„’ ~ ​ ( β„³ , π’Œ t , 𝒗 t ) = βˆ’ 2 ​ ⟨ β„³ ​ π’Œ t , 𝒗 t ⟩ + β€– β„³ ⊀ ​ 𝒗 t β€– 2 2 \tilde{\mathcal{L}}(\mathcal{M};\bm{k}_{t},\bm{v}_{t})=-2\langle\mathcal{M}\bm{k}_{t},\bm{v}_{t}\rangle+\|\mathcal{M}^{\top}\>\bm{v}_{t}\|^{2}_{2} and Ο• ⁑ ( β‹… ) \phi(\cdot) is a kernel ( Irie et al., 2022a ; Irie and Gershman, 2025 ) . Although this design enhances the Hebbian learning rule by enforcing a unit-norm constraint for the single-neuron, it has been reported to empirically underperform models based on Delta learning rule ( Irie et al., 2022a ) . To further enhance the Delta rule through the design of more expressive objectives, recently, Behrouz et al. (2025b) suggested going beyond Euclidean spaces and use L p = βˆ₯ β‹… βˆ₯ p p L_{p}=\|\cdot\|_{p}^{p} norm for the internal regression objective, showing better empirical performance and robustness in long context tasks compared to Delta rule and its variants.

While the majority of learning rules are online update mechanisms–meaning that at each state, the models only need to keep the memory and the current (batch of) input–Omega rule ( Behrouz et al., 2025a ) suggest an update rule based on a set of past (batches of) inputs (or all inputs). More specifically, given a memory β„³ \mathcal{M} with an arbitrary structure, keys and values π’Œ , 𝒗 ∈ ℝ d \bm{k},\bm{v}\in\mathbb{R}^{d} , an arbitrary objective β„’ ~ ​ ( β„³ , π’Œ t , 𝒗 t ) \tilde{\mathcal{L}}(\mathcal{M};\bm{k}_{t},\bm{v}_{t}) , and a kernel Ο• ⁑ ( β‹… ) \phi(\cdot) , Omega rule is defined as: β„³ t = Ξ± t ​ β„³ t βˆ’ 1 βˆ’ βˆ‘ i = t βˆ’ c + 1 t Ξ³ t , i ​ β„’ ~ ​ ( β„³ t , Ο• ⁑ ( π’Œ i ) , 𝒗 i ) , \displaystyle\mathcal{M}_{t}=\alpha_{t}\>\mathcal{M}_{t-1}-\sum_{i=t-c+1}^{t}\gamma_{t,i}\>\>\tilde{\mathcal{L}}(\mathcal{M}_{t};\phi(\bm{k}_{i}),\bm{v}_{i}), (68) where c β‰₯ 1 c\geq 1 is the local window of cached inputs. Note that in the special case of Ξ³ t , i = 1 \gamma_{t,i}=1 and c c equal to the entire context length, the optimal solution of the above design collapses into an online case, where the update rule only depends on the current state and the current input ( Von Oswald et al., 2023 ) . For further discussion with more details about representing architectures as associative memories and so an optimization problem, we refer the reader to Behrouz et al. (2025b) .

A Note on Gating in Modern Sequence Models. One of the recent architectural changes in modern language models is the gating of a linear layer’s output with the output of the sequence model. Despite significant improvement resulted by this method, it is still unclear that how it enhances the performance. As we discussed in Figure 3 and its corresponding example, the main difference between feedforward network and modern recurrent memory modules (e.g., linear attention ( Katharopoulos et al., 2020 ) or deep memory modules ( Behrouz et al., 2025c ) ) when their initial state of the memory is meta-learned, is the second level in memory modules that perform in-context learning and adapt its state with the context. From this viewpoint, when the initial value of the memory is not meta-learned, it only relies on the in-context adaption of the memory and so there is no persistent memory system that stores the knowledge of pre-training in this block. Therefore, when the initial value of memory is not meta-learned, which is common in earlier variants of linear transformers, the gating of linear attention acts as a persistent memory and the initialization of the memory module.

### 5.1 Revisiting the Human Brain Perspective of Nested Learning

In Section 1.1 , we discussed how structure in human brain is uniform and reusable, and if we need a new architecture in deep learning, or if our beliefs about the heterogeneity of current models need to be revisited. In the previous sections, we observed that both optimization process of neural networks as well as neural architectures can be formulated as a set of nested and/or parallel optimization problems, in which the memory structure is a feedforward layer (e.g., either Deep MLPs, linear layers, etc.) and the objective is optimized with gradient descent or Newton’s methods.

From this perspective, modern architectures, are a set of artificial neurons (i.e., linear or deep feedforward networks), and each group of neurons has their own internal objective and so update mechanism. To this end, as a simple example, let us recall the AdaTransformer in Figure 3 : Given X = { 𝒙 i } i = 1 T X=\{\bm{x}_{i}\}_{i=1}^{T} as the input sequence, the output of the block is computed as (For the sake of simplicity, we assume MLP ( β‹… ) = β‹… W MLP \texttt{MLP}(\cdot)=\cdot\>W_{\texttt{MLP}}\> , and remove normalizations): π’Œ t = 𝒙 t ​ W π’Œ , 𝒗 t = 𝒙 t ​ W 𝒗 , 𝒒 t = 𝒙 t ​ W 𝒒 , \displaystyle\bm{k}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{k}}},\qquad\bm{v}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{v}}},\qquad\bm{q}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}}, π’š attn , t = Attn ( π’Œ t , 𝒗 t , 𝒒 t ) , \displaystyle\bm{y}_{\texttt{attn}_{,t}}={\color[rgb]{0.1641,0.2891,0.4023}\texttt{Attn}}\left(\bm{k}_{t},\bm{v}_{t},\bm{q}_{t}\right), π’š block , t = π’š attn , t W LinAttn , t , where \displaystyle\bm{y}_{\texttt{block}_{,t}}=\bm{y}_{\texttt{attn}_{,t}}\>{\color[rgb]{0.1641,0.2891,0.4023}W_{\texttt{LinAttn}_{,t}}},\>\>\>\quad\qquad\text{where} W LinAttn , t = W LinAttn , t βˆ’ 1 + 𝒗 t π’Œ t ⊀ , \displaystyle{\color[rgb]{0.1641,0.2891,0.4023}W_{\texttt{LinAttn}_{,t}}}={\color[rgb]{0.1641,0.2891,0.4023}W_{\texttt{LinAttn}_{,t-1}}}+{\color[rgb]{0.1641,0.2891,0.4023}\bm{v}_{t}\bm{k}_{t}^{\top}}, (69) The lowest frequency level is responsible for the optimization of W π’Œ {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{k}}} , W 𝒗 {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{v}}} , and W 𝒒 {\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}} , all of which are feedforward networks and so are uniform. The attention itself is also the non-parametric matrix-valued solution of a regression objective, again verifying that the structure is a matrix of artificial neurons (i.e., parameters). Finally, the Linear Attention++ component is equivalent to the optimizing the dot-product similarity of the mappings over the linear class of functions. Therefore, all the parameters are matrix-valued or deep feedforward layers, meaning that the only difference in the components of an architecture is their level, objective, and/or learning update rule.

## 6 Takeaways and Revisiting Common Terms

In the previous sections, we discussed the concept of nested learning and how existing well-known components of neural networks such as popular optimizers and architectures fall under the NL paradigm. In this section, we discuss the takeaways, connection of different concepts, and the implications of NL perspective on common terms.