Nested Learning: The Illusion of Deep Learning Architectures — part 11/14
Cleaned-text code points [152346, 167877). Bibliography omitted here; source.md includes it.
A general formulation for the associative memory-based blocks is to project the data into keys, values, and queries and learns how to map keys to values and how to retrieve from the mapping based on queries. More formally, for a parametric associative memory, let 𝒙 t ∈ ℝ d \bm{x}_{t}\in\mathbb{R}^{d} for t = 1 , … , L t=1,\dots,L be the input, we have: 𝒌 t = 𝒙 t W 𝒌 , 𝒗 t = 𝒙 t W 𝒗 , 𝒒 t = 𝒙 t W 𝒒 , η t = 𝒙 t W η , α t = 𝒙 t W α , \displaystyle\bm{k}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{k}}},\qquad\bm{v}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{v}}},\qquad\bm{q}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}},\qquad\eta_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\eta}},\qquad\alpha_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\alpha}}, (76) min ℳ ℒ ( ℳ , 𝒌 t , 𝒗 t ) , with an optimization algorithm \displaystyle\min_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}}}\>\>\mathcal{L}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}};\bm{k}_{t},\bm{v}_{t}\right),\qquad\qquad\>\>\>{\color[rgb]{0.1641,0.2891,0.4023}\text{with an optimization algorithm}} (77) 𝒚 t = ℳ t 𝒒 t . \displaystyle\bm{y}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{t}}\bm{q}_{t}\>. (78) For the sake of clarity, we use red (resp. blue ) to highlight computations/weight in the upper level (resp. lower level). Similar to example in Figure 3 , we can add a new level for each of W 𝒌 , W 𝒗 , W 𝒒 , W η , W_{\bm{k}},W_{\bm{v}},W_{\bm{q}},W_{\eta}, and W α W_{\alpha} and allow them to be updated in-context. For the sake of efficiency, a simple version is to share the values for all the components in the nested system of associative memories: 𝒌 t = ℳ 𝒌 , t − 1 ( 𝒙 t ) , 𝒗 t = ℳ 𝒗 , t − 1 ( 𝒙 t ) , 𝒒 t = ℳ 𝒒 , t − 1 ( 𝒙 t ) , η t = ℳ η , t − 1 ( 𝒙 t ) , α t = ℳ α , t − 1 ( 𝒙 t ) , \displaystyle\>\bm{k}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{k},t-1}}\left(\bm{x}_{t}\right),\qquad\bm{v}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{v},t-1}}\left(\bm{x}_{t}\right),\qquad\bm{q}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{q},t-1}}\left(\bm{x}_{t}\right),\qquad\eta_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\eta,t-1}}\left(\bm{x}_{t}\right),\qquad\alpha_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\alpha,t-1}}\left(\bm{x}_{t}\right), (79) min ℳ □ ℒ ( ℳ □ , □ t , 𝒗 t ) , with an optimization algorithm , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α } , \displaystyle\min_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square}}}\>\>\mathcal{L}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square}};\square_{t},\bm{v}_{t}\right),\qquad\quad{\color[rgb]{0.1641,0.2891,0.4023}\text{with an optimization algorithm}},\quad\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha\}, (80) min ℳ mem ℒ ( ℳ mem , 𝒌 t , 𝒗 t ) , with an optimization algorithm , \displaystyle\min_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{mem}}}}\>\>\mathcal{L}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{mem}}};\bm{k}_{t},\bm{v}_{t}\right),\qquad{\color[rgb]{0.1641,0.2891,0.4023}\text{with an optimization algorithm}}, (81) 𝒚 t = ℳ mem , t ( 𝒒 t ) , \displaystyle\>\bm{y}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{mem},t}}\left(\bm{q}_{t}\right)\>, (82) where the initial states of all memories, i.e., ℳ □ , 0 {\color[rgb]{0.6094,0.3242,0.3242}\mathcal{M}_{\square,0}} for any □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } \>\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\} are meta-learned across all sequences/contexts. As discussed earlier, the meta-learning of the initial states of memories is essential for both fast-adaption, training stability, robustness to noise in the data.
This design provides a fully adaptive memory, where all the components can adapt themselves in-context. It, however, (1) still lacks self-modification, where the model in response to new data changes its own parameters or learning process ( Schmidhuber, 2003 ) ; (2) has suboptimal design as it shares of keys and values for all the memories. In continual learning, where the model requires consistent weight/knowledge update in response to new data, it is critical for the model to not solely rely on data, and instead learns how to modify itself when it is needed. Motivated by the above points, and inspired by the self-modifying mechanisms that generate their own values based on the context ( Schmidhuber, 2003 ; Schmidhuber, 1993 ; Irie et al., 2022b ) , we present self-modifying deep associative memory where the models generate their own values: 𝒚 t = ℳ memory , t − 1 ( 𝒒 t ) , 𝒌 t = ℳ 𝒌 , t − 1 ( 𝒙 t ) , 𝒗 t = ℳ 𝒗 , t − 1 ( 𝒙 t ) , η t = ℳ η , t − 1 ( 𝒙 t ) , α t = ℳ α , t − 1 ( 𝒙 t ) , \displaystyle\bm{y}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{memory},t-1}}\left(\bm{q}_{t}\right),\qquad\bm{k}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{k},t-1}}\left(\bm{x}_{t}\right),\qquad\bm{v}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{v},t-1}}\left(\bm{x}_{t}\right),\qquad\eta_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\eta,t-1}}\left(\bm{x}_{t}\right),\qquad\alpha_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\alpha,t-1}}\left(\bm{x}_{t}\right), (83) 𝒗 ^ □ , t = ℳ □ , t − 1 ( 𝒗 t ) , (Generating its own values for each memory) \displaystyle\hat{\bm{v}}_{\square,t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\bm{v}_{t}\right),\qquad\>\>\quad\text{(Generating its own values for each memory)} (84) min ℳ □ ℒ ( ℳ □ , 𝒌 t , 𝒗 ^ □ , t ) , with an optimization algorithm , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } , \displaystyle\min_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square}}}\>\>\mathcal{L}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square}};\bm{k}_{t},\hat{\bm{v}}_{\square,t}\right),\qquad{\color[rgb]{0.1641,0.2891,0.4023}\text{with an optimization algorithm}},\quad\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\}, (85) where 𝒒 t = 𝒙 t W 𝒒 \bm{q}_{t}=\bm{x}_{t}{\color[rgb]{0.6094,0.3242,0.3242}W_{\bm{q}}} is the only non-adaptive projection, η t \eta_{t} is the learning rate in optimization process, and α t \alpha_{t} is the retention gate (forget gate or weight decay) in the optimization process. Note that, again, the initial states of all memories, i.e., ℳ □ , 0 {\color[rgb]{0.6094,0.3242,0.3242}\mathcal{M}_{\square,0}} for any □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } \>\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\} are meta-learned across all sequences/contexts, and so are optimized in the higher levels (or outer-loop).
Learning the mappings for associative memory modules (see Equation 85 ) requires a choice of optimization algorithm as well as an objective ℒ \mathcal{L} that measures the quality of mappings. A simple and common choice for objective and optimization process are L 2 L_{2} -regression loss, and gradient descent algorithm. As for the objective, we use L 2 L_{2} -regression loss, i.e., ℒ ( ℳ , 𝒌 , 𝒗 ) = ‖ ℳ ( 𝒌 ) − 𝒗 ‖ 2 2 \mathcal{L}(\mathcal{M};\bm{k},\bm{v})=\|\mathcal{M}(\bm{k})-\bm{v}\|_{2}^{2} . As discussed earlier (see Section 4.5 ), the choice of optimizer highly depends on the context of optimization. For example, gradient descent from associative memory perspective is based on dot-product similarity and so the update at each step, is solely based on the input and does not incorporate the previous data samples to the update. When performing optimization in the token space, however, we know tokens are highly correlated. Therefore, following our discussion in Section 4.5 , we use our DGD with weight decay, resulting in general update rule of: 𝒚 t = ℳ memory , t − 1 ( 𝒒 t ) , 𝒌 t = ℳ 𝒌 , t − 1 ( 𝒙 t ) , 𝒗 t = ℳ 𝒗 , t − 1 ( 𝒙 t ) , η t = ℳ η , t − 1 ( 𝒙 t ) , α t = ℳ α , t − 1 ( 𝒙 t ) , \displaystyle\bm{y}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{memory},t-1}}\left(\bm{q}_{t}\right),\qquad\bm{k}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{k},t-1}}\left(\bm{x}_{t}\right),\qquad\bm{v}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{v},t-1}}\left(\bm{x}_{t}\right),\qquad\eta_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\eta,t-1}}\left(\bm{x}_{t}\right),\qquad\alpha_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\alpha,t-1}}\left(\bm{x}_{t}\right), (86) 𝒗 ^ □ , t = ℳ □ , t − 1 ( 𝒗 t ) , (Generating its own values for each memory) \displaystyle\hat{\bm{v}}_{\square,t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\bm{v}_{t}\right),\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\>\>\>\quad\text{(Generating its own values for each memory)} (87) ℳ □ , t = ℳ □ , t − 1 ( α t 𝑰 − η t 𝒌 t 𝒌 t ⊤ ) − η t ∇ ℒ ℳ □ , t − 1 ( ℳ □ , t − 1 ; 𝒌 t , 𝒗 ^ □ , t ) , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } . \displaystyle{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t}}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\alpha_{t}\bm{I}\>-\>\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)-\eta_{t}\nabla\mathcal{L}_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}};\bm{k}_{t},\hat{\bm{v}}_{\square,t}\right),\quad\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\}. (88) Here, the architecture of the memories are arbitrary and even we are not forced to use the same architecture for all components. We use a 2-layer MLP block as the architecture of all the memories: ℳ □ ( ⋅ ) = ( ⋅ ) + W □ , 1 σ ( W □ , 2 ( ⋅ ) ) . \displaystyle\mathcal{M}_{\square}(\cdot)=(\cdot)+W_{\square,1}\sigma(W_{\square,2}(\cdot)). (89)
### 8.2 Fast and Parallelizable Training
In the above, we discussed how to design a model that can learn to generate its own latent values and so modify itself. The main challenge from the practical point of view is the efficiency of the method and if its training is parallelizable. We follow the chunk-wise training algorithm of non-linear update rules ( Sun et al., 2024 ; Behrouz et al., 2025c ) and use update frequency of f □ = L C □ f_{\square}=\frac{L}{C_{\square}} , where L L is the context length. While there is no limitation to use different chunk-sizes, in our experiments, we use two different value of chunk sizes, one for the update of ℳ memory ( ⋅ ) \mathcal{M}_{\text{memory}}(\cdot) and the other for all the other memories in the self-referential Titans.
In more details, given an input sequence { 𝒙 t } t = 1 L \{\bm{x}_{t}\}_{t=1}^{L} and chunk size 1 ≤ C ≤ L 1\leq C\leq L , we split the sequence into ⌈ L C ⌉ \lceil\frac{L}{C}\rceil chunks of { 𝒙 ( ( i − 1 ) C + t ) } t = 1 C \{\bm{x}_{((i-1)C+t)}\}_{t=1}^{C} for i = 1 , … , ⌈ L C ⌉ i=1,\dots,\lceil\frac{L}{C}\rceil , and then generate all elements in Equation 86 at the end of each chunk for the next chunk. This allows for generating all the elements for the entire chunk in parallel, before starting the computation for this chunk. Furthermore, to update the memory modules based on Equation 88 , we take the gradient with respect to the last state of the previous chunk. Again, this allows for computing all the gradients for the next chunk in parallel. In more details, given this chunk-wise updating procedure, the update rule for the self-referential Titans is computed as:
𝒚 t = ℳ memory , C × ⌈ t C ⌉ ( 𝒒 t ) , 𝒌 t = ℳ 𝒌 , C × ⌈ t C ⌉ ( 𝒙 t ) , 𝒗 t = ℳ 𝒗 , C × ⌈ t C ⌉ ( 𝒙 t ) , η t = ℳ η , C × ⌈ t C ⌉ ( 𝒙 t ) , α t = ℳ α , C × ⌈ t C ⌉ ( 𝒙 t ) , \displaystyle\bm{y}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\text{memory},C\times\lceil\frac{t}{C}\rceil}}\left(\bm{q}_{t}\right),\quad\bm{k}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{k},C\times\lceil\frac{t}{C}\rceil}}\left(\bm{x}_{t}\right),\quad\bm{v}_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\bm{v},C\times\lceil\frac{t}{C}\rceil}}\left(\bm{x}_{t}\right),\quad\eta_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\eta,C\times\lceil\frac{t}{C}\rceil}}\left(\bm{x}_{t}\right),\quad\alpha_{t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\alpha,C\times\lceil\frac{t}{C}\rceil}}\left(\bm{x}_{t}\right), 𝒗 ^ □ , t = ℳ □ , C × ⌈ t C ⌉ ( 𝒗 t ) , (Generating its own values for each memory) \displaystyle\hat{\bm{v}}_{\square,t}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,C\times\lceil\frac{t}{C}\rceil}}\left(\bm{v}_{t}\right),\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\>\qquad\text{(Generating its own values for each memory)} ℳ □ , t = ℳ □ , t − 1 ( α t 𝑰 − η t 𝒌 t 𝒌 t ⊤ ) − η t ∇ ℒ ℳ □ , C × ⌈ t C ⌉ ( ℳ □ , C × ⌈ t C ⌉ ; 𝒌 t , 𝒗 ^ □ , t ) , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } . \displaystyle{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t}}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\alpha_{t}\bm{I}\>-\>\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)-\eta_{t}\nabla\mathcal{L}_{{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,C\times\lceil\frac{t}{C}\rceil}}}\left({\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,C\times\lceil\frac{t}{C}\rceil}};\bm{k}_{t},\hat{\bm{v}}_{\square,t}\right),\qquad\>\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\}. (90) Here, the architecture of the memories are arbitrary and even we are not forced to use the same architecture for all components. We use a 2-layer MLP block as the architecture of all the memories: ℳ □ ( ⋅ ) = ( ⋅ ) + W □ , 1 σ ( W □ , 2 ( ⋅ ) ) . \displaystyle\mathcal{M}_{\square}(\cdot)=(\cdot)+W_{\square,1}\sigma(W_{\square,2}(\cdot)). (91) Since all the gradients as well as new keys, values, learning-rates, and weight decays can be computed in parallel before starting the processing of the current chunk, the above updates accepts the fast parallelizable dual form that is discussed by Sun et al. (2024) and Behrouz et al. (2025c) . To better illustrate the above update rule for self-referential Titans, let us derive the recurrent formula for the simplest case of matrix-valued memory. We derive the recurrent form for two different objectives: • Dot-product similarity ℒ ( ℳ , 𝒌 , 𝒗 ) = − ⟨ ℳ 𝒌 , 𝒗 ⟩ \mathcal{L}(\mathcal{M};\bm{k},\bm{v})=-\langle\mathcal{M}\bm{k},\bm{v}\rangle : Given this objective and linear memory, the gradient is calculated as 𝒗 𝒌 ⊤ \bm{v}\bm{k}^{\top} , which results in update rule of: ℳ □ , t = ℳ □ , t − 1 ( α t 𝑰 − η t 𝒌 t 𝒌 t ⊤ ) − η t 𝒗 ^ □ , t 𝒌 t ⊤ , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } \displaystyle{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t}}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\alpha_{t}\bm{I}\>-\>\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)-\eta_{t}\hat{\bm{v}}_{\square,t}\bm{k}_{t}^{\top},\qquad\>\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\} (92)
• L 2 L_{2} -regression loss: Given this objective and linear memory, the gradient is calculated as ( ℳ 𝒌 − 𝒗 ) 𝒌 ⊤ (\mathcal{M}\bm{k}-\bm{v})\bm{k}^{\top} , which results in update rule of: ℳ □ , t = ℳ □ , t − 1 ( α t 𝑰 − η t 𝒌 t 𝒌 t ⊤ ) − η t ( ℳ □ , C × ⌈ t C ⌉ 𝒌 t − 𝒗 ^ □ , t ) 𝒌 t ⊤ , □ ∈ { 𝒌 , 𝒗 , 𝒒 , η , α , memory } . \displaystyle{\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t}}={\color[rgb]{0.1641,0.2891,0.4023}\mathcal{M}_{\square,t-1}}\left(\alpha_{t}\bm{I}\>-\>\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)-\eta_{t}\left(\mathcal{M}_{\square,C\times\lceil\frac{t}{C}\rceil}\bm{k}_{t}-\hat{\bm{v}}_{\square,t}\right)\bm{k}_{t}^{\top},\qquad\>\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\}. (93)
### 8.3 Hope Neural Learning Module