Nested Learning: The Illusion of Deep Learning Architectures β part 10/14
Cleaned-text code points [139098, 152346). Bibliography omitted here; source.md includes it.
Independent (Head-wise) Continuum Memory Systems. In this variant, we keep the knowledge transfer process in Equation 73 , but change the output computation in Equation 70 . While the previous formulation designs the memory system as a sequence of blocks, and so making their input/out dependent to each other, this variant uses independent blocks with different context length and then combine them using an aggregation process: π t = Agg β ( MLP ( f k ) β ( π t ) , MLP ( f k β 1 ) β ( π t ) , β― , MLP ( f 1 ) β ( π t ) ) . \displaystyle\bm{y}_{t}=\texttt{Agg}\left(\texttt{MLP}^{(f_{k})}(\bm{x}_{t}),\texttt{MLP}^{(f_{k-1})}(\bm{x}_{t}),\cdots,\texttt{MLP}^{(f_{1})}(\bm{x}_{t})\right). (74) The above Agg β ( β
) \texttt{Agg}\left(\cdot\right) is an arbitrary function that aggregates all the inputs to compute the output. For example, one straightforward and simple design choice is to use a learnable weighted sum of the input.
CMS Design Helps with Continual Learning. Based on the design of CMS, a fair question is to ask: Why and how CMS can help with longer context length and generally continual learning. Here, we provide a simple answer to this question: Viewing MLP blocks in CMS as the storage of modelβs knowledge catastrophic forgetting can happen when we update a block and as its result, the old knowledge stored in its parameters are forgotten. In CMS design, however, when updating an arbitrary block of MLP ( f s ) β ( β
) \texttt{MLP}^{(f_{s})}(\cdot) for some 1 β€ s β€ k 1\leq s\leq k , the potentially forgotten knowledge from MLP ( f s ) β ( β
) \texttt{MLP}^{(f_{s})}(\cdot) is still stored in other components such as MLP ( f s β² ) β ( β
) \texttt{MLP}^{(f_{s^{\prime}})}(\cdot) , where s β² < s s^{\prime}<s . Also, in this case (i.e., the knowledge is already forgotten from MLP ( f s ) β ( β
) \texttt{MLP}^{(f_{s})}(\cdot) but it is still in MLP ( f s β² ) β ( β
) \texttt{MLP}^{(f_{s^{\prime}})}(\cdot) for s β² < s s^{\prime}<s ) the knowledge transfer through backpropagation (for their initial state) can circle back the knowledge to MLP ( f s ) β ( β
) \texttt{MLP}^{(f_{s})}(\cdot) , resulting in a loop through time dimension, and so hardly forgetting important knowledge.
Is CMS Efficient Enough? A common concern when updating the parameters of a model in a continual manner is its efficiency. Therefore, a fair question is to ask if CMS causes significant computational overhead for the model. To answer this question, let us recall from Section 5 that modern recurrent neural networks are also continually updating a subset of their parameters (i.e., their memory state). These parameter updates, however, take advantage of sequence parallelization as well as updating only a small number of parameters. To this end, for CMS, we highlight two points: β’ In the CMS design, at each time, updates are restricted to blocks approaching their scheduled update time (based on their frequency). As a simple example, consider a Transformers but with replacing its MLP blocks with CMS (later in Section 8 , refer to this variant as Hope -Attention). Let the model have L layer L_{\text{layer}} layers, 4 levels of MLP blocks in CMS with highest frequency of f ^ \hat{f} , and hidden dimension of d in d_{\text{in}} . On average, the update cost is for πͺ β‘ ( 1 f ^ Γ L layer 5 Γ d in 2 ) \mathcal{O}\left(\frac{1}{\hat{f}}\times\frac{L_{\text{layer}}}{5}\times d_{\text{in}}^{2}\right) of parameters, which consists of only a small number of parameters at each time.
β’ The update mechanism of Equation 71 , not only helps with the enhancing the persistent memory of the model, but it also unlocks the sequence parallelization for higher frequency levels. More specifically, for input π i \bm{x}_{i} when i β’ 0 β ( mod β C ( β ) ) i\nequiv 0\>\>(\texttt{mod}\>C^{(\ell)}) there is no sequential process inside the chunk and so all the computations for tokens correspond to different values of i β’ 0 β ( mod β C ( β ) ) i\nequiv 0\>\>(\texttt{mod}\>C^{(\ell)}) can be done in parallel. The details of such training algorithm is the same as the training procedure in Sun et al. (2024) ; Behrouz et al. (2025c) .
Therefore, in summary, CMS can be fast in practice, mainly due to the fact that it updates only small number of parameters at each time, and also its design unlocks sequence parallelization.
### 7.2 Continuum Memory System In Optimizers
As a proof of concept and to support the effectiveness of CMS in different context flows, in this section we present Multi-scale Momentum/Memory Muon (M3) optimizer. Particularly, we aim to use NLβs associative memory viewpoint to design an optimizer that not only compress the recent gradients effectively, but it also has a capability of incorporating the information about long past gradients. In Appendix B ( Equation 101 ) we discuss that how Adam optimizer is an instance of associative memory, in which the gradients are mapped to their variance until that point. Following the discussion about the need of long-context capability of optimizers in Section 4.3 , we first replace the simple associative memory formulation of H {H} term in Equation 102 with our CMS (independent variant, Equation 74 ) with a two-level memory system, which we refer to the memories as π΄ ( 1 ) \bm{M}^{(1)} and π΄ ( 2 ) \bm{M}^{(2)} : π΄ t ( 1 ) = π΄ t β 1 ( 1 ) + Ξ² 1 β π t , \displaystyle\bm{M}^{(1)}_{t}=\bm{M}^{(1)}_{t-1}+\beta_{1}\bm{g}_{t}, π΄ t ( 2 ) = π΄ t ( 2 ) β Ξ² 2 β { β i = t β C ^ t π i if t β‘ 0 β ( mod β C ^ ) 0 otherwise, \displaystyle\bm{M}^{(2)}_{t}=\bm{M}^{(2)}_{t}-\beta_{2}\begin{cases}&\sum_{i=t-\hat{C}}^{t}\bm{g}_{i}\quad\quad\text{if}\>\>\>\>\>t\equiv 0\>\>\>(\text{mod}\>\>\hat{C})\\ &0\qquad\quad\>\>\qquad\text{otherwise,}\end{cases} (75) where C ^ \hat{C} is the chunk size that we update the lower-frequency momentum term. Finally, to aggregate the momentum terms (the choice of Agg β ( β
) \texttt{Agg}(\cdot) in Equation 74 ), we use a simple weighted summation with the use of parameter Ξ± > 0 \alpha>0 as the coefficient of π΄ t ( 2 ) \bm{M}^{(2)}_{t} . Following our discussion on the importance of Newton-Schulz T β ( β
) \texttt{Newton-Schulz}_{T}(\cdot) to map the gradients to a proper metric space (see Section 4.2 and Equation 43 ), following Muon ( Jordan et al., 2024 ) , we use Newton-Schulz T β ( β
) \texttt{Newton-Schulz}_{T}(\cdot) on the output of the momentum terms, before aggregating them with weighted sum. This helps the associative memory to better manage its capacity by updating its parameters in a proper direction. The pseudocode for M3 is in 1 . In summary, one can say that M3 is the combination of Adam ( Kingma and Ba, 2014 ) , Muon ( Jordan et al., 2024 ) , and our Continuum Memory System.
Notably, this optimizer is designed as a proof-of-concept to support the design of CMS. The M3 optimizer per se, however, might suffer from computational overhead and so face challenges when scaling to larger networks (see Figure 9.6 ). Furthermore, it is notable that the main point of M3 design based on CMS is to delay the update of the memory to gain longer-context. Similar studies such as the work of Pagliardini et al. (2025) ; Devan Selvaraj (2025) , while using multiple momentum terms, approach this long-context momentum for optimizers using a controlled learning rate for the additional momentum terms.
### 7.3 Ad-hoc Level Stacking: Initializing CMS with Pre-Trained Models
In our discussion on Figure 3 , we observed that the initial state of the memory modules are optimized in lower-frequency levels and so one can interpret them as MLP blocks in the vanilla Transformer architectures ( Vaswani et al., 2017 ) . Therefore, a natural question is if we can leverage pre-trained models to initialize CMS blocks. One of the important advantages of NL is its flexibility to view and modify parameters in different levels. That is, since each level has its own context flow and optimization process, one can simply initialize the parameters in each level independently so that it helps the model to adapt faster to the levelsβ context flow. To this end, in this section, we suggest initializing the parameters in a level with a modelβs pre-trained weights. More formally, given a CMS with { MLP ( f i ) β ( β
) } i = 1 k \{\texttt{MLP}^{(f_{i})}(\cdot)\}_{i=1}^{k} , and a set of pre-trained MLP blocks { MLP pre-trained i β ( β
) } i = 1 k \{\texttt{MLP}_{\text{pre-trained}_{i}}(\cdot)\}_{i=1}^{k} we use Equation 71 to update { MLP ( f i ) β ( β
) } i = 1 k \{\texttt{MLP}^{(f_{i})}(\cdot)\}_{i=1}^{k} in different levels; we, however, use the trained parameters of { MLP pre-trained i β ( β
) } i = 1 k \{\texttt{MLP}_{\text{pre-trained}_{i}}(\cdot)\}_{i=1}^{k} as the initial state of CMS blocks: MLP 0 ( f i ) β ( β
) = MLP pre-trained i β ( β
) \texttt{MLP}^{(f_{i})}_{0}(\cdot)=\texttt{MLP}_{\text{pre-trained}_{i}}(\cdot) .
Why This Initialization Should Work? In NL, when there is a knowledge transfer process between two levels, the higher frequency level can take advantage of the knowledge stored in the lower frequency level and so adapt faster to its own context flow. The internal learning rate in the higher-frequency level, however, can control the capacity of the model for adaptability. That is, consider the above case, where all the blocks are initialized with pre-trained MLP blocks, setting Ξ· t ( β ) β 0 \eta_{t}^{(\ell)}\rightarrow 0 keeps the updated memory blocks close to their initial states, resulting in directly using of pre-trained blocks, without adaption. Later in Section 9 , we use this method to adapt pre-trained Transformer architectures to the Hope βs setup.
## 8 Hope : A Self-Referential Learning Module with Continuum Memory
As we discussed earlier in Section 5.1 , architectures in nested learning are uniform, i.e., a set of feedforward neural network blocks, each of which with its own context, update frequency, and internal objective. Sequence modelsβa common term to refer to blocks often with the highest update frequency that fuse information across tokens in the input sequenceβare critical components for memory management and in-context learning ability of models. Following our earlier discussion in Section 5 , modern sequence models can be seen as associative memories and so are nested optimization problems. From this perspective, global softmax attention or its more expressive higher-order variants are perfect memories (forcing to cache all past tokens) with frequency update of infinity as they are non-parametric solutions for optimizing (local) L 2 L_{2} -regression objective with Nadaraya-Watson estimators ( Fan, 2018 ; Zhang et al., 2022 ) (see Equation 62 ). Therefore, parametric solutions (e.g., modern RNNs) for the similar objectives and when the parameter search space are the same (i.e., matrix-valued memory) are not expected to outperform softmax attention when the model size and data scales. To this end, and to design powerful sequence models, we need to understand where Transformers are limited and how one can overcome such limitations.
From the nested learning perspective, Transformers are two-level components, where projections and MLP blocks are optimized in the first level and the second level is responsible for in-context learning with finding the non-parametric solution and so conditioning the output on the context. This design, however, has limited computational depth as also stated in recent studies on state-tracking and similar computational capabilities of models ( Merrill et al., 2024 ; Sanford et al., 2024 ; Grazzi et al., 2025 ) . Furthermore, Transformersβ parameters are static throughout their context, meaning that their found solution to map tokens in the context (since it is non-parametric solution) remains the same and so they lack the ability to modify themselves (at least in-context). More specifically, the initial linear blocks, W π , W π , W_{\bm{k}},W_{\bm{v}}, and W π W_{\bm{q}} , that projects input data to keys, values, and queries, are fixed after the pre-training stage (i.e., are in the first level) and so the Transformerβs ability to contextualize and map tokens is bounded by the knowledge stored in these blocks. For example, given a 1-layer Transformer, the projection of each token is a function of the token itself and its position; therefore, as an example, it can miss the diverse possible encodings of words whose meaning depend on the context, rather than the word itself. Although with increasing the depth of the model this issue might fade in later layers, we should not rely on the depth to compensate the models ability as it still is a bottleneck to unleash the capability of the model in earlier layers.
To overcome the above challenge, recently, the use of short convolutions and canon layers ( Allen-Zhu, 2025 ) have became a de facto component in modern models. Despite their success in mixing local tokens, still the models are fundamentally limited to adapt to the context and capture the global information beyond the local mixing. In the next part, we discuss a fundamental solution by presenting self-referential Titans that allows all the components to perform in-context learning, and adapt and modify themselves:
### 8.1 Deep Self-Referential Titans