Nested Learning: The Illusion of Deep Learning Architectures — part 9/14
Cleaned-text code points [123336, 139098). Bibliography omitted here; source.md includes it.
Memory and Learning. For a long period of time, in machine learning models, memory have been treated as a separate block with a clear distinction between its parameters and other components. Such designs often assume a short and/or long-term memory blocks, where short-term memory is responsible for the local context, while long-term memory is the storage for the persistent knowledge in models. In human brain, however, memory is considered as a distributed interconnected system without a clear known components that are independently responsible for short or long-term memory. In NL, we build upon a common terminology for memory and learning in neuropsychology literature, indicating that: Memory is a neural update caused by an input and so learning is the process of acquiring useful memory ( Okano et al., 2000 ) . From this viewpoint, any update by gradient descent (or any other optimization algorithms) in any levels of neural learning module is considered as a form of memory. Interestingly, our findings in Section 4.1 on gradient descent being (self-referential) associative memory is aligned with this terminology. Furthermore, based on this terminology, in continuum memory system, the neural updates are applied at different frequencies and so memories are stored with different time scales, resulting in more robust memory management with respect to catastrophic forgetting.
A General Note on Parameters of a Model. The parameters of a model are one of its critical components that shape the units for knowledge storage, internal computations, and adaptability. Over the past decades, only (a subset of) the parameters in the architecture of machine learning models have been referred to as the learnable entities, mainly due to the fact that they are the only components that we have been directly and intentionally optimizing over the training data. From the NL viewpoint, such parameters are placed in the lowest frequency level (the highest level) and are updated for every (batch of) samples. They are, however, are not the only parameters that contributes to the model internal computation, knowledge storage, and adaptability. As discussed earlier in Section 4.2 , momentum is an example of such cases, where its parameters are updated over time (by gradient descent) and stores the knowledge about the loss landscape of the model so far. Such information are critical when the model is continually learn, mainly due to the fact that to find an effective solution, the optimizer needs to have more information about the global properties of the loss landscape. Another example of such cases is the memory (or hidden state) of recurrent neural networks. Although those parameters are not directly optimized in the lowest frequency level, they store important knowledge about the current context. With the change of the context, the compressed knowledge in these parameters are removed due to the lack of knowledge transfer between these levels.
More Computations per Neuron. One common misinterpretation about the concept of NL is to restrict the model design and stacking multiple levels to CMS case. In general, stacking levels can help the model to enhance the depth of computations and also perform more internal computations per each parameter in the lowest frequency level. One example of such designs is Muon optimizer and NewtonSchulz k ( ⋅ ) \texttt{NewtonSchulz}_{k}(\cdot) operation (see Equation 42 - 44 ), which we showed that is equivalent to an internal optimization process. In this design, per each step of momentum update, we need k k -steps of internal process to learn how to map gradients to an orthogonal space.
In-Context Learning. Throughout this paper, we use the most general definition of “in-context learning” and refer to it as the ability of a model to adapt to and learn from a given context. Following the definition of NSAM , each block or level has its own context flow and so any neural update or adaptation to that context is considered as a form of in-context learning. Due to the popularity of Transformers as well as being the first model that in-context learning is studied for, the concept of in-context learning sometimes is referred to as conditioning the output on the entire context. Considering the general term of in-context learning, this formulation is only one of the instances of in-context learning, which we referred to as non-parametric in-context learning . In general, however, the memory in recurrent models is performing in-context learning, in which the output is conditioned on the compressed context . Therefore, from NL perspective, all the levels are performing in-context learning but on their own context flow with their own learning update and optimization process.
Building on this definition, in-context learning is a model’s capability that is transparent from its NL representation, and per se it is not an emergent characteristic but a direct consequence of having multiple levels in the NL representation of the neural learning module. Although this might seem contradictory with previous claims about ICL being an emergent characteristic ( Brown et al., 2020 ; Singh et al., 2023 ) , it is notable that the good performance of the model in ICL tasks also requires a powerful low-frequency level, enabling the high frequency level to adapt fast. When the model is not well-trained, the higher-frequency level is on its own to learn from the context. This setup might result in a poor performance, mainly due to the fact that there might be not enough data in context to allow the high-frequency parameters to converge.
(Test-Time) Learning/Memorization. Recently, the concept of test time training ( Sun et al., 2024 ; Wang et al., 2025 ) or test time memorization ( Behrouz et al., 2025b ) has gained popularity as a backbone framework to design powerful sequence models. In these frameworks, given the context, a new component/block aims to compress the context into its parameters using a learning rule and objective function. In this formulation when the context is removed the acquired in-context knowledge diminishes along with it. As also discussed in the previous part, this update mechanism and learning process is indeed an instance of “parametric in-context learning” :
Notably, when moving toward continual learning setup, there is no test-time or training-time and so it can be misleading to refer to (parametric) in-context learning as test-time learning/memorization.
Pre-training and Test Time. From the nested learning perspective, the lowest frequency level (i.e., the highest level) corresponds to a learning phase that is often referred to as pre-training. Accordingly, pre-training is one of the levels and so has its own context flow (i.e., pre-training dataset), objective (e.g., next token prediction), and optimization process (e.g., AdamW). Accordingly, one can interpret the pre-training as one of the possible instances of in-context learning, where the context is the entire pre-training data.
This formulation and viewpoint is specifically important when we shift from pre-training paradigm to models that are capable of continually interact and learn from data/world (e.g., Sutton (2025) ):
Continual Learning. From NL’s perspective each phase of training for a model is defined as one of the low-frequency levels, which by design, we might want to stop the data processing in one level (e.g., “End of Pre-training”), or continue it without any knowledge transfer to other levels (e.g., conventional formulation of in-context learning in Transformers). Accordingly, any machine learning model, no matter if it is during its pre-training or at test time, is performing continual learning as given a data sample, it requires performing internal computations to provide the output. However, the knowledge from that learning might not last or transfer to more persistent levels, mainly due to the lack of knowledge transfer between levels.
We also revisit some of the common terms in the architectural design in the following:
Existing Architectural Backbones and Hybrid Models. As discussed earlier in Section 5.1 , from the NL’s perspective all modern architectures are uniform and in fact are feedforward layers (linear or non-linear MLP blocks) that are trained based on their own context flow and optimization problem. When viewing models from deep learning perspective, we see the final solution of such optimization problem (i.e., we see attention rather than a non-parametric solution to a regression loss), which results in the illusion of having distinct and non-uniform architectures.
Another important aspect of understanding the existing architectures during their so-called “pre-training” phase is to understand how they transfer their knowledge from one level to another.
Neural Learning Module as an Inter-Connected System. One of the critical messages in NL is the fact that neural learning modules are inter-connected systems, meaning that the design of each component can significantly affect the design of other parts. This fact motivates follow up and future studies to better understand how one can properly design a neural learning module with all components work together in a harmony. As an example of such, the context of optimizers (i.e., gradients) is generated by the architecture component. Therefore, different architectures might show different characteristics in the generated gradient patterns and so one optimizer might not be the best option for all the architectures ( Zhang et al., 2024b ) .
Optimizers vs. Learned Optimizers. Finally, we want to emphasize that our formulation of momentum, gradient descent, and/or other gradient-based optimizers show that they are associative memory modules aiming to compress the data and gradients into their parameters. Such update and compression process is based on gradient descent and so has a very similar nature to the learning process of learned optimizers. From the NL’s viewpoint, both vanilla optimizers as well as learned optimizers are instances of the same concept but with different frequency and context flow: Although the parameters of learned optimizers are located in the lowest frequency level (i.e., to train and be optimized along with other parameters in the pre-training), the parameters of vanilla optimizers are located in their own level and so has their own gradient flow.
## 7 Continuum Multi-Timescale Memory System
Existing architectural backbones consist of (1) a working memory module (e.g., attention), which is responsible to actively fuse the information across sequence length, and (2) a feed-forward layer (e.g., MLP) that fuse information across features and acts as the persistent memory or knowledge storage of pre-training phase. From the NL perspective, pre-training is the phase that the most outer level of the learning module is updated over its limited context flow. Accordingly, in the continual setup, such pre-training phase is also rarely updated over time, and so its corresponding knowledge storage needs to rarely be updated over time. Given this intuition, we extend the traditional view-point of long-term/short-term memory system and suggest a knowledge storage feed-forward for each level (frequency domain).
### 7.1 Continuum Memory System (CMS)
Given the definition of frequency (Definition 2 ), Continuum Memory System (CMS) is formalized as a chain of MLP blocks MLP ( f 1 ) ( ⋅ ) , … , MLP ( f k ) ( ⋅ ) \texttt{MLP}^{(f_{1})}(\cdot),\dots,\texttt{MLP}^{(f_{k})}(\cdot) , each of which associated with a chunk size of C ( ℓ ) := max i C ( i ) f ℓ C^{(\ell)}:=\frac{\max_{i}C^{(i)}}{f_{\ell}} such that given input x = { 𝒙 1 , … , 𝒙 T } x=\{\bm{x}_{1},\dots,\bm{x}_{T}\} the output of the chain is calculated as (we disregard normalizations for the sake of clarity): 𝒚 t = MLP ( f k ) ( MLP ( f k − 1 ) ( ⋯ MLP ( f 1 ) ( 𝒙 t ) ) ) , \displaystyle\bm{y}_{t}=\texttt{MLP}^{(f_{k})}(\texttt{MLP}^{(f_{k-1})}(\cdots\texttt{MLP}^{(f_{1})}(\bm{x}_{t}))), (70) where the parameters of ℓ \ell -th MLP block, i.e., 𝜽 ( f ℓ ) \bm{\theta}^{(f_{\ell})} , are updated every C ( ℓ ) C^{(\ell)} steps: 𝜽 i + 1 ( f ℓ ) = 𝜽 i ( f ℓ ) − { ∑ t = i − C ( ℓ ) i η t ( ℓ ) f ( 𝜽 t ( f ℓ ) , 𝒙 t ) if i ≡ 0 ( mod C ( ℓ ) ) , 0 otherwise . \displaystyle\bm{\theta}^{(f_{\ell})}_{i+1}=\bm{\theta}^{(f_{\ell})}_{i}-\begin{cases}\sum_{t=i-C^{(\ell)}}^{i}\eta^{(\ell)}_{t}f(\bm{\theta}^{(f_{\ell})}_{t};\bm{x}_{t})&\text{if \>\>}i\equiv 0\>\>(\texttt{mod}\>C^{(\ell)}),\\ 0&\text{otherwise}.\end{cases} (71) Here η t ( ℓ ) \eta^{(\ell)}_{t} are learning rates corresponds to 𝜽 ( f ℓ ) \bm{\theta}^{(f_{\ell})} , and f ( ⋅ ) f(\cdot) is the error component of an arbitrary optimizer (e.g., ∇ ℒ ( 𝜽 t ( f ℓ ) , 𝒙 t ) \nabla\mathcal{L}(\bm{\theta}^{(f_{\ell})}_{t};\bm{x}_{t}) in gradient descent). The conventional Transformer block ( Vaswani et al., 2017 ) is a special instance of this formulation, where k = 1 k=1 and frequency of update is zero. Note that the objective of ℒ ( ⋅ ) \mathcal{L}(\cdot) is the objective of choice for the task at hand, e.g., for language modeling it is next token prediction objective. It is notable that Equation 71 provides an important interpretation: parameters 𝜽 t ( f ℓ ) \bm{\theta}^{(f_{\ell})}_{t} are responsible for compressing their own context into the their parameters and so they are a representative of abstract knowledge of their context.
As discussed in Section 3.3 , different levels might have different process of knowledge transfer. Accordingly, while the above formulation suggests a spectrum of memory systems in different levels and so with different frequencies, their connections can vary based on the design. In the following, we discuss some potential variants:
Nested Continuum Memory Systems. The first variant is a fully nested continuum memory system, in which the initial state of the MLP block in level s + 1 s+1 is meta-learned in level s s . This design allows for higher-order in-context learning ability, where each of the levels has its own context flow and re-initialized after the end of the context. More specifically, given an arbitrary 1 ≤ s ≤ k 1\leq s\leq k , 𝜽 0 ( f s + 1 ) = arg min Φ 𝔼 𝒯 ∼ 𝒞 ( s ) [ ℓ ( Θ , 𝒯 , Φ ) ] , \displaystyle\bm{\theta}^{(f_{s+1})}_{0}=\underset{\Phi}{\arg\min}\;\;\mathbb{E}_{\mathcal{T}\sim\mathcal{C}^{(s)}}\Biggl[\ell(\Theta,\mathcal{T};\Phi)\Biggr], (72) where 𝒞 ( s ) \mathcal{C}^{(s)} is the context length of the MLP block in s s -th level. Following this design, at the end of the optimization process of each block (i.e., after ⌈ C ( s ) / C ( s + 1 ) ⌉ \lceil C^{(s)}/C^{(s+1)}\rceil steps.) the value of the memory will be re-initialized to 𝜽 0 ( f s + 1 ) \bm{\theta}^{(f_{s+1})}_{0} . Note that the update mechanism of each block in its own level remain unchanged (i.e., Equation 71 ).
Sequential Continuum Memory Systems. In the second variant, the MLP blocks are located sequentially (i.e., the output of the MLP block in level s s is the input for the MLP block in level s + 1 s+1 ) and also the initial state of MLP blocks are all connected through backpropagation in the lowest frequency level. Given an arbitrary 1 ≤ s ≤ k 1\leq s\leq k , 𝜽 0 ( f s ) = arg min Φ 𝔼 𝒯 ∼ 𝒞 ( 1 ) [ ℓ ( Θ , 𝒯 , Φ ) ] , \displaystyle\bm{\theta}^{(f_{s})}_{0}=\underset{\Phi}{\arg\min}\;\;\mathbb{E}_{\mathcal{T}\sim\mathcal{C}^{(1)}}\Biggl[\ell(\Theta,\mathcal{T};\Phi)\Biggr], (73) where 𝒞 ( 1 ) \mathcal{C}^{(1)} is the context length of the MLP block in the lowest frequency level. Since the initial state of all memories are meta-learned in the lowest frequency, the most persistent knowledge of all components is the compression of the same context flow.