Nested Learning: The Illusion of Deep Learning Architectures — part 2/14
Cleaned-text code points [15720, 31637). Bibliography omitted here; source.md includes it.
In this paper, we aim to present a unifying learning paradigm that not only provides new insights about existing algorithms, methods, and architectures, but it also reveals a new dimension to stacking layers in deep learning with enhancement of the computational depth, and continual learning ability of models. After discussing preliminary concepts and backgrounds in § 2 , we present:
Nested Learning Paradigm ( § 3 ). To answer the questions raised above and to provide new insights on overcoming the design challenges in continual learning, architecture design, and computational depth of modern deep learning models, we present Nested Learning (NL)–a learning paradigm that allows each component of the machine learning model to have its own internal gradient flow on its own context in multiple levels, representing a model and its learning process (i.e., optimization) as an inter-connected system of nested, multi-level, and/or parallel optimization problems. We argue that the optimization process and the learning algorithms/architectures are fundamentally the same concepts but are in different levels of a system with different context (i.e., gradient vs. tokens). Furthermore, they are two inter-connected components where the learning algorithm/architecture generates the context for optimizers (i.e., the gradients), advocating for designing architecture-specific optimizers. We discuss different ways of knowledge transfer between levels, resulting in unifying and generalizing concepts like meta-learning, in-context learning, recurrent neural networks, hypernetworks, etc.
Optimizers and Architectures as Learning Module ( § 4 , § 5 ) . Building on the NL’s viewpoint, we argue that training a deep neural network with backpropagation process and gradient descent is a compression and an optimization problem that aims to train an associative memory to map layers’ inputs to their corresponding local error in the prediction. Accordingly, we argue that pre-training is a form of in-context learning, where the context is the entire pre-training data and the layers are compressing the context into their parameters. We demonstrate that such arguments are also valid for other popular gradient-based optimizers and they are associative memories that aim to compress the gradients into their parameters. From NL’s terminology, gradient-based optimizers such as gradient descent with momentum, Adam ( Kingma and Ba, 2014 ) , and AdaGrad ( Duchi et al., 2011 ) can be decomposed into a two-level nested optimization problems, each of which is optimized with a simple gradient descent. In particular, this viewpoint makes it apparent that for compressing the gradients, in theory, Adam is the optimal associative memory with respect to the element-wise L 2 L_{2} regression objective.
We revisit previous findings on representing architectures as associative memories ( Behrouz et al., 2025b ) and decompose their optimization process into a set of nested optimization problems, all of which optimized with gradient descent. Building on the above findings–i.e., popular gradient-based optimizers and modern architectures are both a set of nested and/or parallel optimization problems–we argue that the combination of these two–i.e., training an architecture with a specific optimizer–can also be represented as a set of nested and/or parallel optimization problem. Therefore, a neural learning module (a joint system of architecture and its training/optimization process) is a uniform model, in which all elements are linear or deep MLPs, while they are optimizing their own internal objective in different levels with different frequencies.
Building upon the associative memory perspective of optimizers, we design a set of new learning updates (optimization steps) with more expressive memory structure or memory management in compressing the gradients. In particular, we argue that the choice of optimizer depends on the context of the optimization. A powerful optimizer for compressing the gradients might not be the best choice for compressing the tokens. To this end, we present a new variant of gradient descent, called Delta Gradient Descent (DGD), that its update not only depends on the current input, but also the state of the weight of the neural network, resulting in capturing the dependencies of data samples without i.i.d. assumption.
Main Takeaways and Revisiting Common Terms: Continual and In-context Learning, Pre-Training, and Learning ( § 6 ) . We discuss the main takeaways of NL about principal concepts and revisit some common terms: (1) We argue that continual learning can be viewed as a learning problem on a sequences of incoming contexts or episodes where different levels are responsible for compressing their own in-context knowledge and transfer it to higher levels. Based on this, we advocate for designing models and pipelines that do not rely on test/training phase, and rather continually manage their knowledge and memory; (2) In-context learning is the characteristic of “having multiple nested levels”. Accordingly, Transformers in-context learning comes from being a non-parametric solution to a certain regression objective on tokens, while modern recurrent models uses parametric learning processes in their lower levels; (3) We further revisit other terminologies such as learning/memorization, hybrid architectures, looped architectures, and learned optimizers.
Continuum Memory System, Self-Referential Titans, and Hope ( § 7 , § 8 ). We generalize the traditional viewpoint of “long-term/short-term memory” (LSM) by presenting the Continuum Memory Systems (CMSs) and see memory as a distributed inter-connected system with a spectrum of frequency updates. In this design, higher-frequency neurons are responsible for fast adaption but store memories/knowledge for a short period of time, while lower frequency neurons are responsible for more persistent knowledge. Comparing to LSM, we show that this multi-frequency design results in a loop process for memory of the model, meaning that knowledge can partially be recovered when it is forgotten. While we mainly design this memory system as a replacement of MLP blocks in Transformers, we take advantage of this intuition to design Multi-scale Momentum Muon (M3) optimizer–an optimization algorithm with multiple momentum terms–further supporting the importance of CMSs design in different contexts.
Evaluations ( § 9 ). To support the effectiveness of our proofs-of-concept as well as the importance of nested learning design, we perform experimental evaluation on (1) Continual learning and in-context learning tasks including (i) learning new language, (ii) class incremental learning, and (iii) question/answering on a new corpus; (2) Long context understanding tasks, including needle-in-a-haystack ( Hsieh et al., 2024 ) and BABILong ( Kuratov et al., 2024 ) benchmarks; (3) Language modeling and common-sense reasoning tasks; (4) In-context recall and memorization tasks; (5) Language recognition tasks; and (6) comparing different optimizers, including our M3 optimizer. Our results indicate the effectiveness of NL viewpoint in designing models with continual learning ability, having multiple levels of computations, and self-referential process.
## 2 Preliminaries
In this section we discuss the notations and review the background concepts.
Notations. We let x ∈ ℝ N × d in x\in\mathbb{R}^{N\times d_{\text{in}}} be the input, ℳ t \mathcal{M}_{t} represent the state of memory/model ℳ \mathcal{M} at time t t , 𝐊 \mathbf{K} be the keys, 𝐕 \mathbf{V} be the values, and 𝐐 \mathbf{Q} be the query matrices. We use bold lowercase letters with subscript t t to refer to the vector corresponds to the input t t (i.e., 𝒌 t , 𝒗 t \bm{k}_{t},\bm{v}_{t} , and 𝒒 t \bm{q}_{t} ). We further refer to the distribution of any random variable 𝒯 \mathcal{T} as p ( 𝒯 ) p(\mathcal{T}) . Throughout the paper, we use simple MLPs with ℒ ℳ ≥ 1 \mathcal{L}_{\mathcal{M}}\geq 1 layers and residual connection as the architecture of the memory module ℳ ( ⋅ ) \mathcal{M}(\cdot) . When it is needed, we parameterized the memory module with 𝜽 ℳ ⊇ { W 1 , W 2 , … , W ℒ ℳ } \bm{\theta}_{\mathcal{M}}\supseteq\{W_{1},W_{2},\dots,W_{\mathcal{L}_{\mathcal{M}}}\} , which at least includes the parameters of linear layers in the MLP. We use superscript with parenthesis to refer to parameters in different levels of nested learning (different update frequency): i.e., either by using level index W ( ℓ ) W^{(\ell)} or its corresponding frequency W ( f ℓ ) W^{(f_{\ell})} .
Gradient Descent. Gradient descent is one of the most widely used optimization algorithms for high-dimensional problems, and its variants are standard tools for training large models. Given an objective ℒ ( ⋅ , ⋅ ) \mathcal{L}(\cdot;\cdot) , the stochastic gradient descent (SGD) with step size η t > 0 \eta_{t}>0 (a.k.a. learning rate) updates parameters by: W t + 1 = W t − η t ∇ W t ℒ ( W t , 𝒙 𝒕 ) , \displaystyle W_{t+1}\;=\;W_{t}-\eta_{t}\nabla_{W_{t}}\mathcal{L}(W_{t};\bm{x_{t}}), (1) for data sample 𝒙 𝒕 \bm{x_{t}} from training set. The gradient descent formulation admits several equivalent characterizations that are useful in analysis. One of those equivalent formulations is steepest - descent in the Euclidean metric, where one step of gradient descent is equivalent to: W t + 1 = arg min W { ⟨ ∇ W ℒ ( W t , 𝒙 𝒕 ) , W ⟩ + 1 2 η t ‖ W − W t ‖ 2 2 } , \displaystyle W_{t+1}\;=\;\arg\min_{W}\;\Big\{\,\langle\nabla_{W}\mathcal{L}(W_{t};\bm{x_{t}}),\,W\rangle\;+\;\tfrac{1}{2\eta_{t}}\|W-W_{t}\|_{2}^{2}\,\Big\}, (2) which is minimizing a first - order Taylor approximation regularized by a quadratic proximal term. The GD step above is precisely a proximal update on the linearization of ℒ ( ⋅ , ⋅ ) \mathcal{L}(\cdot;\cdot) at W t W_{t} , revealing an implicit bias toward small moves in L 2 L_{2} -distance. Accumulating steps (with a constant learning rate η \eta ) yields the follow - the - regularized - leader ( FTRL ) form W t + 1 = arg min W { ⟨ ∑ s = 1 t ∇ ℒ ( W s , 𝒙 s ) , W ⟩ + 1 2 η ‖ W − W 1 ‖ 2 2 } , \displaystyle W_{t+1}\;=\;\arg\min_{W}\Big\{\;\bigg\langle\sum_{s=1}^{t}\nabla\mathcal{L}(W_{s};\bm{x}_{s}),\,W\bigg\rangle\;+\;\frac{1}{2\eta}\,\|W-W_{1}\|_{2}^{2}\Big\}, (3) whose solution is W t + 1 = W 1 − η ∑ s = 1 t ∇ ℒ ( W s , 𝒙 s ) W_{t+1}=W_{1}-\eta\sum_{s=1}^{t}\nabla\mathcal{L}(W_{s};\bm{x}_{s}) . These two formulations are used in this paper interchangeably. However, our discussions and formulations are generally valid for many other optimization algorithms as well.
Meta Learning. Designing an effective machine learning model often requires making decisions about its architecture parameterized by 𝜽 ∈ Θ \bm{\theta}\in\Theta , objective ℒ ( θ ) \mathcal{L}(\theta) , and an optimizer, aiming to iteratively optimize the objective. Meta learning paradigm (or learning to learn) ( Schmidhuber et al., 1996 ; Finn et al., 2017 ; Akyürek et al., 2022 ; Irie et al., 2025 ) aim to automate a part of such decisions by modeling it as a two-level optimization procedure, in which the outer model aims to learn to set parameters for the inner procedure to maximize the performance across a set of tasks. That is, given an objective parameterized by a parameter Φ \Phi : i.e., ℓ ( 𝜽 , 𝒟 , Φ ) \ell(\bm{\theta},\mathcal{D};\Phi) , one can formalize the outer loop process as optimizing parameter Φ \Phi over a set of tasks: Φ ∗ = arg min Φ 𝔼 𝒯 i ∼ p ( 𝒯 ) [ ℓ ( θ , 𝒯 i , Φ ) ] , \displaystyle\Phi^{\ast}=\underset{\Phi}{\arg\min}\;\;\mathbb{E}_{\mathcal{T}_{i}\sim p(\mathcal{T})}\Biggl[\ell(\theta,\mathcal{T}_{i};\Phi)\Biggr], (4) where p ( 𝒯 ) p(\mathcal{T}) is the distribution of tasks. While initial studies on meta-learning used supervised settings for the outer loop ( Schmidhuber et al., 1996 ) , recently, a more flexible family of methods that use an unsupervised process for the outer loop has gained popularity ( Brown et al., 2020 ; Chen et al., 2022 ; Qu et al., 2025 ; Finn et al., 2017 ; Akyürek et al., 2022 ) . In addition to the growing interest to use meta learning methods for a diverse set of downstream tasks ( Finn et al., 2017 ; Chen et al., 2022 ; Qu et al., 2025 ; Munkhdalai et al., 2019 ; Irie et al., 2025 ) , in recent years, it also has shown popularity as a paradigm to design powerful sequence models ( Sun et al., 2024 ; Behrouz et al., 2025b ; Behrouz et al., 2025c ) .
Fast Weight Programmers (FWPs). Fast weight programmers (more recently refer to as linear Transformers) ( Schmidhuber, 1992 ; Hinton and Plaut, 1987 ; Ba et al., 2016 ; Schlag et al., 2021 ) , which also have been motivated from neuroscience perspective ( Gershman et al., 2025 ) , are recurrent neural networks whose memory (or hidden state) is matrix - valued : a time - varying fast - weight matrix ℳ t ∈ ℝ d out × d key \mathcal{M}_{t}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{key}}} that serves as a short - term memory. A separate “programmer” (slow net) maps each input 𝒙 t ∈ ℝ d in \bm{x}_{t}\in\mathbb{R}^{d_{\mathrm{in}}} to query, key, and value vectors and updates the fast weights online. A basic (Hebbian/outer - product) FWP—often called vanilla FWP—update its parameters with: ℳ t = α t ℳ t − 1 + 𝒗 t ϕ ( 𝒌 t ) ⊤ , \displaystyle\mathcal{M}_{t}\;=\;\alpha_{t}\mathcal{M}_{t-1}\;+\;\bm{v}_{t}\phi(\bm{k}_{t})^{\top}, (5) and retrieve from memory with y t = ℳ t ϕ ( q t ) y_{t}\;=\;\mathcal{M}_{t}\,\phi(q_{t}) , where ϕ ( ⋅ ) \phi(\cdot) is an element-wise feature map (often applied to both keys and queries). Unlike traditional RNNs or early variants of modern RNNs ( Schmidhuber and Hochreiter, 1997 ; Botev et al., 2024 ; Sun et al., 2023 ) with vector states, the matrix ℳ t \mathcal{M}_{t} is the recurrent state; it is written by rank - one update and read by a matrix–vector multiplication, providing a compact, learnable key–value memory with constant state size across time.
In-context Learning. The concept of “in-context learning” initially defined by Brown et al. (2020) as the ability of a language model to leverage knowledge acquired during pre-training in order to infer and perform a new task solely based on its context (e.g., few examples, or natural language instructions). This broad and general definition, which simply is applicable for any language model with any architectural backbones and/or objective, later was formalized in a way that only described in-context learning for Transformer architectures trained with next token prediction objectives. Accordingly, despite extensive research on the algorithms/problems that a transformer-based model can learn in-context ( Akyürek et al., 2022 ; Akyürek et al., 2024b ; Dherin et al., 2025 ; Zhang et al., 2024a ) , the in-context learning as its general form is relatively underexplored. Throughout this paper, we use the most general definition of “in-context learning” and refer to it as the ability of a model to adapt itself to and learn from a given context. Our NL formulation connects ICL with the concept of associative memory, offering a unified explanation for the ICL capabilities of models regardless of their architectural backbone and/or objectives.
## 3 Nested Learning
This section discusses the motivations, formal definitions, and general high-level implications of Nested Learning (NL). We start with a formulation of associative memory and then by using step-by-step examples, we build the intuition behind architecture decomposition and its connection to modeling a neural network as an integrated system of optimization problems. We aim to first show how existing methods and concepts in deep learning fall under the NL paradigm and then we present new formulations that go beyond traditional methods and/or provide insights on how to improve existing algorithms and designs.
### 3.1 Associative Memory