Nested Learning: The Illusion of Deep Learning Architectures — part 1/14

Cleaned-text code points [0, 15720). Bibliography omitted here; source.md includes it.

# Nested Learning: The Illusion of Deep Learning Architecture

###### Abstract

Over the last decades, developing more powerful neural architectures and simultaneously designing optimization algorithms to effectively train them have been the core of research efforts to enhance the capability of machine learning models. Despite the recent progresses, particularly in developing Language Models (LMs), there are fundamental challenges and unanswered questions about how such models can continually learn/memorize, self-improve, and find effective solutions . In this paper, we present a new learning paradigm, called Nested Learning (NL), that coherently represents a machine learning model with a set of nested, multi-level, and/or parallel optimization problems, each of which with its own “ context flow ”. Through the lenses of NL, existing deep learning methods learns from data through compressing their own context flow, and in-context learning naturally emerges in large models. NL suggests a philosophy to design more expressive learning algorithms with more “ levels ”, resulting in higher-order in-context learning and potentially unlocking effective continual learning capabilities. In addition to its neuro-scientific motivation, we advocate for NL by presenting three core contributions: (1) Expressive Optimizers: We show that known gradient-based optimizers, such as Adam, SGD with Momentum, etc., are in fact associative memory modules that aim to compress the gradients’ information (by gradient descent). Building on this insight, we present other “more expressive" optimizers with deep memory and/or more powerful learning rules; (2) Self-Modifying Learning Module: Taking advantage of NL’s insights on learning algorithms, we present a sequence model that learns how to modify itself by learning its own update algorithm; and (3) Continuum Memory System: We present a new formulation for memory system that generalizes the traditional viewpoint of “long-term/short-term memory”. Combining our self-modifying sequence model with the continuum memory system, we present a continual learning module, called Hope , showing promising results in language modeling, knowledge incorporation, and few-shot generalization tasks, continual learning, and long-context reasoning tasks.

“We cannot solve our problems with the same thinking we used when we created them!"

— Attributed to Albert Einstein

## 1 Introduction

For decades, AI research has focused on designing machine learning algorithms that learn from data ( Pitts, 1943 ; McCulloch, 1949 ; McCulloch and Pitts, 1948 ; Samuel, 1959 ) or experience ( Silver and Sutton, 2025 ; Sutton et al., 1998 ; Connell and Mahadevan, 1999 ) ; often by optimizing an objective ℒ ⁡ ( 𝜽 ) \mathcal{L}(\bm{\theta}) over parameters 𝜽 ∈ Θ \bm{\theta}\in\Theta with gradient-based methods. While traditional machine learning techniques required careful engineering and domain expertise to design feature extractors, limiting their ability to directly process and learn from natural data ( LeCun et al., 2015 ) , deep representation learning offered a fully automated alternative to discover the representations needed for the task. Thereafter, deep learning has been an inseparable part of the large-scale computational models with seminal success in chemistry and biology ( Jumper et al., 2021 ) , games ( Silver et al., 2016 ; Silver et al., 2018 ) , computer vision ( Krizhevsky et al., 2012 ; Dosovitskiy et al., 2021 ) , and multimodal and natural language understanding ( Comanici et al., 2025 ; Liu et al., 2024a ; Achiam et al., 2023 ) .

Stacking of multiple layers, as it is done in deep learning models, provides the models with better expressive power in representing complex features, and more internal computations (e.g., #FLOPS) ( Montúfar et al., 2014 ; Poole et al., 2016 ; Hestness et al., 2017 ) , all of which are critical and desirable characteristics for static tasks that require in-distribution predictions over an a-priori fixed set. This deep design, however, is not a universal solution to all the challenges and cannot help the expressive power of the models in multiple aspects, for example: (i) The computational depth of deep models might not change with more layers ( Merrill et al., 2022 ; Sanford et al., 2024 ) , leaving their ability to implement complex algorithms untouched compared to traditional shallow approaches ( Merrill et al., 2024 ) ; (ii) The capacity of some class of parameters might show marginal improvement with increasing the depth/width of the model ( Kaplan et al., 2020 ) ; (iii) The training process might converge to a suboptimal solution, mainly due to the suboptimal choice of the optimizer or its hyperparameters; and (iv) The model’s ability to fast adapt to a new task, continually learn, and/or generalize to out-of-distribution data might not change with stacking more layers and requires more careful designs.

The core part of the efforts to overcome the above challenges and to enhance the capability of deep learning models concentrate on: (1) developing more expressive class of parameters (i.e., neural architectures) ( Schmidhuber and Hochreiter, 1997 ; Fukushima, 1980 ; Vaswani et al., 2017 ; Krizhevsky et al., 2012 ; Behrouz et al., 2025c ) ; (2) introducing objectives that can better model the tasks ( Rumelhart et al., 1986 ; Goodfellow et al., 2020 ; Alshammari et al., 2025 ; Kingma and Welling, 2014 ; Hjelm et al., 2019 ) ; (3) designing more efficient/effective optimization algorithms to find better solutions or with more resilience to forgetting ( Kingma and Ba, 2014 ; Jordan et al., 2024 ; Gupta et al., 2018 ; Farajtabar et al., 2020 ) ; and (4) scaling the model size to enhance its expressivity, when the “right” choice of architecture, objective, and optimization algorithms are made ( Hoffmann et al., 2022 ; Brown et al., 2020 ; Kaplan et al., 2020 ) . Collectively, these advancements and new findings on scaling patterns of deep models have established the foundations upon which Large Language Models (LLMs) have been built.

The development of LLMs marks a pivotal milestone in deep learning research: a paradigm shift from task-specific models to more general-purpose systems with various emergent capabilities as a result of scaling the “right” architectures ( Brown et al., 2020 ; Schaeffer et al., 2023 ) . Despite all their success and remarkable capabilities in diverse sets of tasks ( Comanici et al., 2025 ; Nijkamp et al., 2023 ; Wang et al., 2023 ) , LLMs are largely static after their initial deployment phase, meaning that they successfully perform tasks learned during pre- or post-training, but are unable to continually acquire new capabilities beyond their immediate context. The only adaptable component of LLMs is their in-context learning ability–a (known to be emergent) characteristic of LLMs that enables fast adaption to the context and so perform zero- or few-shot tasks ( Brown et al., 2020 ) . Beyond in-context learning, recent efforts to overcome the static nature of LLMs either are computationally expensive, require external components, lack generalization, and/or might suffer from catastrophic forgetting ( Eyuboglu et al., 2025 ; hongzhou yu et al., 2025 ; Akyürek et al., 2024a ) , which has led researchers to question if there is a need to revisit how to design machine learning models and if a new learning paradigm beyond stacking of layers is required to unleash the capabilities of LLMs in continual setups.

Current Models only Experience the Immediate Present. As an analogy and to better illustrate the static nature of LLMs, we use the example of anterograde amnesia–a neurological condition where a person cannot form new long-term memories after the onset of the disorder, while existing memories remain intact ( Scoville and Milner, 1957 ) . This condition limits the person’s knowledge and experiences to a short window of present and long past–before the onset of the disorder–which results in continuously experiencing the immediate present as if it were always new. The memory processing system of current LLMs suffer from a similar pattern. Their knowledge is limited to either, the immediate context that fits into their context window, or the knowledge in MLPs that stores long-past, before the onset of “end of pre-training.” This analogy, has motivated us to take inspiration from neurophysiology literature and how brain consolidate its short-term memories.

### 1.1 Human Brain Perspective and Neurophysiological Motivation

Human brain is highly efficient and effective when it comes to continual learning, which is often attributed to neuroplasticity—the brain’s remarkable capacity to change itself in response to new experiences, memories, learning, and even damage ( Pascual-Leone et al., 2005 ; Johnston, 2009 ) . Recent studies support that the formation of Long-term memory involves at least two distinct but complementary consolidation processes ( Goto et al., 2021 ; Frey and Morris, 1997 ; Yang et al., 2024 ) : (1) A rapid “online” consolidation (also known as synaptic consolidation) phase occurs immediately or soon after learning, even during wakefulness. This is when new and initially fragile memory traces are stabilized and begin transferring from short-term to long-term storage; (2) An “offline” consolidation (also known as systems consolidation) process repeats the replay of the recently encoded patterns—during sharp - wave ripples (SWRs) in the hippocampus, coordinated with cortical sleep spindles and slow oscillations—strengthens and reorganizes the memory and supports transfer to cortical sites ( Ji and Wilson, 2007 ; Peyrache et al., 2009 ; Foster and Wilson, 2006 ) .

Coming back to the analogy of anterograde amnesia, evidence indicates that the condition can impact both stages, but especially the online consolidation phase, mainly due to the fact that hippocampus is the gateway for encoding new declarative memories, and so its damage means new information never will be stored in long-term memory. As mentioned above, the design of LLMs, and more specifically Transformer-based backbones, suffers from a similar condition after the pre-training phase. That is, the information provided in the context, never impacts the long-term memory parameters (e.g., feedforward layers), and so the model is not capable of acquiring new knowledge or skill, unless the information is still stored in the short-term memory (e.g., in-context or attention). To this end, although the second stage is equally, or even more, crucial for the consolidation of memories, and its absence can damage the process and might cause loss of memory ( Drummond et al., 2000 ; Yoo et al., 2007 ) , in this work, we focus on the first stage: memory consolidation as an online process. As discussed earlier, the memory processing, its online consolidation, and so continual learning ability of humans are known to be highly relied on the neuroplasticity as well as neural oscillations ( Bliss and Collingridge, 1993 ; Buzsaki and Draguhn, 2004 ; Klinzing et al., 2019 ) .

Multi Time scale Processing System. Brain oscillations (also known as brainwaves)–a rhythmic fluctuations in brain activity–is not mere byproducts of brain function but is increasingly understood to play a crucial role in various cognitive functions such as attention, memory, and decision-making, and to be a core mechanism for organizing neural computation, coordinating communication between brain regions, and gating the synaptic plasticity that underlies learning and memory ( Fries, 2015 ; Fell and Axmacher, 2011 ; Cavanagh and Frank, 2014 ) . These brainwaves are the results of the brain coordinating its computations in different timescales and frequency updates, where each frequency determines how often groups of brain neurons become active and share updated information. More specifically, such neural oscillations are typically categorized into distinct frequencies, each of which has been associated with different cognitive functions and, critically, different timescales of information processing: ranging from (1) fast Gamma waves (frequency of 30 30 - 150 150 Hz) that are mainly associated with sensory information to (2) Beta waves (frequency of 13 13 - 30 30 Hz) that are mainly associated with active thinking ( Buzsaki and Draguhn, 2004 ; Buschman and Miller, 2007 ; Lundqvist et al., 2016 ) , and (3) slow Delta and Theta waves (frequency of 0.5 0.5 - 8 8 Hz), mainly responsible for memory consolidation and learning ( Diekelmann and Born, 2010 ; Marshall et al., 2006 ; Ngo et al., 2013 ; Staresina et al., 2015 ; Heusser et al., 2016 ; Daume et al., 2024 ) .

In deep learning models, however, the weights of the architectures are fixed at test time and also it is common in pre-training to use the same update rate for all the blocks/layers in the model. Later, in Section 6 , however, we show that in-context learning provides an extreme case of this design and in fact, Transformer architectures are based on two extreme frequencies of update: i.e., ∞ \infty and 0 0 for attention and MLP blocks, respectively.

Brain’s Uniform and Reusable Structure. As discussed earlier, neuroplasticity is the brain’s remarkable capability to change itself in response to new memories, knowledge, and even damage ( Pascual-Leone et al., 2005 ; Johnston, 2009 ) . This characteristic suggests a uniform architecture where neural elements are not rigidly dedicated to one function but are instead reusable, capable of being flexibly redeployed to support different cognitive needs. One real-world example of neural reusability is hemispherectomy–the surgical removal or disabling of one cerebral hemisphere, usually to alleviate severe epilepsy. Amazingly, if this surgery is done in childhood, patients can lead largely normal lives into adulthood with high functioning cognition and intact neural network organization that contains all the same core brain networks present in a typical two-hemisphere brain (networks for language, vision, etc.). This extraordinary outcome provides real-life proof of the brain’s uniform architecture. That is, even half a brain can reallocate resources and reorganize so that the person can function extremely well. Such cases, along with documented instances of individuals living relatively normally with missing pieces of cortex, highlight the brain’s uniform and reusable structure.

Furthermore, this interpretation of brain’s uniform and reusable structure demonstrate that memory in human brain is not an isolated system in some specific areas, and mainly is distributed across brain. That is, contrary to the traditional models of memory that often implied that different types of memory reside in distinct brain structures (e.g. short-term memory in frontal cortex vs. long-term memory in the hippocampus and cortex), modern research advocate for distributed neural circuits memory processing across multiple regions ( Christophel et al., 2017 ; Roy et al., 2022 ; Kitamura et al., 2017 ) .

The modern deep learning architectures in recent years, however, at least on the surface, seem to be heterogeneous and are based on a combination of a subset of self-attention variants ( Vaswani et al., 2017 ) , modern recurrent neural networks ( Katharopoulos et al., 2020 ; Schlag et al., 2021 ; Peng et al., 2025a ; Behrouz et al., 2025c ) , canon layers ( Allen-Zhu, 2025 ) , global convolutions ( Poli et al., 2023 ; Hasani et al., 2023 ) , and MLP blocks ( Shazeer, 2020 ) . This raises the question of whether we need a new uniform architecture, or if our beliefs about the heterogeneity of current models need to be revisited.

### 1.2 Contributions and Roadmap