Nested Learning: The Illusion of Deep Learning Architectures — part 13/14

Cleaned-text code points [183293, 196798). Bibliography omitted here; source.md includes it.

The results are reported in Table 1 . Large models show significant performance drop with increasing the sequence length, where all fail around 128K-256K context length. The RAG augmented model also show a drop with increasing the context, but it is able to relatively maintain its performance after 256K context length. Among fine-tuned models, Titans, ARMT, and Hope show competitive results until 1M context length, but the performance of both Titans and ARMT drop fast after that point. Hope maintains its good performance even for 10M context length, mainly due to its CMS design. It is notable that the performance of all small models, including Hope , can drop significantly when used without fine-tuning. The reason is, compressing large context (e.g., 10M), in addition to a powerful memory management in the high-frequency level, requires enough capacity to compress 10M tokens or at least tokens that are needed for the final answer. The fine-tuning step helps the models to adjust their lower-frequency levels to adapt fast and so properly manage their memory in the higher-frequency levels.

### 9.3 Hope : Language Modeling and Common-Sense Reasoning

In this section, we aim to study Hope as a backbone of a language model and evaluate it on common language modeling and common-sense reasoning tasks with the setup of: • Datasets: We evaluate Hope and baselines on Wikitext ( Merity et al., 2017 ) , LMB ( Paperno et al., 2016 ) , PIQA ( Bisk et al., 2020 ) , HellaSwag ( Zellers et al., 2019 ) , WinoGrande ( Sakaguchi et al., 2021 ) , ARC-easy (ARC-e) and ARC-challenge (ARC-c) ( Clark et al., 2018 ) , SIQA ( Sap et al., 2019 ) , and BoolQ ( Clark et al., 2019 ) benchmarks.

• Baselines: As for the baselines, similar to Section 9.2 , we use RetNet ( Sun et al., 2023 ) and DeltaNet ( Schlag et al., 2021 ) as the representatives of the models that are purely based on Hebbian- or Delta-rule, and two modern matrix-valued recurrent models with the best performance compared to others: i.e., RWKV-7 ( Peng et al., 2025b ) and Comba ( Hu et al., 2025 ) . As another group of baselines, we compare with attention-free deep memory modules with diverse internal attentional bias of dot-product, L 2 L_{2} , and L p L_{p} regression: i.e., TTT ( Sun et al., 2024 ) , Miras ( Behrouz et al., 2025b ) , DLA ( Behrouz et al., 2025a ) and Titans ( Behrouz et al., 2025c ) . Finally, we also compare with Transformers ( Vaswani et al., 2017 ; Dubey et al., 2024 ) as well as the hybrid of attention and linear RNN, Samba ( Ren et al., 2024 ) .

• Training: We train models with about 760M and 1.3B parameters, trained with 30B and 100B tokens, respectively, from a mixture of FineWeb-Edu ( Penedo et al., 2024 ) and long-context documents with a vocabulary size of 32K to train all the models from scratch. All models are optimized using AdamW with tuned learning rate for each model and with the default optimizer configuration in Behrouz et al. (2025c) .

The results are reported in Table 2 . Hope outperforms all the baselines on the average performance in both language modeling and common-sense reasoning benchmarks. Interestingly, with scaling the parameters, Hope show higher performance gain compare to other attention-free models.

### 9.4 Hope : In-context Recall Tasks and MAD Synthetic Benchmark

In-context recall is often referred to as one of the challenging benchmarks for attention-free models. In this section, we follow Arora et al. (2024) and perform experiments on SWDE ( Lockard et al., 2019 ) , NQ ( Kwiatkowski et al., 2019 ) , DROP ( Dua et al., 2019 ) , FDA ( Arora et al., 2023 ) , SQUAD ( Rajpurkar et al., 2016 ) , and TQA ( Kembhavi et al., 2017 ) to evaluate the effectiveness of Hope ’s design. We use the same set of baselines and experimental setup as the above previous section. The results are reported in Section 9.4 . Transformers achieve the best performance, while Hope show competitive results, outperforming all the attention-free baselines and closing the gap with Transformers.

We also study the performance of Hope on MAD benchmark ( Poli et al., 2024 ) , which is a synthetic benchmark, evaluating the performance of models in recall, memorization, compression, and copying tasks. The results are reported in Section 9.4 . Hope achieves the best results compared to baselines.

### 9.5 Language Recognition Tasks

One of the critical limitations of Transformers is on non-parallelizable tasks, where the recurrence plays a significant rule in achieving proper performance. An example of such tasks is state tracking problem, where the model needs to track its state given a sequence of instructions (movement, etc.), and several studies have shown that Transformers both in theory and practice significantly underperforms models with non-linear recurrence ( Merrill et al., 2024 ; Grazzi et al., 2025 ) . In this section, we focus on formal language recognition tasks and follow the construction of the benchmark by Irie et al. (2023) . The results are reported in Table 5 . Hope achieves the perfect score on all the tasks, similar to other non-linear recurrent models; e.g., LSTM Schmidhuber and Hochreiter, 1997 and SRWM ( Irie et al., 2022b ) . The main advantage of Hope , however, is the fact that when it is needed, it has parallelizable training and so can scale to larger scales for language modeling tasks.

### 9.6 Hope : Ablation Studies and Scaling

In this section, we first study the importance of design choices we have made for the Hope architecture by performing ablation studies, where we remove or change one of the Hope ’s components at a time. Note that, in the previous experiments, we have evaluated the significance of some components: E.g., Table 1 and Figure 7 have already been shown the effect of number of levels as well as the frequency of update on the performance of Hope in continual learning. In this section, to compare different variants, we use the average perplexity of models in language modeling as well as the average accuracy in common-sense reasoning tasks. The results are reported in Table 6 . (1) The first row, replaces Delta Gradient Descent with a simple gradient descent in the design of self-modifying Titans; (2) The second row removes the momentum term in the self-modifying Titans; (3) removes the weight decay; (4) removes the CMS from Hope ’s architecture; (5, 6, 7) transfer the projections for 𝒌 , 𝒗 , 𝒒 \bm{k},\bm{v},\bm{q} from higher-frequency level to the lowest-frequency level. All the components of Hope contribute to its superior performance in these tasks and removing or changing each of them can damage the model’s perplexity in language modeling and/or accuracy in common-sense reasoning tasks.

### 9.7 Expressive Optimizers

In this section, we evaluate the performance of our M3 optimizer from both aspects of effectiveness in finding the solution and also efficiency of training in large scales:

ImageNet. In this experiment, we focus on ViT ( Dosovitskiy et al., 2021 ) architecture for vision tasks and pre-train it on ImageNet-21K, which consists of 11M images corresponding to 10,450 classes. We use a patch size of 16, MLP dimension of 1536 and 3072 for the two scales of 24M and 86M models, respectively. We control all other components and finetuned hyperparameters for each optimizer separately to make sure a fair comparison. We vary the optimizer for training the ViT to understand which of the optimizers find more effective solutions in the same number of training steps. The training and test loss of trained models with different optimizers are reported in Figure 9.6 . Our M3 shows the best training/test loss compared to both AdamW and Muon.

Efficiency in Large Models. Due to the small size of the model that is needed for ImageNet, we also perform an efficiency comparison when training a language model. To this end, we use two scales of 140M and 1.3B and train the same Transformer model using different optimizers of Muon ( Jordan et al., 2024 ) , AdaMuon ( Si et al., 2025 ) , and our M3. The results are reported in Figure 9.6 . Our M3 optimizer, due to the use of multiple momentums (memories) is reletively slower compare to the Muon optimizer, and show on par efficiency with AdaMuon.

## 10 Conclusion

In this paper, we introduced Nested Learning (NL) a new learning paradigm, in which modern machine learning systems are modeled as inter-connected, multi-level optimization problems, each with its own context flow and update frequency. Within this view, both architectures and optimizers are instances of nested systems of associative memories that compress their own context (either is tokens, gradients, or higher-level signals) into internal parameters. This perspective reframes pre-training, in-context learning, and continual learning as manifestations of the same underlying mechanism: learning to compress and reuse context at different levels and time scales. This perspective recasts backpropagation, momentum, and preconditioning as associative memory mechanisms, and explains a wide family of existing methods as specific design points in a larger (previously hidden) space.

Building on NL’s viewpoint, we derived generalized gradient-based updates: e.g., Delta Gradient Descent, Delta Momentum, Multi-scale Momentum Muon (M3), and reinterpreted modern sequence architectures nested associative memories. To enhance the memory processing, we introduced Continuum Memory System (CMS), a new formulation for memory that generalizes the traditional viewpoint of “long-term/short-term memory blocks”. Our Hope architecture based on a self-modifying Titans and CMS improves continual learning and long-context reasoning capabilitites, while remaining competitive as a general backbone.

Is Catastrophic Forgetting Solved? While Hope and CMS have shown promising results in reducing catastrophic forgetting in the tasks we empirically studied, the undesirable phenomenon of catastrophic forgetting is not “solved” in general. From nested learning viewpoint on learning and backpropagation process, catastrophic forgetting is a natural consequence of compression, where the limited capacity of the network forces the model to forget so that it retains capacity for new information. We view NL as a roadmap rather than a destination: it suggests that progress on continual learning, long-context reasoning, modern optimizers, and self-modifying models will come from better exploiting the extra design axis of levels rather than from ever-deeper static networks.

## Appendix A Generalized Formulation for Nested Learning and Nested Systems

In this section, we discuss the generalized version of nested systems and NSAM , we discussed in Section 3 :

###### Definition 6 ((Generalized) Nested System) .

A (ordered) nested system is a system with K K (ordered) levels such that each level 1 ≤ k ≤ K 1\leq k\leq K consists of a set of optimization problems { ( ℒ i ( k ) , 𝒞 i ( k ) , 𝚯 i ( k ) ) } i = 1 N k \{(\mathcal{L}^{(k)}_{i},\mathcal{C}^{(k)}_{i},\bm{\Theta}^{(k)}_{i})\}_{i=1}^{N_{k}} , where ℒ i ​ ( ⋅ , ⋅ ) \mathcal{L}_{i}(\cdot;\cdot) is the optimization objective in the i i -th problem, 𝒞 i \mathcal{C}_{i} is its context (the data that is optimized on), 𝚯 i \bm{\Theta}_{i} is the set of its parameters, and each optimization problem is optimized using gradient descent, or equivalently: 𝜽 i t + 1 ( k ) = arg ⁡ min 𝚽 i ( k ) ​ ℒ i ( k ) ​ ( 𝚽 i ( k ) , x t + 1 ) + 1 2 ​ η i t + 1 ( k ) ​ ‖ 𝚽 i ( k ) − 𝜽 i t ( k ) ‖ 2 2 where ​ x t + 1 ∼ 𝒞 i ( k ) , and 𝚽 i ( k ) ∈ 𝚯 i ( k ) . \displaystyle{\bm{\theta}_{i}}^{(k)}_{t+1}=\arg\min_{\bm{\Phi}^{(k)}_{i}}\>\>\mathcal{L}^{(k)}_{i}(\bm{\Phi}^{(k)}_{i};x_{t+1})+\frac{1}{2{\eta_{i}}^{(k)}_{t+1}}\>\|\bm{\Phi}^{(k)}_{i}-{\bm{\theta}_{i}}^{(k)}_{t}\|^{2}_{2}\qquad\text{where}\>\>x_{t+1}\thicksim\mathcal{C}^{(k)}_{i},\quad\text{and}\quad{\bm{\Phi}_{i}}^{(k)}\in\bm{\Theta}^{(k)}_{i}. (98)

###### Definition 7 ((Generalized) Nested System of Associative Memories) .

A nested system of associative memory ( NSAM ) is a system with K K (ordered) levels such that each level 1 ≤ k ≤ K 1\leq k\leq K consists of a set of optimization problems { ( ℒ i ( k ) , 𝒞 i ( k ) , 𝚯 i ( k ) ) } i = 1 N k \{(\mathcal{L}^{(k)}_{i},\mathcal{C}^{(k)}_{i},\bm{\Theta}^{(k)}_{i})\}_{i=1}^{N_{k}} , where 𝒞 i = { ( 𝐤 j ( i ) , 𝐯 j ( i ) ) } j = 1 L i \mathcal{C}_{i}=\{(\bm{k}^{(i)}_{j},\bm{v}^{(i)}_{j})\}_{j=1}^{L_{i}} is a set of ground-truth mappings, ℒ i ​ ( ⋅ , ⋅ , ⋅ ) \mathcal{L}_{i}(\cdot;\cdot,\cdot) is the optimization objective that measures the quality of memory learned mappings in the i i -th problem, 𝚯 i \bm{\Theta}_{i} is the set of memory parameters, and each optimization problem is optimized using gradient descent: 𝜽 i t + 1 ( k ) = arg ⁡ min 𝚽 i ( k ) ​ ℒ i ( k ) ​ ( 𝚽 i ( k ) , 𝒌 t + 1 ( i ) , 𝒗 t + 1 ( i ) ) + 1 2 ​ η i t + 1 ( k ) ​ ‖ 𝚽 i ( k ) − 𝜽 i t ( k ) ‖ 2 2 where ​ ( 𝒌 t + 1 ( i ) , 𝒗 t + 1 ( i ) ) ∼ 𝒞 i ( k ) , and ​ 𝚽 i ( k ) ∈ 𝚯 i ( k ) . \displaystyle{\bm{\theta}_{i}}^{(k)}_{t+1}=\arg\min_{\bm{\Phi}^{(k)}_{i}}\>\>\mathcal{L}^{(k)}_{i}(\bm{\Phi}^{(k)}_{i};\bm{k}^{(i)}_{t+1},\bm{v}^{(i)}_{t+1})+\frac{1}{2{\eta_{i}}^{(k)}_{t+1}}\>\|\bm{\Phi}^{(k)}_{i}-{\bm{\theta}_{i}}^{(k)}_{t}\|^{2}_{2}\quad\>\>\text{where}\>\>(\bm{k}^{(i)}_{t+1},\bm{v}^{(i)}_{t+1})\thicksim\mathcal{C}^{(k)}_{i},\>\>\>\>\text{and}\>\>\>\>{\bm{\Phi}_{i}}^{(k)}\in\bm{\Theta}^{(k)}_{i}. (99)

## Appendix B Adam, AdaGrad, and Other Similar Optimizers as Associative Memory Modules