Nested Learning: The Illusion of Deep Learning Architectures β part 12/14
Cleaned-text code points [167877, 183293). Bibliography omitted here; source.md includes it.
In the previous sections, we first discussed Continuum Memory System (CMS) that allows for more persistent storage of memories and defines memory as a spectrum of blocks with different frequencies of update. Due to the larger capacity and constraints for scaling the parameters, often CMS requires simple learning rule but higher capacity to store more persistent knowledge. On the other hand, in the previous section, we discussed the design of a self-modifying Titans, where it can generate its own keys and so learning update to better adapt to the context. Contrary to CMS, the self-modifying Titans has a small capacity but is using a complex and expressive learning rule. Accordingly, these two systems seem to be complementary and their combination can enhance the model expressiveness from different aspects.
To this end, we present Hope architecture: A neural learning module that incorporates self-modifying Titans followed by Continuum Memory System. The Hope design is illustrated in Figure 5 . Formally, let π t β β d \bm{x}_{t}\in\mathbb{R}^{d} for t = 1 , β¦ , L t=1,\dots,L be the input, the Hope forward pass is defined as (we remove the normalization and convolution layers for the sake of clarity): π t = β³ memory , t β 1 β ( π t ) , π t = β³ π , t β 1 β ( π t ) , π t = β³ π , t β 1 β ( π t ) , Ξ· t = β³ Ξ· , t β 1 β ( π t ) , Ξ± t = β³ Ξ± , t β 1 β ( π t ) , \displaystyle\bm{o}_{t}={\mathcal{M}_{\text{memory},t-1}}\left(\bm{q}_{t}\right),\qquad\bm{k}_{t}={\mathcal{M}_{\bm{k},t-1}}\left(\bm{x}_{t}\right),\qquad\bm{v}_{t}={\mathcal{M}_{\bm{v},t-1}}\left(\bm{x}_{t}\right),\qquad\eta_{t}={\mathcal{M}_{\eta,t-1}}\left(\bm{x}_{t}\right),\qquad\alpha_{t}={\mathcal{M}_{\alpha,t-1}}\left(\bm{x}_{t}\right), (94) π ^ β‘ , t = β³ β‘ , t β 1 β ( π t ) , \displaystyle\hat{\bm{v}}_{\square,t}={\mathcal{M}_{\square,t-1}}\left(\bm{v}_{t}\right), (95) β³ β‘ , t = β³ β‘ , t β 1 ( Ξ± t π° β Ξ· t π t π t β€ ) β Ξ· t β β β³ β‘ , t β 1 ( β³ β‘ , t β 1 ; π t , π ^ β‘ , t ) , β‘ β { π , π , π , Ξ· , Ξ± , memory } . \displaystyle{\mathcal{M}_{\square,t}}={\mathcal{M}_{\square,t-1}}\left(\alpha_{t}\bm{I}\>-\>\eta_{t}\bm{k}_{t}\bm{k}_{t}^{\top}\right)-\eta_{t}\nabla\mathcal{L}_{{\mathcal{M}_{\square,t-1}}}\left({\mathcal{M}_{\square,t-1}};\bm{k}_{t},\hat{\bm{v}}_{\square,t}\right),\quad\square\in\{\bm{k},\bm{v},\bm{q},\eta,\alpha,\text{memory}\}. (96) π t = MLP ( f k ) ( MLP ( f k β 1 ) ( β― MLP ( f 1 ) ( π t ) ) ) , \displaystyle\bm{y}_{t}=\texttt{MLP}^{(f_{k})}(\texttt{MLP}^{(f_{k-1})}(\cdots\texttt{MLP}^{(f_{1})}(\bm{o}_{t}))), (97) where the blockβs output for token t t is π t \bm{y}_{t} . In our experiments, we also normalize π \bm{q} and π \bm{k} with L 2 L_{2} normalization and also use local convolutions with window size of 4.
Hope-Attention. We also use another variant of Hope , in which we simply replace the self-modifying Titans with softmax global attention ( Vaswani et al., 2017 ) .
## 9 Experiments
In this section, we empirically evaluate the performance of different components we discussed throughout the paper. More specifically, (1) we first focus on the presented optimization algorithms and compare them with state-of-the-art methods; (2) We then focus on in-context and continual learning tasks and show how the nested learning paradigm and more specifically higher order in-context learning enhances the capabilities of models. We compare the continuum memory system with simple MLP layers and discuss how a pre-trained model can be adapted to be a continual learner; (3) We then focus on the language modeling and long context understanding of Hope model and compare it with Transformers and modern recurrent architectures. The details of the experiments and their setups are explained in the corresponding sections.
### 9.1 Hope : Continual Learning and Long Context Understanding
One of the main goals of NL is to enhance the continual learning capabilities, and so in this section, we evaluate NL and its implications such as Continuum Memory System (CMS) and Hope on multiple continual learning and long context understanding tasks. For each of the tasks, we use the best reported results on the benchmark as the baselines.
Class Incremental Learning. First, we focus on class-incremental learning tasks on three datasets: β’ CLINC ( Larson et al., 2019 ) : CLINC is a multi-domain intent classification benchmark designed for task-oriented dialog systems, with a special focus on detecting out-of-scope (OOS) queries. It has 150 in-scope intent classes spanning 10 broad domains (e.g. Banking, Travel, Home, Weather, Small Talk, etc.) with 23.7K total queries, of which 22.5K are in-scope and 1.2K are out-of-scope.
β’ Banking ( Casanueva et al., 2020 ) : Banking dataset is a single-domain intent classification benchmark focused on fine-grained customer service queries in the banking domain. In this dataset, each example is a short customer query (e.g. βHow can I reset my card PIN?β) that must be classified into the correct banking intent/category. There are 3083 total examples labeled with the 77 intents (heavily imbalanced classes).
β’ DBpedia ( Auer et al., 2007 ) : DBpedia is a text classification benchmark from Wikipedia where article abstracts are expected to be categorized into ontology topic classes. In other words, given a short Wikipedia description, the goal is to predict its high-level topic/category (such as whether the article is about a book, a film, an animal, a place, etc.). The dataset has over 340K examples labeled across those 70 second-level classes, but we sample 10K training and 1K test instances for the 70-class DBpedia task.
As the backbone of our Hope models we use Llama3-8B and Llama-3B ( Dubey et al., 2024 ) , and then employ our technique discussed in Section 7.3 to make the MLP blocks capable of adaption, placing them in different levels with different frequency updates followed by continual pre-training with 15B tokens. Following Momeni et al. (2025) , we use simple in-context learning (ICL) capability of Llama-3 models (with the same process of continual pre-training with 15B tokens but without any change in the MLP blocks), Elastic Weight Consolidation (EWC) ( Kirkpatrick et al., 2017 ) , and In-context Continual Learning with an External Learner (InCA) ( Momeni et al., 2025 ) as the baselines of our evaluation. The results are reported in Figure 6 . Hope shows the best performance across all continual learning baselines, including models with external learner (i.e., InCA). Comparing Hope with ICL, the main difference comes from Hope βs multiple levels of in-context learning (or equivalently, different frequency of updates for MLP blocks), indicating the effectiveness of CMSβs design for enhancing continual learning capabilities. Furthermore, the superior performance of Hope , compared to InCA and EWC, indicates that the knowledge transfer between levels plays a critical role in the performance of the model.
The Effect of Levels on In-context Learning. Despite showing improvement when using CMS in the above tasks, to better understand and evaluate the effect of levels and their frequency on the in-context level ability of the model, we perform in-context question/answering and multi-key long context understanding. More specifically, we use: β’ LongHealth ( Adams et al., 2025 ) : This is a benchmark for long-context clinical question answering with multiple-choice QA tasks based on extensive fictional patient records, testing an LLMβs ability to extract and reason over detailed medical documents. The dataset includes 20 comprehensive patient case documents (across various diseases), each about 5.1Kβ6.8K words in length, and we use 200 questions sampled from patient records.
β’ QASPER ( Dasigi et al., 2021 ) : This benchmark is an information-seeking QA dataset centered on full-length NLP research papers. In particular, it contains around 5K QA pairs grounded in around 1.6L NLP research papers. Also, we use the full text of each paper as the context for the model.
β’ MK-NIAH ( Hsieh et al., 2024 ) : We use the task of multiple keys in needle-in-haystack from RULER ( Hsieh et al., 2024 ) . This setup requires models to not only locate but also extract multiple pieces of information distributed throughout a long text.
As for the baseline, we use ICL, which is the same as Hope with 1-level of memory, and also DuoAttention ( Xiao et al., 2025 ) . It is notable that methods such as Cartridges ( Eyuboglu et al., 2025 ) have shown promising performance, even better than DuoAttention and sometimes ICL. Here, however, we exclude their comparison with Hope mainly due to the fact that Hope has higher memory usage and there are fundamental differences in their computational costs (e.g., self-studying, etc.), requiring further experiments with careful and controlled design in the future studies. For the variants of our model, we use different number of memory levels with different frequencies, which are grouped based on the lowest frequency. Note that the lowest-frequency memory corresponds to the most persistent memory of the model and so we expect models with higher lowest frequency to be more adaptive.
The results are reported in Figure 7 . Hope with any number of levels and also with any lowest frequency outperforms both ICL baseline as well as the efficient DuoAttention. Furthermore, comparing the Hope βs variants with themselves, the results suggest that: (1) The more levels of memory can help with in-context learning capability of the model and enhances its long-term memory and so long-context understanding; and (2) The higher the lowest-frequency the lower the performance. As discussed above, the lowest-frequency memory is the most persistent memory of the model and so increasing it means that the model has a weaker but more adaptive long-term memory. Due to the fact that the frequency of update directly affects the efficiency of the model, one might find βLowest Frequency = 2Kβ an optimal setup as it offers significantly more efficient forward pass, while achieves close performance with βLowest Frequency = 512β.
Learning a New Language In-Context. Throughout the paper, we argued that pre-training can be seen as an in-context learning process, where the context is the entire pre-training data. Accordingly, when we use more levels in the neural learning module, or performing in-context learning on different context flows, we expect the model to show better adaptability and continual learning capabilities. To this end, with combining two existing benchmarks of MTOB ( Tanzer et al., 2024 ) and Manchu ( Pei et al., 2025 ) , we design a new continual learning task that the LLM learns two new languages in context and is expected to translate phrases to English. Then for our Continual Translation of a Novel Language (CTNL) task, we consider two setups: (1) Learning and testing the performance of the model on each language separately (in red color). This is a baseline we use to measure the catastrophic forgetting when comparing with the next setting; and (2) The model, first, learns these two languages in a sequential manner (in blue color) and then is asked to translate the phrases to English. We use ICL as the baseline, and to study the importance of multi-level design of Hope , we use different variants of Hope with 1, 2, and 3 additional levels of memory, which we refer to as Hope -1, Hope -2, and Hope -3, respectively.
The results are reported in Table 1 . More specifically, each point is the performance of one model in one of the setups such that the π \bm{x} -axis (resp. π \bm{y} -axis) indicates the model ChRF in translation of Manchu β \rightarrow English (resp. Kalamang β \rightarrow English). In the first setup (in-context translation without continual learning), all Hope βs variants performs better or on-par compared to ICL, supporting the importance of CMS design. In the second setup (i.e., continual translation), however, ICL faces dramatic performance drop and almost rely on its capabilities achieved in its pre-training (i.e., catastrophic forgetting about the knowledge in the context). On the other hand, increasing the memory levels in the Hope shows clear improvement and Hope -3 almost recovers the ICL capability in the first setup, without continual learning. These results further support the importance of CMS design in continual learning and the ability of the model to adapt itself to the new tasks.
### 9.2 Hope : Long Context Understanding
In the previous section, we evaluated the performance of Hope -Attention, when the MLP blocks are adapted to perform in-context learning. In this part, we evaluate the performance of Hope in long context understanding, when it starts learning from scratch. To this end, we use about 50B tokens from a mixture of FineWeb-Edu ( Penedo et al., 2024 ) and long-context documents with a vocabulary size of 32K to train all the models from scratch. All models are optimized using AdamW with tuned learning rate for each model and with the default optimizer configuration in Behrouz et al. (2025c) . We focus on two popular benchmarks of RULER ( Hsieh et al., 2024 ) and BABILong ( Kuratov et al., 2024 ) :
Needle-in-a-Haystack (NIAH) Tasks. In the first part, we focus on the needle-in-a-haystack with different setups of: (1) single needle but different types (i.e., pass-key, number, and uuid), (2) multi-key, (3) multi-query, and (4) multi-value, all follows Hsieh et al. (2024) . As for the baselines, we use RetNet ( Sun et al., 2023 ) and DeltaNet ( Schlag et al., 2021 ) as the representative of the models purely based on Hebbian- and Delta-rule, and experimented with diverse set of modern linear recurrent models and so used the linear models with the best performance: i.e., RWKV-7 ( Peng et al., 2025b ) and Comba ( Hu et al., 2025 ) . As another group of baselines, we also compare with deep memory modules with dot-product and L 2 L_{2} regression objectives: i.e., DLA ( Behrouz et al., 2025a ) , and Titans ( Behrouz et al., 2025c ) .
The results are reported in Table 1 . Comparing with other attention-free models, Hope achieves the best performance across all tasks and levels of difficulties. Particularly, comparing to linear memories, deep memory modules show better performance in longer sequences, mainly due to their higher memory capacity to compress more tokens. Comparing Hope with Titans, the superior performance of Hope , specifically in longer context length supports the importance of both self-referential update as well as the CMS design. Finally, we also evaluate the performance of Hope -Attention and compare it with Transformers to better understand the contribution of CMS design. The results indicate that Hope -Attention achieves a better performance compared to Transformers, supporting the advantage of having CMS in Hope -Attention design.
BABILong. Next, we evaluate the Hope βs performance on BABILong benchmark ( Kuratov et al., 2024 ) and compare it with (1) large models such as GPT4 and GPT4o-mini ( Achiam et al., 2023 ) ; (2) middle-size Llama-8B model ( Dubey et al., 2024 ) with its RAG augmented version; and (3) the state-of-the-art small models in this tasks: i.e., RMT ( Bulatov et al., 2022 ) , ARMT ( Rodkin et al., 2024 ) , and Titans ( Behrouz et al., 2025c ) . We follow the original setup of the benchmark and fine-tune the small models with the same process as Kuratov et al. (2024) .