Nested Learning: The Illusion of Deep Learning Architectures

arxiv:2512.24695

Original source · Paper JSON

Data, training, optimization · 2025-12-31

Cites (11) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12
13. Part 13
14. Part 14

Contents

• Nested Learning: The Illusion of Deep Learning Architecture
• Abstract
• 1 Introduction
• 1.1 Human Brain Perspective and Neurophysiological Motivation
• 1.2 Contributions and Roadmap
• 2 Preliminaries
• 3 Nested Learning
• 3.1 Associative Memory
• Definition 1 (Associative Memory) .
• 3.2 Nested Optimization Processes
• Definition 2 (Update Frequency) .
• Definition 3 (Nested System) .
• Definition 4 (Nested System of Associative Memories) .
• 3.3 Knowledge Transfer Between Levels
• 4 Optimizers as Learning Modules
• 4.1 Backpropagation as an Associative Memory
• 4.2 Momentum-based Optimizers as Associative Memories
• 4.3 Long Context in Optimizers: An Example of Continual Learning with Orthogonal Tasks
• 4.4 More Expressive Designs for Momentum as an Associative Memory
• 4.5 Going Beyond Simple Gradient Descent and Momentum
• Definition 5 (Generalized Gradient Descent (GGD) Learning Rule) .
• 5 Existing Architectures as Neural Learning Modules
• 5.1 Revisiting the Human Brain Perspective of Nested Learning
• 6 Takeaways and Revisiting Common Terms
• 7 Continuum Multi-Timescale Memory System
• 7.1 Continuum Memory System (CMS)
• 7.2 Continuum Memory System In Optimizers
• 7.3 Ad-hoc Level Stacking: Initializing CMS with Pre-Trained Models
• 8 Hope : A Self-Referential Learning Module with Continuum Memory
• 8.1 Deep Self-Referential Titans
• 8.2 Fast and Parallelizable Training
• 8.3 Hope Neural Learning Module
• 9 Experiments
• 9.1 Hope : Continual Learning and Long Context Understanding
• 9.2 Hope : Long Context Understanding
• 9.3 Hope : Language Modeling and Common-Sense Reasoning
• 9.4 Hope : In-context Recall Tasks and MAD Synthetic Benchmark
• 9.5 Language Recognition Tasks
• 9.6 Hope : Ablation Studies and Scaling
• 9.7 Expressive Optimizers
• 10 Conclusion
• Appendix A Generalized Formulation for Nested Learning and Nested Systems
• Definition 6 ((Generalized) Nested System) .
• Definition 7 ((Generalized) Nested System of Associative Memories) .
• Appendix B Adam, AdaGrad, and Other Similar Optimizers as Associative Memory Modules
• Appendix C Delta Gradient Descent with Normalization
• Instructions for reporting errors