Learning to Continually Learn via Meta-learning Agentic Memory Designs — part 1/5

Cleaned-text code points [0, 15845). Bibliography omitted here; source.md includes it.

# Learning to Continually Learn via Meta-learning Agentic Memory Designs

###### Abstract

The statelessness of foundation models bottlenecks agentic systems’ ability to continually learn, a core capability for long-horizon reasoning and adaptation. To address this limitation, agentic systems commonly incorporate memory modules to retain and reuse past experience, aiming for continual learning during test time. However, most existing memory designs are human-crafted and fixed, which limits their ability to adapt to the diversity and non-stationarity of real-world tasks. In this paper, we introduce ALMA ( A utomated meta- L earning of M emory designs for A gentic systems), a framework that meta-learns memory designs to replace hand-engineered memory designs, therefore minimizing human effort and enabling agentic systems to be continual learners across diverse domains. Our approach employs a Meta Agent that searches over memory designs expressed as executable code in an open-ended manner, theoretically allowing the discovery of arbitrary memory designs, including database schemas as well as their retrieval and update mechanisms. Extensive experiments across four sequential decision-making domains demonstrate that the learned memory designs enable more effective and efficient learning from experience than state-of-the-art human-crafted memory designs on all benchmarks. When developed and deployed safely, ALMA represents a step toward self-improving AI systems that learn to be adaptive, continual learners. All code is open-sourced at https://github.com/zksha/alma.git .

###### Keywords:

## 1 Introduction

Agentic systems powered by Foundation Models (FMs) have enabled autonomous decision-making and task execution across a wide range of domains ( Yao et al., 2023 ; Wang et al., 2023 ; Yang et al., 2024 ) . However, the stateless nature of FMs during inference challenges agentic systems to accumulate experience and continually learn from past interactions, causing them to repeatedly solve tasks from scratch and limiting their ability to improve over time ( Child et al., 2019 ; Bulatov et al., 2022 ) . Memory addresses this limitation by enabling agentic systems to continually store and reuse past experiences ( Zhong et al., 2024 ; Packer et al., 2024 ) . This allows accumulated experience to inform decision-making in future tasks, aiming for continual learning over time.

However, memory design, the architectural specification that determines how memories are represented, stored, retrieved, and updated, is still predominantly handcrafted by humans. Diverse domains require distinct memory designs to leverage unique aspects of experience, resulting in human researchers manually tailoring a wide array of designs to specific tasks ( Zhang et al., 2024b ; Hu et al., 2026 ) . For example, in conversational agents, memory may focus on retaining facts about the user, such as preferences and personal information ( Chhikara et al., 2025 ; Rasmussen et al., 2025 ) . In contrast, for strategic games, memory should extract abstract skills and strategies from previous interactions rather than details that may change across episodes ( Tang et al., 2025 ; Ouyang et al., 2025 ) . Therefore, manually identifying an optimal memory design for each domain is both difficult and labor-intensive.

A recurring theme in machine learning history is that handcrafted components in AI systems are eventually replaced by learned, more effective ones ( Clune, 2020 ; Hutter et al., 2019 ; Sutton, 2019 ) . A classic example of this paradigm is the transition from hand-designed features to learned representations in computer vision ( Krizhevsky et al., 2012 ) . More recently, the trend of “learning to learn” has expanded to learning neural architectures ( Zoph and Le, 2017 ) , optimization algorithms ( Andrychowicz et al., 2016 ) , training environments ( Wang et al., 2019 ; Zhang et al., 2024a ; Faldor et al., 2025 ) , and designs for agentic systems ( Hu et al., 2025 ; Zhang et al., 2025e ) , among others. In this paper, we investigate whether agentic systems can learn to continually learn by automating the design of their memory components. This enables memory designs to be specialized for diverse domains without relying on manual engineering.

We propose ALMA ( A utomated meta- L earning of M emory designs for A gentic systems), a framework that adopts a Meta Agent to explore novel memory designs for agentic systems through open-ended exploration. We use code as the search space, giving ALMA the theoretical potential to build any memory design. As shown in Figure 1 , the Meta Agent samples previously explored memory designs from an archive, where all explored memory designs and their evaluation logs are stored. Then the Meta Agent reflects on the sampled memory designs to generate new ideas and plans, which are implemented in code. The new designs are subsequently validated and evaluated, with the resulting logs added back to the archive to guide future sampling.

We evaluate our method across four sequential decision-making domains, including ALFWorld ( Shridhar et al., 2021 ) , TextWorld ( Côté et al., 2019 ) , Baba Is AI ( Cloos et al., 2025 ) , and MiniHack ( Samvelyan et al., 2021 ) . These domains serve as testbeds for evaluating an agent’s capability to continually learn from experience. In these domains, agents benefit from storing and reusing their experiences to acquire effective strategies and useful information that is absent from the FMs’ pre-trained knowledge. The results show that ALMA can discover memory designs that are tailored for the needs of different domains, consistently surpassing state-of-the-art human-designed memory baselines ( Table 1 ). Beyond superior performance, our learned memory designs are more cost-efficient than most human-designed memory baselines ( Figure 6 ). Furthermore, the learned memory designs scale performance more effectively with memory size and learn faster when facing task distribution shifts ( Figures 4 and 5 ). Overall, the memory designs learned by ALMA demonstrate a superior ability to help agentic systems continually learn, paving the way towards developing lifelong learning agents in dynamic environments.

## 2 Related Work

#### Memory for Agentic Systems.

Memory enables agentic systems to continually learn despite the statelessness of FMs ( Hu et al., 2026 ; Zhang et al., 2025i ) . Existing work has categorized memory mechanisms into token-level memory, parametric memory, and latent memory ( Hu et al., 2026 ) . Token-level memory defines databases and update rules to store information and extract insights from interaction trajectories between agents and environments. When addressing a new task, the system retrieves relevant experiences via defined procedures, which are then appended to the prompts of agentic systems as context ( Chhikara et al., 2025 ; Rasmussen et al., 2025 ; Nan et al., 2025 ; Zhang et al., 2025a ; Ouyang et al., 2025 ; Suzgun et al., 2025 ; Zhao et al., 2024 ) . For example, G-Memory ( Zhang et al., 2025a ) extracts experiences and stores them in a graph-based database, and retrieves relevant experiences for new tasks via graph traversal. Reasoning Bank ( Ouyang et al., 2025 ) performs task-wise experience generation during memory updates, while Dynamic Cheatsheet ( Suzgun et al., 2025 ) includes accumulation of insights across trajectories. Current works also explore training models to select from predefined operations for storing and reusing experience in memory ( Zhou et al., 2025 ; Yan et al., 2026 ; Liang et al., 2026 ; Wang et al., 2025 ) . Other categories of memory include parametric and latent memory, which encode experience implicitly within the model’s weights ( Qin et al., 2024 ; Zhang et al., 2025g ) or latent hidden states ( Wang et al., 2024a ; Zhang et al., 2025b ; Zou et al., 2025 ) rather than relying on explicit text-based retrieval. The “Learning to learn” paradigm can theoretically support the learning of all categories of memory. In this paper, we focus on token-level memory, as it allows for faster evaluation without additional model training and provides more interpretable, transferable memory representations.

#### AI-generating algorithms and the “Learning to learn” paradigm.

AI-generating algorithms ( Clune, 2020 ) and automated machine learning ( Hutter et al., 2019 ) aim to replace hand-engineered components with automatically learned ones. The approach has three key pillars: (1) meta-learning architectures, (2) meta-learning learning algorithms, and (3) the automatic generation of learning environments ( Clune, 2020 ) . Neural Architecture Search ( Elsken et al., 2019 ; Lu et al., 2019 ; Hu et al., 2023 ) primarily contributes to the first pillar by automating the discovery of neural architectures. Research in MAML ( Finn et al., 2017 ) , Meta-RL ( Wang et al., 2017 ; Duan et al., 2016 ; Norman and Clune, 2024 ) , and recent Automated Algorithm Design ( Liu et al., 2024 ; Liu et al., 2026 ) has explored the second pillar by focusing on “Learning to learn”, developing algorithms that enable agents to adapt their learning behavior across tasks ( Beaulieu et al., 2020 ; Lu et al., 2024a ) . The third pillar includes approaches such as POET ( Wang et al., 2019 ; Wang et al., 2020 ) , as well as more recent work on dynamic environment generation with agentic systems ( Aki et al., 2024 ; Guo et al., 2025 ; Zhang et al., 2024a ) . Our work is more related to the second pillar, as ALMA learns how memory designs can better enable agentic systems to continually learn. Furthermore, recent AI-generating algorithms and automated machine learning approaches have leveraged FMs to explore new components in AI systems through code generation ( Hu et al., 2025 ; Zhang et al., 2025e ; Faldor et al., 2025 ; Yamada et al., 2025 ; Lu et al., 2024b ; Lu et al., 2024a ; Lu et al., 2025a ; Lehman et al., 2024 ) . Similarly, we adopt a Meta Agent powered by FMs to propose novel memory designs by programming in code.

#### Automated Design of Agentic Systems.

Previous work has studied the automated design of agentic systems to learn better components in agentic systems ( Hu et al., 2025 ; Zhang et al., 2025f ; Zhou et al., 2024 ; Yin et al., 2025 ; Zhuge et al., 2024 ; Rosser and Foerster, 2025 ; Zhang et al., 2025c ; Shang et al., 2025 ; Zhang et al., 2025h ; Zhang et al., 2025c ; Ye et al., 2025 ; Zhang et al., 2025e ) . Since recent works such as ADAS ( Hu et al., 2025 ) employ code as the search space for agentic system designs, they theoretically enable the learning of the memory component. For example, AgentSquare ( Shang et al., 2025 ) attempts to co-learn memory designs along with other components like tools in agentic systems. However, while existing works evaluate only the one-shot performance of agentic systems, ALMA explicitly optimizes for a memory design’s capability to facilitate continual learning from past experience, enabling the discovery of more effective memory designs. A concurrent work to ALMA also explores an approach for learning memory designs ( Zhang et al., 2025d ) . However, it relies on the initialization with many existing handcrafted memory designs and greedy selection of top-performing designs, which limits open-ended exploration. In contrast, ALMA adopts an open-ended exploration of memory designs and learns from scratch. Open-ended exploration is shown to be important to search for high-performance structure ( Lehman and Stanley, 2008 ; Lehman and Stanley, 2011a ; Lehman and Stanley, 2011b ; Conti et al., 2018 ; Stanley, 2019 ) . Therefore, incorporating open-ended exploration into the search for optimal memory designs presents a promising avenue. Our ablation studies show that indeed open-ended exploration can learn better memory designs than greedy-selection-based optimization (Appendix C.2 ).

## 3 Learning of Memory Designs

We propose ALMA, a framework for learning memory designs within a code-based search space via open-ended exploration, discovering memory designs that effectively collect, summarize, and reuse experience. Specifically, a Meta Agent ( Hu et al., 2025 ) iteratively proposes a new candidate memory design by reflecting on the results of previously discovered designs, implements the new design through code generation and debugging, and evaluates its performance by integrating it into an agentic system. The pseudocode of our learning process is provided in Appendix A.6 .

### 3.1 Search Space for Memory Designs

The search space defines all memory designs that can be represented and discovered. In this paper, we use code as the search space, thus theoretically allowing all possible memory designs to be discovered due to the Turing completeness of many programming languages, i.e., Python in our case. Representing memory designs in code also enables interpretability and allows FMs to leverage their prior knowledge acquired during pretraining about agentic systems and coding ( Hu et al., 2025 ) . Due to the vastness of the code-based search space, it is inefficient for the Meta Agent to construct all basic functions of a memory design from scratch. We therefore provide the Meta Agent a simple abstraction that facilitates exploration while preserving flexibility. Our abstraction design is inspired by common patterns observed in existing handcrafted memory designs ( Nan et al., 2025 ; Chhikara et al., 2025 ; Ouyang et al., 2025 ; Zhang et al., 2025a ; Tang et al., 2025 ; Suzgun et al., 2025 ) . The abstraction allows the memory module to interact with the agentic system through two primary interfaces: general_update() and general_retrieve() . After collecting new interactions with the environment, the agent calls general_update() to extract useful experience into memory. When faced with a new task, the agent calls general_retrieve() to access relevant experience. Internally, each interface can coordinate multiple sub-modules. Each sub-module implements its update() and retrieve() logic, and optionally maintains its own database when needed. The output of one sub-module can serve as input for subsequent sub-modules, enabling hierarchical and modular memory designs. For example, ReasoningBank ( Ouyang et al., 2025 ) employs a vector-database sub-module to store task descriptions, along with an additional sub-module that extracts experience learned from individual tasks with FMs. Similarly, G-memory ( Zhang et al., 2025a ) incorporates a graph-database sub-module to model linkages between extracted insights and a vector-database sub-module to store task descriptions. We implement the described abstraction in Python Abstract Classes, and the code is available in Appendix A.1 .

### 3.2 Evaluation of Memory Designs

Based on existing work ( Nan et al., 2025 ; Chhikara et al., 2025 ; Ouyang et al., 2025 ; Zhang et al., 2025a ; Tang et al., 2025 ; Suzgun et al., 2025 ; Hu et al., 2026 ; Wang et al., 2024b ; Zhong et al., 2024 ; Packer et al., 2024 ; Fang et al., 2026 ; Zhao et al., 2024 ) , we summarize that memory components in agentic systems generally operate in two phases.

Memory Collection Phase. The goal of this phase is not task success, but rather collecting knowledge and updating memory for later use in the Deployment Phase. There are two common ways to obtain raw trajectories for updating memory at this stage: (1) leveraging existing agent trajectories from available datasets ( Zhong et al., 2024 ; Chhikara et al., 2025 ) , or (2) starting from an empty memory state and collecting trajectories by running the agent on tasks ( Fang et al., 2026 ; Tang et al., 2025 ) . This phase typically does not involve retrieving from memory ( Zhong et al., 2024 ; Chhikara et al., 2025 ; Wang et al., 2024b ; Packer et al., 2024 ; Rasmussen et al., 2025 ) .