Learning to Continually Learn via Meta-learning Agentic Memory Designs — part 3/5
Cleaned-text code points [31223, 47007). Bibliography omitted here; source.md includes it.
We propose ALMA, a paradigm that learns to discover new memory designs, enabling agentic systems to continually learn from their past experiences. We evaluate ALMA on a range of sequential decision-making benchmarks and the best-learned designs consistently outperform manually designed baselines, demonstrating the effectiveness of our approach. Furthermore, subsequent experiments show that the learned memory designs exhibit strong scalability, transferability, and cost efficiency. Overall, ALMA provides a principled approach to learn memory design, taking a step toward enabling continual learning in agentic systems.
ALMA represents a step toward continual learning in AI, contributing to the broader goal of developing AI-generating algorithms beyond current manual approaches ( Clune, 2020 ) . However, such AI-generating algorithms introduce unique safety concerns, as components of AI systems are learned rather than designed by humans. The learned components may inadvertently introduce behaviors that deviate from human intentions, ignoring optimization guided by a predefined target metric ( Bostrom, 2017 ; Rokon et al., 2020 ; Ecoffet et al., 2020 ) . Recognizing these challenges, we impose explicit constraints and restrictions in our experimental setup during the learning process. Specifically, each memory design is generated as code by the Meta Agent, they are validated and executed within isolated sandbox environments, with access confined to the sandbox to prevent any interference with the external system, mitigating the risk of unintended actions. We also perform human oversight of learned memory designs to ensure that no potentially harmful actions (e.g., prompt injection) are included in memory designs. In future work, it will be essential to establish a systematic inspection mechanism as the system scales and is deployed, which may include AI and human inspection.
While ALMA demonstrates strong effectiveness in learning memory designs, it has several limitations that open up future research directions. One limitation is that the memory designs are learned using a pre-defined learning set, instead of dynamically learning memory designs when facing new tasks. Ideally, an adaptive continual learner would be able to learn memory designs online from dynamic data, without separating learning and testing phases. While ALMA has the potential to support online learning, we do not pursue this approach due to computational budget constraints, as evaluating each explored memory design would require a large number of rollouts. Therefore, a possible future direction is to enable online learning of memory design that continually adapts to arbitrary domains. Another limitation stems from the fact that we demonstrate learning memory designs in code space. While effective, their capabilities may be limited by the underlying FMs. Future work could explore automated designing and training novel FM architectures with native memory support.
Overall, ALMA represents a step forward toward automated, continual-learning AI. While current limitations prevent us from demonstrating a system that learns both memory and the agentic system, our results highlight its potential to surpass manually designed memory, showing a way towards “learning to continually learn”. When developed with the appropriate safety measures, ALMA holds promise for a general-purpose AI capable of learning to continually learn across domains and autonomously adapting to new ones.
## Impact Statement
This paper introduces ALMA, a paradigm that automatically learns novel memory designs, thereby enabling agentic systems to continually learn. By leveraging a Meta Agent with open-ended exploration, ALMA mitigates the intensive manual effort conventionally required to handcrafted memory designs for diverse domains. Furthermore, our findings highlight a path toward agentic systems capable of tailoring their memory designs to any context, fostering the development of continual learning of AI. These insights empower practitioners to: (1) automate the development of domain-specific memory designs for specialized fields such as medicine, finance, and software engineering; and (2) uncover memory paradigms that have yet to be identified through human intuition. Ultimately, this work advances the automation of memory design for agentic systems and serves as a foundation for future research on learning memory designs.
## Acknowledgments
This research was supported by the Vector Institute, the Canada CIFAR AI Chairs program, a grant from Schmidt Futures, an NSERC Discovery Grant, and a generous donation from Rafael Cosman. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute ( https://vectorinstitute.ai/partnerships/current-partners/ ). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors.
## Appendix A Algorithmic Details
### A.1 Search Space
### A.2 Evaluation of Memory design
We first formally define a memory design as a triplet ℳ = ( U , D , R ) \mathcal{M}=(U,D,R) , where U U and R R denote the update and retrieval processes that interface with the agentic system, and D D represents the internal storage structure used for memory retention. To evaluate a memory design on a dataset 𝒢 \mathcal{G} , we split 𝒢 \mathcal{G} evenly into two subsets: a set 𝒢 collection \mathcal{G}_{\text{collection}} for memory collection and a set 𝒢 deployment \mathcal{G}_{\text{deployment}} for evaluation during the Deployment Phase. During the Memory Collection Phase, the agentic system interacts with the environment to finish tasks in 𝒢 collection \mathcal{G}_{\text{collection}} without any memory access, producing interaction logs that are used to perform memory updates and construct a static memory. During the Deployment Phase, the agentic system retrieves relevant contexts from the static memory, which are then integrated into the agentic system to facilitate task execution within 𝒢 deployment \mathcal{G}_{\text{deployment}} . The performance on 𝒢 deployment \mathcal{G}_{\text{deployment}} reflects the ability of a memory design to collect and utilize knowledge.
Furthermore, when evaluated on a dataset with task distribution shift during the Deployment Phase, the agentic system first retrieves existing knowledge to construct the knowledge context for the new task, and subsequently updates the memory with the resulting interaction log.
#### Memory Collection Phase.
𝒢 collection \mathcal{G}_{\text{collection}} is used to collect a set of interaction trajectories 𝒯 collection = { τ collection ( j ) } j = 1 N \mathcal{T}_{\text{collection}}=\{\tau^{(j)}_{\text{collection}}\}_{j=1}^{N} without any memory access. Each trajectory τ collection ( j ) \tau^{(j)}_{\text{collection}} takes the form τ collection ( j ) = ( s 1 ( j ) , a 1 ( j ) , s 2 ( j ) , a 2 ( j ) , … , s n j ( j ) , a n j ( j ) ) , a t ( j ) ∼ π ( ⋅ ∣ s ≤ t ( j ) , a < t ( j ) ) \tau^{(j)}_{\text{collection}}=\big(s^{(j)}_{1},a^{(j)}_{1},s^{(j)}_{2},a^{(j)}_{2},\ldots,s^{(j)}_{n_{j}},a^{(j)}_{n_{j}}\big),\quad a^{(j)}_{t}\sim\pi(\cdot\mid s^{(j)}_{\leq t},a^{(j)}_{<t}) where s s denotes the environment state, and a a denotes the action produced by the fixed agentic policy π \pi . For each trajectory, we further obtain a task-level feedback f collection ( j ) = F ( τ collection ( j ) ) f^{(j)}_{\text{collection}}=F\!\left(\tau^{(j)}_{\text{collection}}\right) where F ( ⋅ ) F(\cdot) denotes the benchmark-specific evaluation function that assigns a scalar score to each trajectory. We then sequentially update the memory database using the collected interaction logs D j = U ( τ collection ( j ) , f collection ( j ) , D j − 1 ) D_{j}=U\!\left(\tau^{(j)}_{\text{collection}},f^{(j)}_{\text{collection}},D_{j-1}\right) where the memory state is updated after each collected trajectory. The final collected memory is denoted as D N D_{N} .
#### Deployment Phase.
During the Deployment Phase, we evaluate the performance of the agentic system, with knowledge retrieved from D N D_{N} , on the deployment set 𝒢 deployment = { g deployment ( i ) } i = 1 K \mathcal{G}_{\text{deployment}}=\{g^{(i)}_{\text{deployment}}\}_{i=1}^{K} .
For each task g deployment ( i ) g^{(i)}_{\text{deployment}} , we first retrieve relevant knowledge from the memory state D N D_{N} e ( i ) = R ( D N , s 1 ( i ) ) e^{(i)}=R(D_{N},s^{(i)}_{1}) where s 1 ( i ) s^{(i)}_{1} denotes the initial state, including observation and task objective of the current task. The agent then interacts with the environment, resulting in a trajectory τ deployment ( i ) \tau^{(i)}_{\text{deployment}} , where each action is sampled according to a t ( i ) ∼ π ( ⋅ ∣ s ≤ t ( i ) , a < t ( i ) , e ( i ) ) a^{(i)}_{t}\sim\pi(\cdot\mid s^{(i)}_{\leq t},a^{(i)}_{<t},e^{(i)}) The feedback is calculated as f deployment ( i ) = F ( τ deployment ( i ) ) f^{(i)}_{\text{deployment}}=F(\tau^{(i)}_{\text{deployment}}) . The overall performance of the memory design ℳ \mathcal{M} is then computed as the average over all deployment tasks f ℳ = 1 K ∑ i = 1 K f deployment ( i ) f_{\mathcal{M}}=\frac{1}{K}\sum_{i=1}^{K}f^{(i)}_{\text{deployment}} When performing Deployment Phase with distribution shift tasks, memory state is further updated sequentially using the collected trajectories τ deployment ( i ) \tau^{(i)}_{\text{deployment}} and feedback signals f deployment ( i ) f^{(i)}_{\text{deployment}} , producing D N + i D_{N+i} . The updated memory state is then used for retrieval when performing the ( i + 1 ) (i+1) -th task.
### A.3 Evaluation Cost
#### End-to-End Memory Cost.
This metric quantifies the cumulative computational overhead incurred by the FMs to generate the content integrated into the agentic system with static mode in Deployment Phase. The cost includes: (1) the expense during the Memory Collection Phase to produce the memory state D N D_{N} ; and (2) the cumulative cost of generating knowledge { e ( i ) } i = 1 K \{e^{(i)}\}_{i=1}^{K} for all tasks during the Deployment Phase. We suggest that this provides a fair and unbiased way to measure the cost of a memory design, independent of its effectiveness. During the Memory Collection Phase, the agentic system interacts with environments without memory access, ensuring that all trajectories are unaffected by the memory design’s performance. Since these trajectories are the only input used to produce memory, the costs of producing the memory state D N D_{N} and generating relevant knowledge during static deployment depend solely on these trajectories and are therefore independent of the memory design’s effectiveness. We present a summary of the costs associated with testing a memory design across all benchmarks, including GPT-4o-mini and text-embedding-3-small .
#### Token Size of Retrieved Knowledge.
We measure the token count of the retrieved context e ( i ) e^{(i)} for each task during the static Deployment Phase. This metric quantifies the informational overhead injected into the agent’s prompt, serving as a key proxy for measuring the inference-time efficiency of the memory design ℳ \mathcal{M} . The token size is also independent of effectiveness of a memory design, since it is dependent solely on trajectories collected in Memory Collection Phase. We utilize the GPT-5-mini tokenizer to calculate the token sizes.
### A.4 Memory Design Archive and Sampling Process
#### Memory Design Archive.
To iteratively explore new memory designs, we maintain a memory design archive 𝒜 l \mathcal{A}_{l} that stores all designs discovered up to learning step l l . For each candidate design ℳ \mathcal{M} , we perform a stratified sampling process ( Thompson, 2012 ) after its evaluation on a large deployment set. Specifically, instead of storing the entire interaction logs, we extract a small subset of size K f K_{f} as a proxy for the design’s performance. This subset is denoted as 𝒮 ℳ = { ( e ( i ) , τ deployment ( i ) , f deployment ( i ) ) } i = 1 K f \mathcal{S}_{\mathcal{M}}=\{(e^{(i)},\tau^{(i)}_{\text{deployment}},f_{\text{deployment}}^{(i)})\}_{i=1}^{K_{f}} where each tuple contains the retrieved context e ( i ) e^{(i)} , the resulting trajectory τ deployment ( i ) \tau^{(i)}_{\text{deployment}} , and the task-level feedback f deployment ( i ) f_{\text{deployment}}^{(i)} .
The archive further records the overall performance f ℳ f_{\mathcal{M}} and the times a memory design is being sampled t ℳ t_{\mathcal{M}} , which tracks the number of times ℳ \mathcal{M} has been sampled and evaluated across l l learning steps. Formally, at step l l , the memory design archive is defined as the collection of these records 𝒜 l = { ( ℳ , f ℳ , 𝒮 ℳ , t ℳ ) | ℳ discovered before step l } . \mathcal{A}_{l}=\Big\{(\mathcal{M},f_{\mathcal{M}},\mathcal{S}_{\mathcal{M}},t_{\mathcal{M}})\;\Big|\;\mathcal{M}\text{ discovered before step }l\Big\}.
#### Sampling Process.
Similar to prior exploration approaches ( Ecoffet et al., 2021 ; Zhang et al., 2025e ) , given the current memory design archive, we perform a sampling process to sample memory designs that will be used in the next learning step. For each memory design ℳ i \mathcal{M}_{i} , we first compute the normalized performance relative to a baseline f 0 f_{0} f ^ ℳ i = σ ( f ℳ i − f 0 ) = 1 1 + exp ( − λ ( f ℳ i − f 0 ) ) \hat{f}_{\mathcal{M}_{i}}=\sigma(f_{\mathcal{M}_{i}}-f_{0})=\frac{1}{1+\exp(-\lambda(f_{\mathcal{M}_{i}}-f_{0}))} where f 0 f_{0} denotes the performance of the agentic system on 𝒢 deployment \mathcal{G}_{\text{deployment}} without memory access. Subtracting f 0 f_{0} ensures that the normalized score reflects the performance gain provided by the memory design. The final sampling score of each memory design is then given by J i ≡ J ℳ i = f ^ ℳ i − α log ( 1 + t ℳ i ) J_{i}\equiv J_{\mathcal{M}_{i}}=\hat{f}_{\mathcal{M}_{i}}-\alpha\log(1+t_{\mathcal{M}_{i}}) where t ℳ i t_{\mathcal{M}_{i}} is the number of times ℳ i \mathcal{M}_{i} has been sampled, and α \alpha controls the strength of the penalty. We then compute the sampling probability using a softmax over the final sampling scores p i = exp ( J i / T − max j J j / T ) ∑ j = 1 | 𝒜 l | exp ( J j / T − max j J j / T ) , i = 1 , … , | 𝒜 l | p_{i}=\frac{\exp\big(J_{i}/T-\max_{j}J_{j}/T\big)}{\sum_{j=1}^{|\mathcal{A}_{l}|}\exp\big(J_{j}/T-\max_{j}J_{j}/T\big)},\quad i=1,\dots,|\mathcal{A}_{l}| where T T is a temperature parameter controlling the smoothness of the distribution. The sampling mechanism prioritizes high-performing yet under-sampled memory designs to balance exploration and exploitation. Meanwhile, designs with moderate performances retain a non-zero sampling probability to maintain diversity throughout the learning process. Finally, we sample memory designs according to { p i } \{p_{i}\} : ℳ sampled ⊂ { ℳ 1 , … , ℳ | 𝒜 l | } , ℳ sampled ∼ Categorical ( p 1 , … , p | 𝒜 l | ) \mathcal{M}_{\text{sampled}}\subset\{\mathcal{M}_{1},\dots,\mathcal{M}_{|\mathcal{A}_{l}|}\},\quad\mathcal{M}_{\text{sampled}}\sim\text{Categorical}(p_{1},\dots,p_{|\mathcal{A}_{l}|})
### A.5 Prompts of Meta Agent
### A.6 Pseudocode
## Appendix B Experiment Details
### B.1 Benchmark Details
We evaluate ALMA on four sequential decision-making benchmarks: ALFWorld ( Shridhar et al., 2021 ) , TextWorld ( Côté et al., 2019 ) , Baba Is AI ( Cloos et al., 2025 ) , and MiniHack ( Samvelyan et al., 2021 ) . All benchmarks involve multi-step interactions, providing a suitable testbed for evaluating how agents use memory designs to learn from experience.