Learning to Continually Learn via Meta-learning Agentic Memory Designs — part 2/5

Cleaned-text code points [15845, 31223). Bibliography omitted here; source.md includes it.

Deployment Phase. This is the main stage where the agentic system is deployed to applications and expected to achieve task success. Given the memory collected after the Memory Collection Phase, the agentic system retrieves from memory for each incoming task and attempts to solve it. This stage can operate in two modes: dynamic , which updates memory with newly collected trajectories from incoming tasks ( Zhang et al., 2025a ; Ouyang et al., 2025 ) , or static , which keeps memory fixed ( Tang et al., 2025 ; Rasmussen et al., 2025 ) .

We evaluate each memory design by first running the Memory Collection Phase and then recording the success rate during the Deployment Phase, using an identical, fixed agentic system, with the memory design being evaluated. The two modes in the Deployment Phase serve different purposes: static mode assesses how effectively the agent leverages a fixed memory to solve new tasks, while dynamic mode measures how well a memory design adapts to a new task distribution through dynamic updates and retrieval. When learning memory designs, we use only static mode to reduce variance during evaluations. The general_update and general_retrieve modules contribute to the performance of the memory design through memory collection and knowledge retrieval, respectively. When testing the best-learned memory design, we evaluate under both modes. More details on evaluations are available in Appendix A.2 .

### 3.3 Open-Ended Exploration

Similar to ADAS ( Hu et al., 2025 ) , we propose a framework for open-ended exploration of memory designs with a Meta Agent programming novel memory designs in code. The algorithm proceeds as follows: (1) An archive is initialized with Python abstract classes as an empty memory design template. (2) The Meta Agent samples previously discovered designs from the archive, reflects on their performance outcomes, and proposes new memory designs. (3) The newly proposed designs are evaluated on the target domain. If errors occur during evaluation, the Meta Agent performs self-reflection to refine the design, repeating this process up to three times if necessary. (4) Finally, the memory design is added to the archive along with its evaluation results and logs, and the process continues with the updated archive until the maximum number of iterations is reached.

Sampling Memory Designs. We sample memory designs from the archive to serve as stepping stones for the exploration of new memory designs at each step. Similar to DGM ( Zhang et al., 2025e ) , each design in the archive is assigned a sampling probability roughly proportional to its success rate and inversely proportional to the number of times it is sampled. This prioritizes designs that perform well but have been sampled less frequently, enabling a balance between refining successful designs and exploring underexplored candidates. All designs maintain non-zero sampling probabilities, keeping all potential improvements reachable. Details are provided in Appendix A.4 .

Proposing New Memory Design by the Meta Agent. After sampling previous memory designs from the archive, the Meta Agent reflects on their evaluation results and proposes new memory designs by implementing them in code. Specifically, the Meta Agent first proposes ideas, then plans for the new designs through analyzing the sampled designs along with their success rate and interaction logs. Based on the plans, the Meta Agent implements the new memory designs in code and performs trial runs. Runtime errors during the trial runs trigger debugging reflections to refine the implementations. Finally, the new memory designs are evaluated. Their evaluation logs are added to the memory design archive to support future sampling. Prompts for planning, implementation, and debugging are in Appendix A.5 .

## 4 Experiments

### 4.1 Experiment Setup

We perform ALMA on four sequential decision-making benchmarks (Section 4.2 ). For each benchmark, the dataset is divided into a learning set and a testing set, which are used for memory design learning and for the testing of the learned memory designs to ensure testing on unseen data. Both sets are further split evenly, with the first half used for the Memory Collection Phase and the second half for the Deployment Phase ( Section 3.2 ). To reduce variance, we execute the Deployment Phase three times for learning and testing, and report the average success rate with the standard error. For clarity, in the following sections, “success rate” refers to the average success rate over three runs.

At each learning step, we sample up to five memory designs from the archive without replacement. We expect the choice between sampling with or without replacement to have minimal impact (Appendix B.3 ). We run ALMA for 11 learning steps to discover 43 memory designs.

During the learning of memory designs, GPT-5 is used in the Meta Agent, while GPT-5-nano is used in the fixed agentic system for evaluating memory designs. While the Meta Agent can choose to use any model (or train models) as FMs within memory designs, we provide GPT-4o-mini , GPT-4.1 , and text-embedding-3-small as tools for the Meta Agent to explore workflows such as extracting insights or computing semantic similarity. More details on the learning process setup are available in Appendix B.3 .

During the testing of the learned memory designs, we test under two agentic system settings: the first employs GPT-5-nano , matching the learning setup. The second replaces it with a more capable FM, GPT-5-mini , to assess the transferability of memory designs to stronger agentic systems ( Table 1 ). We use GPT-4o-mini and text-embedding-3-small as FMs across all memory designs during testing to ensure a fair comparison. More details on the testing setup are available in Appendix B.4 .

### 4.2 Benchmarks

We adopt four sequential decision-making benchmarks with varying levels of difficulty ( Paglieri et al., 2025 ) . In all benchmarks, agentic systems perceive the environment through text and interact with it by generating actions in natural language. The benchmarks are well-suited for evaluating memory designs because they require acquiring and utilizing experience that does not exist in the pre-trained knowledge of FMs ( Paglieri et al., 2025 ) . We test ALMA on four benchmarks: (1) ALFWorld ( Shridhar et al., 2021 ) , a text-based simulation of embodied household tasks that challenges agents to ground language instructions to action sequences in kitchen environments; (2) TextWorld ( Côté et al., 2019 ) , text adventure games that require systematic exploration and reasoning to navigate partially observable environments; (3) Baba Is AI ( Cloos et al., 2025 ) , a strategic puzzle game where agents must manipulate the rules of the game, requiring complex reasoning and adaptation to changing game mechanics; (4) MiniHack ( Samvelyan et al., 2021 ) , a simplified version of NetHack ( Küttler et al., 2020 ) that challenges agents with long-horizon decision-making in procedurally generated dungeon environments.

For ALFWorld, we are using the default setting ( Shridhar et al., 2021 ) . For all other benchmarks, we use BALROG ( Paglieri et al., 2025 ) , following its settings and prompt for interactions during learning and final testing. More details on benchmarks are available in Appendix B.1 .

### 4.3 Baselines

We select four representative state-of-the-art human-designed memory systems as baselines. These baselines are entirely handcrafted, with all aspects of the design (e.g., what to store, how to update, and how to retrieve) manually specified and inserted retrieved knowledge into the context of agentic systems, representing various mainstream approaches in prior work. More details are available in Appendix B.2 .

Trajectory Retrieval. A straightforward but strong baseline that retrieves similar trajectories ( Park et al., 2023 ; Xu et al., 2024 ) . Task descriptions and trajectories are stored during memory update. During memory retrieval, the most relevant past trajectory is retrieved as knowledge based on the similarity of the task description embedding.

ReasoningBank. The memory designs ( Ouyang et al., 2025 ) organize and extract experience on a per-task basis, enabling the retrieval of experience relevant to each task. During memory update, experiences are extracted from each trajectory by FMs and stored together with the corresponding task description. The experiences corresponding to the most similar task description are retrieved as knowledge.

Dynamic Cheatsheet (Cumulative). A global semantic memory incrementally accumulates experience by FMs across all past trajectories ( Suzgun et al., 2025 ) . The trajectories are sequentially used to update a global cheatsheet, which works as knowledge across all tasks.

G-memory. A hierarchical graph-based memory design ( Zhang et al., 2025a ) . During the update of memory, it builds a hierarchical memory where insights and key steps in trajectories are connected in a graph based on task descriptions. When facing new tasks, it traverses the graph to retrieve relevant insights and key steps as knowledge.

### 4.4 Results

For each benchmark, we select the memory design achieving the highest success rate during the learning of memory designs as the best-learned memory design. We then evaluate both human-designed baselines and the best-learned memory designs using the testing process described in Section 4.1 with static mode in the Deployment Phase ( Section 3.2 ).

As shown in Table 1 Top, with the agentic system powered by GPT-5-nano , the learned memory designs achieve an overall improvement of 6.2% compared to the no-memory baseline, and outperform all state-of-the-art human-designed memory baselines. These improvements are consistent across benchmarks, demonstrating that learned memory designs can more effectively store and reuse past experience to continually learn than manual designs.

We further evaluate learned memory designs by changing the FM in the agentic system from GPT-5-nano to GPT-5-mini ( Table 1 Bottom). The learned memory designs improve the overall average success rate by 12.8% over the no-memory baseline and outperform all human-designed baselines. These results demonstrate that the discovered memory designs generalize across different FMs, indicating that the improvements are robust and not tied to a specific model. The larger improvement observed with more capable FMs (12.8% vs 6.2%) yields a delta of 6.6%. This delta exceeds those of all human-designed memory baselines, suggesting that our learned memory designs provide stronger support to more capable agentic systems.

To provide a closer look at the learning process, we show the learning process on Baba Is AI and the tree visualization of the resulting memory design archive ( Figure 2 ). The Meta Agent explores diverse new designs on the path toward the best-learned memory design, building upon designs with moderate success rates that nevertheless have the potential to evolve into the optimal memory design. As shown in Figure 2 , during open-ended exploration, the Meta Agent incrementally introduces mechanisms like property validation and spatial object normalization. While these mechanisms may not yield immediate performance gains, they serve as stepping stones that contribute to the best-learned memory design when key mechanisms (e.g., strategy switching) are introduced. Analyses and visualizations of the learning processes for the other benchmarks are available in the Appendix C.1 .

To evaluate the effectiveness of the open-ended exploration in ALMA, we conduct an ablation study comparing against greedy search, which always samples the design with the highest success rate to propose new designs. More details on greedy search are available in Appendix B.3 . We perform the learning process with greedy search on ALFWorld, resulting in success rates of 11.9% and 77.1% when testing the best-learned memory design with the agentic system powered by GPT-5-nano and GPT-5-mini , respectively (Appendix C.2 ). In both cases (for GPT-5-nano , 11.9% vs 12.4%; for GPT-5-mini , 77.1% vs 87.1%), success rates are lower than those achieved by designs learned with open-ended exploration, demonstrating the benefits of an exploration-driven learning process over greedy search.

We visualize the learned memory designs for each domain ( Figure 3 ), showing that ALMA discovers effective memory structures adapted to diverse task requirements. For games with explicit object interaction goals (e.g., ALFWorld and TextWorld), the learned memory designs tend to store fine-grained knowledge, such as spatial relationships between objects and room layouts. In contrast, for tasks that require more complex reasoning (e.g., Baba Is AI and MiniHack), the memory designs favor domain-specific abstract strategy, including strategy libraries and plan synthesis. This pattern indicates that ALMA automatically specializes memory designs to the demands of each domain.

To study how performance scales with experience, we evaluate memory designs using different sizes of tasks during the Memory Collection Phase and using static mode during the Deployment Phase. As shown in Figure 4 , compared to manual memory designs, the learned memory design achieves higher performance faster with limited data and scales better as more trajectories are provided, demonstrating both higher sample efficiency and stronger scalability. We observe that increasing the trajectory size does not necessarily lead to a monotonic improvement in success rate, which is expected since trajectories may vary in quality ( Ouyang et al., 2025 ) .

We evaluate the adaptability of learned memory designs to the distribution shift of tasks. To evaluate adaptation under distribution shift, memory is collected from 70 tasks in the valid_seen during the Memory Collection Phase and evaluated on the valid_unseen using dynamic mode during the Deployment Phase (Appendix B.4 ). This tests whether memory designs can adapt to tasks with distribution shift in valid_unseen by dynamically updating memory. As shown in Figure 5 , the learned memory design achieves a success rate of 84.1% on ALFWorld, outperforming all human-designed baselines and demonstrating more effective adaptation under task distribution shift.

We measure the cost efficiency of learned memory designs by studying their end-to-end memory cost, defined as the total cost of FMs required to transform raw interaction logs into retrieved content inserted into the agentic system’s context across the Memory Collection and Deployment Phases. We also include the token size of the retrieved content. The details are available in Appendix A.3 . As shown in Figure 6 , compared to all manual designs, our learned memory designs achieve an average success rate of 53.9%, with lower overall end-to-end memory cost across all benchmarks ($0.09) and appropriate token sizes of retrieved content. The results show ALMA effectively balances memory efficiency and task performance. Since the current algorithm does not explicitly optimize for cost efficiency, we anticipate that incorporating techniques such as multi-objective optimization in future work could enable even more effective exploration of the efficiency-performance Pareto front.

## 5 Conclusion, Safety, and Future Work