Learning to Continually Learn via Meta-learning Agentic Memory Designs — part 4/5

Cleaned-text code points [47007, 62661). Bibliography omitted here; source.md includes it.

ALFWorld. ALFWorld is a text-based embodied environment derived from ALFRED ( Shridhar et al., 2020 ) , where agents interact with household environments through natural language instructions ( Shridhar et al., 2021 ) . Tasks in ALFWorld require agents to perform multi-step manipulations, such as picking, placing, opening, and toggling objects, often with dependencies between sub-tasks. We follow the standard ALFWorld benchmark configurations, as shown in https://github.com/alfworld/alfworld/blob/master/configs/base_config.yaml . It contains a train dataset with 3553 tasks, a valid_seen with 140 tasks, and a valid_unseen dataset with 134 tasks. The valid_seen dataset only contains objects and rules that appear in the train dataset, while the valid_unseen dataset contrains objects and rules that do not appear in train dataset. Each episode terminates either when the task is successfully completed or when a maximum step limit is reached, providing a binary reward of 1 for success and 0 for failure. In our experiment, we set a maximum step limit of 100. An example interaction is shown in Figure 7 .

TextWorld. TextWorld is a text-based interactive environment designed to evaluate sequential decision-making agents through natural language interaction ( Côté et al., 2019 ) . In TextWorld, the agentic system navigates and manipulates a partially observable world using textual commands, such as moving between rooms, examining objects, and performing object-centric actions. We evaluate our method on the Treasure Hunter and Cooking games, using a total of 52 tasks across both games. Each task requires the agentic system to complete a goal specified in natural language, often involving exploration, information gathering, and multi-step reasoning. We perform evaluation using the BALROG benchmark ( Paglieri et al., 2025 ) and follow its standard configuration and prompting scheme from Lu et al. (2025b) ( https://github.com/balrog-ai/BALROG/blob/main/balrog/config/config.yaml ). An episode terminates when the goal condition is satisfied or when the maximum step limit is reached, with a score ranging from 0 to 1 that measures the degree of task completion. In our experiments, we set the maximum number of interaction steps to 80. An example interaction is shown in Figure 8 .

Baba Is AI. Baba Is AI is a grid-based puzzle environment designed to evaluate sequential decision-making agents in worlds governed by explicitly manipulable symbolic rules ( Cloos et al., 2025 ) . In Baba Is AI, an agentic system operates in a discrete grid world and interacts with objects by moving and pushing word tiles that define the rules of the environment. The transition rules and goal conditions in Baba Is AI are specified as compositional natural language-like statements (e.g., BABA IS YOU , FLAG IS WIN ) that can be modified through interaction. Each task requires the agentic system to explore the grid, reason about the current rule set, and strategically alter rules to achieve the goal, often involving long-horizon planning and non-local dependencies. We evaluate on Baba Is AI using BALROG ( Paglieri et al., 2025 ) , following its standard Baba Is AI benchmark configuration and prompting scheme ( https://github.com/balrog-ai/BALROG/blob/main/balrog/config/config.yaml ), using a total of 52 tasks. An episode terminates when the win condition is satisfied under the current rule configuration or when the maximum step limit is reached, yielding a binary reward of 1 for success and 0 otherwise. In our experiments, we set the maximum number of interaction steps to 20. An example interaction is shown in Figure 9 .

MiniHack. MiniHack is a grid-based sandbox environment built on top of the NetHack Learning Environment (NLE) ( Samvelyan et al., 2021 ; Küttler et al., 2020 ) , designed to evaluate sequential decision-making agents in procedurally generated dungeon worlds. We utilize Navigation games, which require exploration, planning, and strategic resource management. We evaluate on MiniHack using BALROG ( Paglieri et al., 2025 ) , following its standard MiniHack benchmark configuration and prompting scheme ( https://github.com/balrog-ai/BALROG/blob/main/balrog/config/config.yaml ), using a total of 114 tasks. An episode terminates when the goal condition is satisfied or when the maximum step limit is reached, yielding a binary reward of 1 for success and 0 otherwise. In our experiments, we set the maximum number of interaction steps to 100. An example interaction is shown in Figure 10 .

### B.2 Baseline Details

Trajectory Retrieval. Trajectory Retrieval is a straightforward but strong baseline ( Park et al., 2023 ; Xu et al., 2024 ) . Given a task description, we encode it into a vector representation and retrieve the most similar past trajectory from the trajectory database based on cosine similarity. Due to context length constraints, only the Top-1 retrieved raw trajectory is provided as memory to the agentic system.

Dynamic Cheatsheet. Dynamic Cheatsheet (DC) maintains an external memory that tracks the successes and failures of the agentic system ( Suzgun et al., 2025 ) . We adopt the cumulative Dynamic Cheatsheet (DC-Cu), which maintains a single global cheatsheet throughout the evaluation of a memory design. We follow the prompts and workflows in https://github.com/suzgunmirac/dynamic-cheatsheet . During the Memory Collection Phase, trajectories are updated sequentially to incrementally update the global cheatsheet. In the Deployment Phase, the fixed global cheatsheet is provided to the agentic system to evaluate the effectiveness of a shared global memory across tasks.

ReasoningBank. ReasoningBank maintains a structured memory schema in which all memory items are stored ( Ouyang et al., 2025 ) . Each memory item consists of three components: (1) a title, which serves as an identifier summarizing the core strategy; (2) a description, providing a one-sentence summary of the memory item; and (3) the content, which records concise reasoning steps and insights. During the Memory Collection Phase, memory items are constructed for each trajectory by extracting either pitfalls or successful experiences, depending on the outcome of the trajectory. We follow the workflows and prompting strategies described in Ouyang et al. (2025) , while using the ground-truth feedback of each trajectory. During the Deployment Phase, we first retrieve the Top-1 trajectory corresponding to the most similar task description from ReasoningBank, then use memory items associated with the trajectory. The retrieved memory items are provided as memory for the agentic system.

G-Memory. G-Memory uses a graph-based memory design, managing memory across three tiers: the insight, query, and interaction graphs ( Zhang et al., 2025a ) . We follow the workflows, prompts, and default settings provided in https://github.com/bingreeky/GMemory . During the Memory Collection Phase, trajectories with task descriptions are first provided to the query graph. They then traverse upward to the insight graph to extract high-level insights, and downward to the interaction graph to identify core interactions within the trajectories, forming the three-tier graph memory. During the Deployment Phase, a task description is given to the query graph to retrieve the Top-1 most similar trajectory. The retrieved trajectory is then traversed upward and downward to identify the top-3 most similar insights and core interactions. The retrieved insights and core interactions are provided as memory for the agentic system.

### B.3 Additional Details of Learning Process

ALMA is initialized with a memory design generated from scratch by the Meta Agent powered by GPT-5/medium . The generation of memory design follows Python Abstract Classes (Appendix A.1 ) and serves as the starting point for subsequent exploration. During generation, the Meta Agent is provided with Provided Tools containing minimal usage examples for invoking different agent roles (e.g., reasoning or conversational models), as well as embedding functions and database operations.

Starting from this initial memory design, ALMA performs 11 learning steps. At each step, memory designs are sampled from the archive, and the Meta Agent proposes, implements, and evaluates new memory designs. Each evaluated new design, together with its performance metrics and interaction logs, is then added to the memory design archive. This process repeats with the updated archive until the maximum number of learning iterations is reached.

Sampling. We perform the sampling process (Appendix A.4 ) with α = 0.5 \alpha=0.5 to balance the visit-time penalty, and T = 0.5 T=0.5 to trade off between distribution smoothness and the probability of selecting higher-performing memory designs. Sampling is performed without replacement to maintain diversity in the selected subset while still favoring higher-probability designs, ensuring effective exploration and evaluation efficiency. As the memory design archive grows to more memory designs (e.g., 30 to 40), and only a small number of designs (five) are sampled at each step, the difference between sampling with and without replacement is expected to be negligible. Moreover, the purpose of sampling is to encourage exploration of memory designs that achieve high success rates but have been visited infrequently, while preserving a non-zero probability of sampling any memory design in the archive (Appendix A.4 ). This is achieved through a weighted sampling scheme that balances performance and visitation frequency. Under this objective, both sampling with and without replacement are suitable, as they preserve the intended probability structure and maintain effective exploration behavior. At each step, we sample up to five memory designs and perform parallel exploration of new memory designs.

Greedy Search. As an alternative to sampling in the learning process, we perform greedy search, always selecting the best memory design from the memory design archive at each learning step instead of sampling across all designs. This process continues until the total number of proposed designs matches that of the open-ended exploration. In our case, we perform 43 steps. Finally, the best memory design in the memory design archive is selected as the learned design produced by greedy search. The results of greedy search are shown in Section C.2 .

Ideate & Planning. The ideate & planning procedure utilizes sampled memory designs along with their interaction logs to propose ideas and plans for new memory designs. The system prompt for the procedure remains fixed across all benchmarks and throughout the learning process (Appendix A.5 ). For each sampled memory design, the relevant input information includes: (1) the source code of the current sampled memory design; (2) stratified-sampled interaction logs produced during the evaluation of the current sampled design; and (3) the overall success rate of the current sampled memory design. Additionally, we include the following components to adapt to different benchmarks and situations for planning: (1) a one-shot example of plan proposed for a previous memory design, together with the associated performance of the previous design; (2) a benchmark description, depending on the benchmark being learned. The full benchmark description prompt is provided in Benchmark Description .

The output reflection and plans for the new memory design follows below schema:

Implementation. The Meta Agent implements new memory design using the proposed plan. Specifically, we provide several components to the Meta Agent for implementation: (1) trajectory_score_assessment and suggested_changes in proposed plan, serve as instructions for implementation; (2) a one-shot trajectory example depending on the benchmarks as a reference format of possible input of the memory design; (3) Python Abstract Classes, which serve as the reference template for memory design implementation (Appendix A.1 ); and (4) provided tools for efficient implementation of new memory designs, as shown in Provided Tools .

As shown in Provided Tools , the Meta Agent is provided with the option of using GPT-4o-mini or GPT-4.1 as FMs within the memory design, for example, when extracting insights from interaction trajectories. By explicitly including multiple model options (e.g., reasoning model or chat model) in the search space, the Meta Agent can adaptively select the most suitable model based on different reasoning and implementation requirements during memory design construction. During testing, however, we consistently use GPT-4o-mini for all memory designs to ensure a fair comparison under matched model capacity.

We then perform a trial run to validate the implementation of the new memory design. The trial run samples 5 tasks from the learning set randomly, using 2 tasks for the Memory Collection Phase and 3 tasks for the Deployment Phase. If a runtime error occurs during the trial run, the Meta Agent engages in a reflection and debugging process, after which, the trial run is repeated on the debugged implementation. This loop of reflection, debugging, and trial runs iterates up to 3 times to ensure that the memory design implementation is free of bugs.

Evaluation. We integrate the new memory design with the agentic system powered by GPT-5-nano/low . We perform general_update on each trajectory during the Memory Collection Phase. During the Deployment Phase, we use general_retrieve to take the task description as input to retrieve relevant memory. The learning set is evenly split between the two phases. All tasks in the Deployment Phase are executed three times to compute the average success rate, reducing bias caused by variance in individual runs. For dataset splits, we use the first 30 tasks from the train set in ALFWorld as the learning set. For TextWorld and Baba Is AI, the datasets are evenly split into learning and testing sets. For MiniHack, 30% of the dataset is used as the learning set. The datasets for TextWorld, Baba Is AI, and MiniHack are sourced from BALROG ( Paglieri et al., 2025 ) . We perform stratified sampling ( Thompson, 2012 ) to extract interaction logs after the Deployment Phase and store extracted logs in the memory design archive. When a memory design is sampled in future learning steps, the stratified logs allow the Meta Agent to reflect on both successes and failures without needing to examine all interaction logs.

### B.4 Additional Details of Testing

After learning memory designs on each benchmark, we perform testing under both static and dynamic modes for ALFWorld. For other benchmarks, we perform testing under static mode. For TextWorld and Baba Is AI, we use the second halves of the datasets for testing, respectively. For MiniHack, we use the remaining 70% of the dataset for testing. All testing sets are split evenly for the Memory Collection and the Deployment Phase.

Static Mode. We first evaluate the best-learned memory designs and manual baselines on each benchmark using the agentic system powered by GPT-5-nano/low , the same model used during the learning process, to assess the effectiveness of the learned designs. We then test these memory designs with the agentic system powered by GPT-5-mini/medium to evaluate their transferability to a more capable agentic system. Additionally, for the ALFWorld valid_seen dataset, we test the learned best memory design alongside manual baselines while varying the number of trajectories used in the Memory Collection Phase from 0 to 70, using GPT-5-mini/medium as the foundational model in the agentic system.