MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory — part 5/6

Cleaned-text code points [62688, 78041). Bibliography omitted here; source.md includes it.

• Pass@k: This is a standard sampling-based baseline. It generates k k independent candidate solutions for a given query and selects the best one based on the benchmark’s verifier (if available) or reports the success rate if at least one candidate passes. This baseline serves as a measure of the inherent capability of the frozen LLM without any memory persistence.

##### II. Retrieval-Augmented Generation (RAG) Approaches

• RAG ( Lewis et al., 2020 ) : Represents the standard semantic retrieval paradigm. It utilizes an embedding model to encode the current query and retrieves the top- k k most semantically similar past experiences (or documents) from the external memory. These retrieved contexts are then prepended to the prompt to guide the LLM’s generation.

• Self-RAG ( Asai et al., 2023 ) : An advanced RAG variant that incorporates a self-critique mechanism. Unlike standard RAG, Self-RAG performs selective retrieval and uses a critique model (or self-prompting) to verify the relevance and factual correctness of the retrieved content before integrating it into the generation process.

##### III. Agentic Memory Systems

• Mem0 ( Chhikara et al., 2025 ) : A recently proposed memory layer for LLMs that manages memory through structured operations. It employs specific APIs for adding, retrieving, and updating memory, aiming to maintain a personalized and persistent context across interactions.

• MemP ( Fang et al., 2025 ) : A framework focused on procedural memory. It distills past successful trajectories into reusable, procedure-like memory entries. MemP maintains a memory repository using a build-retrieve-update cycle, allowing the agent to recall high-level plans rather than raw trajectory data.

### D.2 Benchmark Datasets

We evaluate performance across four benchmarks selected to cover diverse domains: code generation, OS interactions, embodied decision-making, and multidisciplinary reasoning.

• BigCodeBench ( Zhuo et al., 2025 ) : A challenging benchmark for library-oriented code generation that requires agents to implement complex functionalities using diverse third-party libraries. Unlike traditional benchmarks focused on algorithmic snippets, BigCodeBench emphasizes practical software engineering capabilities. We evaluate on the BigCodeBench-Instruct (Full) split, which tasks the agent with synthesizing complete functional code from natural language instructions across the full range of difficulty levels.

• Lifelong Agent Bench ( Zheng et al., 2025 ) : Designed to evaluate agents in a continuous learning setting involving Operating System (OS) and Database (DB) interactions. It tests the agent’s capacity to adapt to new tools and commands over a long horizon without forgetting previous skills.

• ALFWorld ( Shridhar et al., 2021 ) : An embodied navigation and manipulation benchmark. It requires the agent to solve textual logic puzzles within a simulated household environment (e.g., “put a clean apple in the fridge”). This tests the agent’s ability to learn and retrieve multi-step plans.

• Humanity’s Last Exam (HLE) ( Phan et al., 2025 ) : A rigorous multidisciplinary reasoning benchmark featuring hard problems from mathematics, humanities, and sciences. It serves as a stress test for the agent’s general reasoning capability and its ability to retrieve relevant knowledge for disparate tasks.

## Appendix E Implementation and Reproducibility Details

To facilitate reproducibility, we provide the exact model versions, hyperparameter settings, and environmental configurations used in our experiments.

### E.1 Model Specifications

All LLM reasoning and generation tasks were performed using the models listed in Table 7 . We used the official APIs with a fixed temperature to ensure deterministic evaluation where possible.

### E.2 Hyperparameter Settings

Table 8 details the specific hyperparameters used for MemRL and the baselines. The similarity threshold δ \delta is adaptive to the dataset density; specifically, we determine δ \delta by calculating the pairwise cosine similarity distribution of task descriptions within each benchmark and selecting the threshold at the top 20% quantile. This ensures that only the most relevant historical experiences are considered for retrieval.

### E.3 Data Partitioning

To evaluate the effectiveness of MemRL , we categorize our experiments into Runtime Learning and Transfer Learning settings. Table 9 summarizes the dataset sizes and partitioning strategies used for each benchmark.

† For ALFWorld Transfer Learning, the agent uses the same 3,553 tasks as memory context but is evaluated on 140 novel instances of seen task types.

For benchmarks utilizing random splits ( OS , DB , and BCB ), we use a fixed random seed of 42 to ensure reproducibility. In ALFWorld , the Transfer Learning phase specifically tests generalization to new instances within known categories, ensuring the agent learns procedural patterns rather than specific trajectories.

### E.4 Model Selection and Performance Analysis

We select the backbone model for each benchmark to ensure a valid learning signal relative to task complexity. Since MemRL relies on high-value trajectories to perform utility update over retrieved memories, extremely low initial competence can make the feedback effectively unusable. For instance, on challenging benchmarks such as HLE, weaker models may start at ≈ 4 % \approx 4\% success rate (e.g., GPT-4o-mini ), yielding too few successful trajectories for stable utility estimation; in this scenario, environmental feedback is dominated by noise, which can prevent meaningful learning and hinder convergence.

At the other extreme, using the strongest available models on simpler benchmarks can cause performance saturation (ceiling effects), leaving little headroom to quantify the marginal gains attributable to the memory mechanism. Therefore, we match model capability to each task’s difficulty to avoid both the no-signal situation (insufficient successes) and the ceiling effect (insufficient headroom). This design choice enables a more faithful evaluation of improvements that are intrinsic to MemRL rather than artifacts of an ill-posed performance condition.

Importantly, our evaluation spans backbones of different scales—from “mini” to “pro” tiers (Table 7 )—to test whether MemRL remains effective across capability levels. The resulting performance trends support the scale-invariant robustness of MemRL . Moreover, the cross-model transfer results (Table 5 ) further indicate that the procedural knowledge captured in memory is portable across backbones, suggesting that the learned utility over memories generalizes beyond any single model’s inherent strength.

To further explore the generality of MemRL across different model capabilities and task complexities, we also deployed the GPT-4o-mini model in the ALFWorld benchmark. This model is identical to the backbone used in the Lifelong Agent Bench, allowing for a direct comparison of its performance across environments with varying exploration intensities. As an exploration-intensive environment, ALFWorld poses significant demands on the base model’s reasoning and planning capabilities. The Table 10 and Table 11 present the runtime learning and knowledge transfer abilities of GPT-4o-mini .

These results, using GPT-4o-mini as a consistent backbone, unequivocally demonstrate MemRL ’s superior and generalized effectiveness across all benchmarks. In runtime learning (Table 10 ), MemRL achieves the highest average Last Epoch Success Rate and Cumulative Success Rate, showing significant gains: its average Last Epoch SR is 12.6 % 12.6\% points higher than “No Memory”, and its average CSR surpasses MemP by 11.2 % 11.2\% points. This advantage is particularly pronounced in the challenging ALFWorld environment. Similarly, in transfer learning (Table 11 ), MemRL secures the highest average Success Rate, 10.6 % 10.6\% points higher than “No Memory” and 6.1 % 6.1\% points higher than MemP. These consistent improvements across tasks and learning paradigms confirm that MemRL effectively enhances the GPT-4o-mini backbone’s decision-making capabilities, leveraging learned utility over memories for more robust performance.

## Appendix F Cost and Efficiency Analysis

Real-world deployment of autonomous agents requires balancing performance gains with computational costs. In this section, we analyze the token consumption and runtime latency of MemRL compared to the strong baseline MemP on the compute-intensive Humanity’s Last Exam (HLE) benchmark.

### F.1 Token Consumption

Since MemRL operates as a non-parametric approach without gradient-based fine-tuning, the primary cost arises from LLM API calls. We compare the average token usage per question (Q) across the entire learning trajectory (10 epochs).

MemRL ’s token consumption is comparable to that of MemP, as both methods utilize identical interaction loops (reasoning + summarization). On the HLE benchmark, the average total token consumption per question for MemRL is approximately 32K . This similarity in token usage is because the complexity of MemRL lies in how memories are retrieved and updated (via Q-values), not in how much context is fed to the LLM. Therefore, the significant performance gains of MemRL reported in the main text are achieved without increasing the inference budget.

### F.2 Runtime Latency and Stability

A common concern with two-stage retrieval and reinforcement learning components is the potential for increased latency. We empirically validate the wall-clock time required to complete each epoch (2,500 questions) in Figure 11 .

##### Negligible Algorithmic Overhead.

Figure 11 demonstrates that the runtime of MemRL is commensurate with, and often more stable than, that of MemP. The fluctuations observed (e.g., the spikes in the MemP curve at Epochs 4 and 6) are attributed to external factors such as API network latency and throughput variability, rather than algorithmic complexity.

Specifically, the additional components in MemRL introduce minimal computational cost: • Dual-Stage Retrieval: This involves basic vector dot-products and scalar score weighting, operating in milliseconds.

• RL Update: The Q-value update in a Monte Carlo style on scalars, which is an O ⁡ ( 1 ) O(1) operation.

Compared to the hours required for LLM generation over 2,500 questions, these millisecond-level operations are virtually imperceptible. The stability of the MemRL curve further suggests that our method does not introduce complex blocking operations that would exacerbate network-induced delays.

## Appendix G Extended Limitations and Future Work

This appendix provides a more detailed discussion on the limitations of MemRL and outlines promising avenues for future research, building upon the foundations established in the main paper.

### G.1 Update Protocols and Memory Consolidation

The current step-wise update protocol of MemRL , while enabling rapid adaptation, can introduce high-variance noise in long-horizon trajectories. This presents a challenge for stabilizing value estimation over extended sequences. A valuable future direction involves exploring more robust multi-step update mechanisms, which could offer slower but more stable learning. Furthermore, combining these with periodic consolidation of similar intentions and experiences within the memory bank could significantly improve the spatial efficiency of the memory, reducing redundancy and enhancing retrieval quality.

### G.2 Precise Credit Assignment in Multi-Memory Updates

A significant challenge arises from the credit-assignment ambiguity encountered when multiple experiences from the memory are referenced and updated simultaneously. Determining the precise contribution of each referenced memory to the final outcome is complex and affects the efficiency of learning. Future research could investigate methods for more precise experience attribution, drawing inspiration from techniques like Shapley values ( Shapley and others, 1953 ) , commonly used in cooperative game theory, or value decomposition methods prevalent in multi-agent reinforcement learning ( Rashid et al., 2020 ; Sunehag et al., 2017 ) . Such approaches could lead to more accurate updates and faster convergence.

### G.3 Task Similarity and Generalization

MemRL ’s effectiveness is observed to improve with the number of encountered tasks, aligning well with the characteristics of runtime learning. This is because the algorithm leverages past experiences, and a richer, more diverse set of similar experiences directly enhances its ability to retrieve relevant knowledge. However, when task similarity in the experience base is low, the method may inherently degrade into a less efficient reflection-like behavior, as direct experience transfer is limited. This is not a deficiency of the method but rather a characteristic of its reliance on learned utility from memory. For industrial deployment and real-world applications, this implies that maintaining a sufficiently high “task similarity density” within the agent’s operating environment or its collected memory is crucial. Strategies to achieve this could include active curriculum learning, environment design that promotes diverse yet related tasks, or techniques that facilitate hierarchical abstraction to enable generalization beyond direct, low-level experience matches. Enhancing retrieval mechanisms to proactively identify and adapt to novel task structures, or integrating hierarchical abstraction, are key for improving cross-task generalization.

### G.4 Memory Security and Robustness to Attack

MemRL is sensitive to the quality of feedback, particularly vulnerable to “reward hacking” if the verifier produces false positives. Incorrectly learned high Q-values from spurious feedback can quickly solidify and propagate erroneous behavioral patterns, leading to systemic failures. This highlights a critical challenge: memory security. A maliciously injected sample into the memory bank could rapidly diffuse pollution, potentially causing an intelligent agent to collapse. Future research must address the safety and trustworthiness of agent memories. Fortunately, the mutable nature of experience offers a silver lining: once contamination or attack is identified, polluted experiences can be swiftly pruned, allowing for recovery without disrupting the utility of previously learned valid experiences.

### G.5 Multi-Agent Collaboration and Shared Memory

As many enterprises transition from single agents to multi-agent clusters (swarms), the question of “shared memory” becomes important. Does the MemRL approach support knowledge sharing, allowing lessons learned and updated Q-values from one agent to immediately benefit others? We’ve explored scenarios where the memory, treated as a persistent resource, can indeed be shared among different models or agents (Appendix C.1 ). Furthermore, adopting a multi-agent paradigm, akin to distributed parameter updates, could be highly beneficial. Each agent could accumulate experience and save it as memory during its operation, contributing to a large, collective memory pool. This would enable different agents to implicitly leverage each other’s learned experiences, fostering a more collaborative and efficient learning ecosystem.