MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory — part 1/6

Cleaned-text code points [0, 15690). Bibliography omitted here; source.md includes it.

# MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

###### Abstract

The hallmark of human intelligence is the self-evolving ability to master new skills by learning from past experiences. However, current AI agents struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL , a non-parametric approach that evolves via reinforcement learning on episodic memory. By decoupling stable reasoning from plastic memory, MemRL employs a Two-Phase Retrieval mechanism to filter noise and identify high-utility strategies through environmental feedback. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines, confirming that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates. Code is available at https://github.com/MemTensor/MemRL .

###### Keywords:

## 1 Introduction

Human intelligence balances cognitive stability and episodic plasticity ( Grossberg, 2013 ; McClelland et al., 1995 ; Kumaran et al., 2016 ) via Constructive Episodic Simulation, enabling adaptation without rewiring neural circuitry ( Schacter and Addis, 2007 ; Hassabis and Maguire, 2007 ; Schacter et al., 2012 ; Gick and Holyoak, 1980 ) . Despite their reasoning capabilities, current AI agents struggle to emulate this decoupled self-evolution ( Wei et al., 2022 ; Yao et al., 2023 ; Schick et al., 2023 ; Wang et al., 2023 ) . Specifically, fine-tuning internalizes experience by modifying weights ( Ouyang et al., 2022 ; Stiennon et al., 2020 ; Rafailov et al., 2023 ; Ethayarajh et al., 2024 ) but suffers from computational costs and catastrophic forgetting ( Kirkpatrick et al., 2017 ; Li et al., 2024 ; Wu et al., 2024 ) . Conversely, Retrieval-Augmented Generation (RAG) ( Lewis et al., 2020 ) provides a non-parametric alternative but remains passive, retrieving by semantic similarity rather than utility ( Karpukhin et al., 2020 ; Gao et al., 2024 ) ; this prevents agents from effectively leveraging runtime feedback to distinguish high-value strategies from noise.

This limitation underscores a critical research question: How can we enable an agent to continuously improve its performance after deployment, without compromising the stability of its pre-trained backbone? Our objective is to achieve an agent that evolves with continued usage and rapidly adapts to new tasks after deployment, referred to as Runtime Continuous Learning ( Javed et al., 2023 ; Silver and Sutton, 2025 ; Parisi et al., 2019 ; Wu et al., 2024 ) , all while keeping the backbone model frozen to prevent catastrophic forgetting ( Finn et al., 2017 ; Wei et al., 2025 ) . To address this challenge, inspired by the human cognitive mechanism of constructive simulation, we propose MemRL , an approach that facilitates self-evolving agents by explicitly decoupling the model’s stable cognitive reasoning from dynamic episodic memory. Figure 1 illustrates the conceptual framework of our proposed MemRL . Drawing on tries-and-errors manner in Reinforcement Learning (RL) to estimate expected experience utilities ( Sutton and Barto, 2018 ) , we formalize the interaction between the frozen LLM and external memory as a Markov Decision Process (MDP) ( Puterman, 2014 ) . Unlike traditional methods that optimize the backbone model, MemRL optimizes the policy of memory usage without tuning model weights.

MemRL organizes memory into a structured Intent-Experience-Utility triplet. This structure transforms retrieval from a passive semantic match task into an active decision-making process: Two-Phase Retrieval selects experiences based on their learned Q-values, reflecting expected utility, rather than semantic similarity alone ( Watkins and Dayan, 1992 ) ; Utility-Driven Update refines these Q-values through environmental feedback, applying Monte Carlo style updates ( Metropolis and Ulam, 1949 ) . This closed-loop cycle enables the agent to distinguish high-value memories from similar noise, effectively learning from both success and failure without high computational cost or catastrophic forgetting risks associated with weight updates. As for experiments, we validate MemRL on four diverse benchmarks, including HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench. Our results demonstrate consistent superiority over baselines, achieving relative improvement in exploration-heavy environments. Our in-depth analysis reveals a strong correlation between learned utility and task success, further confirming MemRL ’s effectiveness.

In summary, our contributions are threefold: • We propose a runtime learning framework using Model-Memory decoupling and Intent-Experience-Utility triplet to reconcile the stability-plasticity dilemma, enabling tuning-free agent learning.

• We introduce MemRL , a non-parametric approach enabling agent self-evolution via Two-Phase Retrieval and Utility-Driven Update.

• We conduct extensive evaluations and provide a rigorous analysis for MemRL ’s stability, showing how it ensures task integrity and minimizes forgetting.

## 2 Related Works

##### Runtime Learning

Runtime Learning focuses on the post-deployment improvement of agents through interaction streams rather than offline data, marking a shift toward the “era of experience” ( Silver and Sutton, 2025 ) . Unlike Continual Learning ( Parisi et al., 2019 ; Wu et al., 2024 ) or Test-Time Adaptation ( Sun et al., 2020 ; Wang et al., 2020 ; Liang et al., 2025 ) , which typically update parameters to handle forgetting or distribution shifts, our setting constrains the backbone to remain frozen to ensure stability and efficiency. While recent memory-augmented agents ( Zheng et al., 2025 ; Wei et al., 2025 ; Zhou et al., 2025b ) emphasize memory organization, the selection problem—identifying which experiences to reuse under feedback—remains a critical challenge. Drawing from value-aware episodic control ( Tulving and others, 1972 ; McClelland et al., 1995 ) , we frame runtime learning as identifying valuable episodes. By using interaction feedback to assign utility, our approach guides retrieval and reuse without weight modification, thereby ensuring sustained improvement ( Pritzel et al., 2017 ) .

##### Reinforcement Learning

Reinforcement learning has been widely adopted for LLMs enhancement. A representative paradigm is to construct reward signals from human feedback and optimize the model policy accordingly to align with human preference ( Stiennon et al., 2020 ; Ouyang et al., 2022 ) . Other recent approaches leverage rule-based verifiers to improve LLMs’ reasoning capabilities ( Guo et al., 2025 ; Yu et al., 2025 ) . In parallel, agent-oriented research explores how interaction signals can improve tool use and action decision-making, and investigates mechanisms by which language models execute composite actions in environments ( Schick et al., 2023 ) . Despite the demonstrated effectiveness of reward-driven optimization, these methods generally place learning in the model parameters or additional parametric modules, and thus do not avoid the cost of online updates or the risk of forgetting. In contrast, our method frames memory usage as a learnable decision problem and applies non-parametric reinforcement learning on memory to bypass the risk.

##### Agentic Memory

To avoid the costs of fine-tuning, external memory systems have evolved from a static RAG paradigm to dynamic, governable memory structures ( Lewis et al., 2020 ; Karpukhin et al., 2020 ) . Early agentic memory introduced reflection mechanisms and hierarchical management to handle long context experiences ( Shinn et al., 2023 ; Packer et al., 2024 ) . More recent frameworks have systematized the memory lifecycle, focusing on unified storage and structured indexing for complex tasks ( Li et al., 2025b ; Xu et al., 2025 ; Huang et al., 2025 ; Ye, 2025 ) . Furthermore, adaptive approaches now explore improving retrieval via feedback-driven updates or automated augmentation ( Salama et al., 2025 ; Zhang et al., 2025 ; Li et al., 2025a ; Zhou et al., 2025a ) . However, except for training additional learnable modules, most existing methods still rely predominantly on semantic similarity or heuristic rules, lacking a rigorous metric to evaluate the actual utility of a memory in maximizing returns. Inspired by cognitive theories of memory reconsolidation ( Schacter and Addis, 2007 ; Gick and Holyoak, 1980 ; Nader et al., 2000 ) , MemRL bridges this gap by formulating retrieval as a value-based decision process, learning robust utility estimates (Q-values) from environmental rewards to distinguish high-value experiences.

## 3 Problem Formulation

In this section, we formally define the problem of memory-augmented generation and establish the theoretical link between agent policy and memory retrieval. We adopt the formulation of Memory-Based Markov Decision Process (M-MDP) ( Zhou et al., 2025a ) , and apply our non-parametric reinforcement learning approach to it. Figure 2 provides an illustrative example of this memory-augmented decision process, showing how retrieval outcomes and memory evolution unfold over multiple time steps.

### 3.1 Memory-Augmented Agent Policy

To enable the agent to self-evolution, we adopt the M-MDP framework ( Zhou et al., 2025a ) , defined by the tuple ( 𝒮 , 𝒜 , P , ℛ , γ , ℳ ) (\mathcal{S},\mathcal{A},P,\mathcal{R},\gamma,\mathcal{M}) . Here, 𝒮 \mathcal{S} and 𝒜 \mathcal{A} represent state and action spaces, P P is the transition dynamics, ℛ \mathcal{R} is the reward function of state and action, γ ∈ [ 0 , 1 ) \gamma\in[0,1) denotes the discount factor, and ℳ = ( 𝒮 × 𝒜 × ℝ ) ∗ \mathcal{M}=(\mathcal{S}\times\mathcal{A}\times\mathbb{R})^{*} constitutes the evolving memory bank of past experiences ( Zhou et al., 2025a ) . At each step t t , the agent receives state s t s_{t} and leverages ℳ t \mathcal{M}_{t} to generate a response a t a_{t} maximizing the expected reward. The joint policy π ⁡ ( a t | s t , ℳ t ) \pi(a_{t}|s_{t},\mathcal{M}_{t}) is formulated as the marginal probability over all possible retrieved items m m ( Zhou et al., 2025a ) : π ⁡ ( a t | s t , ℳ t ) = ∑ m ∈ ℳ t μ ⁡ ( m | s t , ℳ t ) ​ p L ​ L ​ M ​ ( a t | s t , m ) . \pi(a_{t}|s_{t},\mathcal{M}_{t})=\sum_{m\in\mathcal{M}_{t}}\mu(m|s_{t},\mathcal{M}_{t})p_{LLM}(a_{t}|s_{t},m). (1) where μ ⁡ ( m | s t , ℳ t ) \mu(m|s_{t},\mathcal{M}_{t}) is the Retrieval Policy for selecting memory contexts, and p L ​ L ​ M ​ ( a t | s t , m ) p_{LLM}(a_{t}|s_{t},m) is the Inference Policy parameterized by a frozen LLM. This approach transforms retrieval from a passive match into an active decision process, effectively accounting for the functional utility of m m in generating successful outcomes a t a_{t} .

In previous RAG or memory-based agentic paradigms, the retrieval policy μ \mu is usually determined by a fixed vector similarity metric, e.g., cosine similarity of embeddings. While effective for semantic matching, such policies fail to account for the utility of a memory, i.e., whether retrieving m m actually leads to a successful outcome a t a_{t} .

### 3.2 Non-Parametric Reinforcement Learning

To overcome static similarity limitations, we operationalize the M-MDP framework by formulating memory retrieval as a value-based decision-making process ( Zhou et al., 2025a ) . Unlike parametric methods optimizing π L ​ L ​ M \pi_{LLM} via weight updates, we optimize the retrieval policy μ ⁡ ( m | s , ℳ ) \mu(m|s,\mathcal{M}) directly within the memory space by mapping M-MDP components to a structured Intent-Experience-Utility triplet:

From Semantic Matching to Decision Making. We instantiate state s s as the User Intent, encapsulated by the current query embedding ( Lewis et al., 2020 ) . Consequently, the action space 𝒜 t \mathcal{A}_{t} becomes dynamic and discrete, corresponding to selecting a specific m m from the memory bank ℳ t \mathcal{M}_{t} ( Zhou et al., 2025a ) . In this formulation, retrieval is not a passive matching task but a strategic decision step taken to augment the generator’s action a a ( Zhou et al., 2025a ) .

Defining Utility via Q-Values. While the agent’s executable action a a is generated by the policy π \pi (as formalized in Sec. 3.1 ), the quality of this generation is strictly conditioned on the retrieved context. Therefore, we adapt the traditional value function Q ⁡ ( s , a ) Q(s,a) to the retrieval phase, defining Q ⁡ ( s , m ) Q(s,m) as the expected utility of the subsequent action a a augmented by memory m m . MemRL learns an optimal retrieval policy μ ∗ \mu^{*} that maximizes this utility: μ ∗ ​ ( m | s , ℳ ) = arg ⁡ max m ∈ ℳ ⁡ Q ⁡ ( s , m ) . \mu^{*}(m|s,\mathcal{M})=\arg\max_{m\in\mathcal{M}}Q(s,m). (2) In this view, the Q-value acts as a critic for the retrieval mechanism, distinguishing memories that strategically aid the generator from irrelevant noise that merely shares high semantic similarity.

Non-Parametric Learning. Since the retrieval action space is decoupled from LLM generation, we perform learning without modifying model weights. Upon receiving environmental feedback r r , we update the Q-value via a Temporal-Difference (TD) error ( Sutton, 1988 ) : Q ⁡ ( s , m ) ← Q ⁡ ( s , m ) + α ⁡ [ r + γ ​ max ⁡ Q ⁡ ( s ′ , m ′ ) − Q ⁡ ( s , m ) ] , Q(s,m)\leftarrow Q(s,m)+\alpha[r+\gamma\max Q(s^{\prime},m^{\prime})-Q(s,m)], (3) or Monte Carlo style rule ( Metropolis and Ulam, 1949 ) : Q new ← Q old + α ⁡ ( r − Q old ) , Q_{\text{new}}\leftarrow Q_{\text{old}}+\alpha\big(r-Q_{\text{old}}\big), (4) where α \alpha is the learning rate. Equation 4 performs as a naturally simplified version of Equation 3 by setting s ′ s^{\prime} as a terminal state to balance complexity and performance, sharing a similar one-step MDP formulation with Guo et al. (2025) . These manners allow utility estimates to converge to true expected returns over time ( Bellman, 1966 ) . By explicitly updating Q-values within the memory structure, MemRL provides a non-parametric learning manner with a theoretical guarantee, enabling agents to self-evolve through interaction.

## 4 MemRL

Building upon the M-MDP formulation defined in Section 3 , we propose MemRL , a framework that enables frozen LLMs to self-evolve via non-parametric reinforcement learning. Instead of modifying the model weights θ \theta , MemRL optimizes the retrieval policy μ ⁡ ( m | s , ℳ ) \mu(m|s,\mathcal{M}) within an evolving memory space. As illustrated in Figure 3 , the framework consists of three core components: (i) a structured Intent-Experience-Utility memory bank, (ii) a Two-Phase Retrieval mechanism that decouples semantic recall from value-aware selection, and (iii) a Runtime Utility Update rule that stabilizes Q-value estimation.

### 4.1 The Intent-Experience-Utility Triplet

To support value-based decision-making, we structure the external memory ℳ \mathcal{M} not merely as key-value pairs, but as a set of triplets: ℳ = { ( z i , e i , Q i ) } i = 1 | ℳ | , \mathcal{M}=\{(z_{i},e_{i},Q_{i})\}_{i=1}^{|\mathcal{M}|}, (5) where z i z_{i} represents the Intent , e i e_{i} stores the raw Experience (e.g., a successful solution trace or trajectory), and Q i Q_{i} denotes the learned Utility . Q i Q_{i} approximates the expected return of applying experience e i e_{i} to intents similar to z i z_{i} , serving as the critic in RL.

### 4.2 From Semantic Recall to Value-Aware Selection