MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

arxiv:2601.03192

Original source · Paper JSON

Agents, open-endedness, AGI · 2026-01-06

Cites (3) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6

Contents

• MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory
• Abstract
• Keywords:
• 1 Introduction
• 2 Related Works
• Runtime Learning
• Reinforcement Learning
• Agentic Memory
• 3 Problem Formulation
• 3.1 Memory-Augmented Agent Policy
• 3.2 Non-Parametric Reinforcement Learning
• 4 MemRL
• 4.1 The Intent-Experience-Utility Triplet
• 4.2 From Semantic Recall to Value-Aware Selection
• 4.3 Non-Parametric RL on Memory
• 4.4 Theoretical Stability Analysis
• 5 Experiments
• 5.1 Experimental Setup
• 5.2 Main Results
• Runtime Learning Results.
• Transferring Results.
• 5.3 Ablations
• 5.3.1 Effectiveness of Runtime RL
• 5.3.2 Impact of Q-Value Weighting
• 5.3.3 Ablation Analysis: Cross-Task vs. Single-Task Optimization
• 5.3.4 Sensitivity to Retrieval Size ( k 1 k_{1} and k 2 k_{2} ).
• 5.4 Discussion
• 5.4.1 Predictive Power of the Q Critic
• 5.4.2 Stability of MemRL
• 5.4.3 Extended Analysis.
• 6 Limitations and Conclusion
• Limitations and Future Work.
• Conclusion.
• Impact Statement
• Appendix A Theoretical Analysis and Proofs
• A.1 Stability Analysis
• Setup.
• Stationary Reward Assumption.
• Expected Convergence of Utility Estimates.
• Theorem A.1 .
• Proof.
• Bounded Variance and Stability.
• Global Stability via EM Convergence.
• A.2 Proof of Theorem A.1 : Convergence of EMA Estimation
• Assumptions.
• Derivation of Error Dynamics.
• Convergence Analysis.
• A.3 Bounded Variance and Global Stability
• Derivation of the Variance Bound.
• Recursive Unrolling.
• Asymptotic Convergence.
• Connection to Phase-A Clustering.
• A.4 Convergence via Variational Inference
• A.4.1 The Convergence Objective
• A.4.2 Variational Objective with Trust Region
• A.4.3 Optimization via Generalized Expectation-Maximization (GEM)
• E-Step (Policy Optimization).
• M-Step (Policy Evaluation via Error Minimization).
• A.4.4 Proof of Convergence
• Appendix B Extended Analysis and Insights
• B.1 Detailed Analysis of Forgetting Dynamics
• B.2 MemRL as a Trajectory Verifier.
• B.3 Impact of Task Similarity on Memory Efficacy
• Appendix C Advanced Capabilities: Transfer and Merging
• C.1 Cross-Model Memory Transferability
• C.2 Multi-Task Memory Merging and Interference Analysis
• Appendix D Baseline and Benchmark Details
• D.1 Baselines
• I. Test-Time Scaling Strategy
• II. Retrieval-Augmented Generation (RAG) Approaches
• III. Agentic Memory Systems
• D.2 Benchmark Datasets
• Appendix E Implementation and Reproducibility Details
• E.1 Model Specifications
• E.2 Hyperparameter Settings
• E.3 Data Partitioning
• E.4 Model Selection and Performance Analysis
• Appendix F Cost and Efficiency Analysis
• F.1 Token Consumption
• F.2 Runtime Latency and Stability
• Negligible Algorithmic Overhead.
• Appendix G Extended Limitations and Future Work
• G.1 Update Protocols and Memory Consolidation
• G.2 Precise Credit Assignment in Multi-Memory Updates
• G.3 Task Similarity and Generalization
• G.4 Memory Security and Robustness to Attack
• G.5 Multi-Agent Collaboration and Shared Memory
• G.6 Dedicated Domains and Hybrid Architectures
• Appendix H Case Study: High-Utility Failure Analysis (Near-Misses)
• Appendix I Prompts
• I.1 Experience Summarization Prompts
• I.2 Generation and Inference Prompts
• Instructions for reporting errors