Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems — part 1/12

Cleaned-text code points [0, 14685). Bibliography omitted here; source.md includes it.

# Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

###### Abstract

In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.

## 1 Introduction

Reinforcement learning provides a mathematical formalism for learning-based control. By utilizing reinforcement learning, we can automatically acquire near-optimal behavioral skills, represented by policies, for optimizing user-specified reward functions. The reward function defines what an agent should do, and a reinforcement learning algorithm determines how to do it. While the reinforcement learning algorithms have been an active area of research for decades, the introduction of effective high-capacity function approximators – deep neural networks – into reinforcement learning, along with effective algorithms for training them, has allowed reinforcement learning methods to attain excellent results along a wide range of domains ( Tesauro,, 1994 ; Hafner and Riedmiller,, 2011 ; Levine and Koltun,, 2013 ; Mnih et al.,, 2013 ; Levine et al.,, 2016 ; Silver et al.,, 2017 ; Kalashnikov et al.,, 2018 ) .

However, the fact that reinforcement learning algorithms provide a fundamentally online learning paradigm is also one of the biggest obstacles to their widespread adoption. The process of reinforcement learning involves iteratively collecting experience by interacting with the environment, typically with the latest learned policy, and then using that experience to improve the policy ( Sutton and Barto,, 1998 ) . In many settings, this sort of online interaction is impractical, either because data collection is expensive (e.g., in robotics, educational agents, or healthcare) and dangerous (e.g., in autonomous driving, or healthcare). Furthermore, even in domains where online interaction is feasible, we might still prefer to utilize previously collected data instead – for example, if the domain is complex and effective generalization requires large datasets.

Indeed, the success of machine learning methods across a range of practically relevant problems over the past decade can in large part be attributed to the advent of scalable data-driven learning methods, which become better and better as they are trained with more data. Online reinforcement learning is difficult to reconcile with this paradigm. While this was arguably less of an issue when reinforcement learning methods utilized low-dimensional or linear parameterizations, and therefore relied on small datasets for small problems that were easy to collect or simulate ( Lange et al.,, 2012 ) , once deep networks are incorporated into reinforcement learning, it is tempting to consider whether the same kind of data-driven learning can be applied with reinforcement learning objectives, thus resulting in data-driven reinforcement learning that utilizes only previously collected offline data, without any additional online interaction ( Kumar,, 2019 ; Fu et al.,, 2020 ) . See Figure 1 for a pictorial illustration. A number of recent works have illustrated the power of such an approach in enabling data-driven learning of policies for dialogue ( Jaques et al.,, 2019 ) , robotic manipulation behaviors ( Ebert et al.,, 2018 ; Kalashnikov et al.,, 2018 ) , and robotic navigation skills ( Kahn et al.,, 2020 ) .

Unfortunately, such data-driven offline reinforcement learning also poses major algorithmic challenges. As we will discuss in this article, many commonly used reinforcement learning methods can learn from off-policy data, but such methods often cannot learn effectively from entire offline data, without any additional on-policy interaction. High-dimensional and expressive function approximation generally exacerbates this issue, since function approximation leaves the algorithms vulnerable to distributional shift, one of the central challenges with offline reinforcement learning. However, the appeal of a fully offline reinforcement learning framework is enormous: in the same way that supervised machine learning methods have enabled data to be turned into generalizable and powerful pattern recognizers (e.g., image classifiers, speech recognition engines, etc.), offline reinforcement learning methods equipped with powerful function approximation may enable data to be turned into generalizable and powerful decision making engines , effectively allowing anyone with a large enough dataset to turn this dataset into a policy that can optimize a desired utility criterion. From healthcare decision-making support to autonomous driving to robotics, the implications of a reliable and effective offline reinforcement learning method would be immense.

In some application domains, the lack of effective offline reinforcement learning methods has driven research in a number of interesting directions. For example, in robotics and autonomous driving, a rapidly growing research topic is the study of simulation to real-world transfer: training policies with reinforcement learning in simulation and then transferring these policies into the real world ( Sadeghi and Levine,, 2017 ; Tan et al.,, 2018 ; Chebotar et al.,, 2019 ) . While this approach is very pragmatic (and often effective), its popularity highlights the deficiency in offline reinforcement learning methods: if it was possible to simply train policies with previously collected data, it would likely be unnecessary in many cases to manually design high-fidelity simulators for simulation-to-real-world transfer. After all, outside of reinforcement learning (e.g., in computer vision, NLP, or speech recognition), transfer from simulation is comparatively much less prevalent, since data-driven learning is so effective.

The goal of this article is to provide the reader with the conceptual tools needed to get started on research in the field of offline reinforcement learning (also called batch reinforcement learning ( Ernst et al.,, 2005 ; Riedmiller,, 2005 ; Lange et al.,, 2012 ) ), so as to hopefully begin addressing some of these deficiencies. To this end, we will present the offline reinforcement learning problem formulation, and describe some of the challenges associated with this problem setting, particularly in light of recent research on deep reinforcement learning and the interaction between reinforcement learning and high-dimensional function approximator, such as deep networks. We will cover a variety of offline reinforcement learning methods studied in the literature. For each one, we will discuss the conceptual challenges, and initial steps taken to mitigate these challenges. We will then discuss some of the applications of offline reinforcement learning techniques that have already been explored, despite the limitations of current methods, and conclude with some perspectives on future work and open problems in the field.

## 2 Offline Reinforcement Learning Problem Statement and Overview

In this section, we will introduce the mathematical formalism of reinforcement learning and define our notation, and then set up the offline reinforcement learning problem setting, where the goal is to learn near-optimal policies from previously collected data. Then, we will briefly discuss some of the intuition behind why the offline reinforcement learning problem setting poses some unique challenges, using a supervised behavioral cloning example.

### 2.1 Reinforcement Learning Preliminaries

In this section, we will define basic reinforcement learning concepts, following standard textbook definitions ( Sutton and Barto,, 1998 ) . Reinforcement learning addresses the problem of learning to control a dynamical system, in a general sense. The dynamical system is fully defined by a fully-observed or partially-observed Markov decision process (MDP).

###### Definition 2.1 (Markov decision process) .

The Markov decision process is defined as a tuple ℳ = ( 𝒮 , 𝒜 , T , d 0 , r , γ ) \mathcal{M}=(\mathcal{S},\mathcal{A},T,d_{0},r,\gamma) , where 𝒮 \mathcal{S} is a set of states 𝐬 ∈ 𝒮 \mathbf{s}\in\mathcal{S} , which may be either discrete or continuous (i.e., multi-dimensional vectors), 𝒜 \mathcal{A} is a set of actions 𝐚 ∈ 𝒜 \mathbf{a}\in\mathcal{A} , which similarly can be discrete or continuous, T T defines a conditional probability distribution of the form T ⁡ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) T(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) that describes the dynamics of the system, 1 1 1 We will sometimes use time subscripts (i.e., 𝐬 t + 1 \mathbf{s}_{t+1} follows 𝐬 t \mathbf{s}_{t} ), and sometimes “prime” notation (i.e., 𝐬 ′ \mathbf{s}^{\prime} is the state that follows 𝐬 \mathbf{s} ). Explicit time subscripts can help clarify the notation in finite-horizon settings, while “prime” notation is simpler in infinite-horizon settings where absolute time step indices are less meaningful. d 0 d_{0} defines the initial state distribution d 0 ​ ( 𝐬 0 ) d_{0}(\mathbf{s}_{0}) , r : 𝒮 × 𝒜 → r:\mathcal{S}\times\mathcal{A}\rightarrow\real defines a reward function, and γ ∈ ( 0 , 1 ] \gamma\in(0,1] is a scalar discount factor.

We will use the fully-observed formalism in most of this article, though the definition for the partially observed Markov decision process (POMDP) is also provided for completeness. The MDP definition can be extended to the partially observed setting as follows:

###### Definition 2.2 (Partially observed Markov decision process) .

The partially observed Markov decision process is defined as a tuple ℳ = ( 𝒮 , 𝒜 , 𝒪 , T , d 0 , E , r , γ ) \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},T,d_{0},E,r,\gamma) , where 𝒮 \mathcal{S} , 𝒜 \mathcal{A} , T T , d 0 d_{0} , r r , and γ \gamma are defined as before, 𝒪 \mathcal{O} is a set of observations, where each observation is given by 𝐨 ∈ 𝒪 \mathbf{o}\in\mathcal{O} , and E E is an emission function, which defines the distribution E ⁡ ( 𝐨 t | 𝐬 t ) E(\mathbf{o}_{t}|\mathbf{s}_{t}) .

The final goal in a reinforcement learning problem is to learn a policy, which defines a distribution over actions conditioned on states, π ⁡ ( 𝐚 t | 𝐬 t ) \pi(\mathbf{a}_{t}|\mathbf{s}_{t}) , or conditioned on observations in the partially observed setting, π ⁡ ( 𝐚 t | 𝐨 t ) \pi(\mathbf{a}_{t}|\mathbf{o}_{t}) . The policy may also be conditioned on an observation history, π ( 𝐚 t | 𝐨 0 : t ) \pi(\mathbf{a}_{t}|\mathbf{o}_{0:t}) . From these definitions, we can derive the trajectory distribution . The trajectory is a sequence of states and actions of length H H , given by τ = ( 𝐬 0 , 𝐚 0 , … , 𝐬 H , 𝐚 H ) \tau=(\mathbf{s}_{0},\mathbf{a}_{0},\dots,\mathbf{s}_{H},\mathbf{a}_{H}) , where H H may be infinite. The trajectory distribution p π p_{\pi} for a given MDP ℳ \mathcal{M} and policy π \pi is given by p π ​ ( τ ) = d 0 ​ ( 𝐬 0 ) ​ ∏ t = 0 H π ⁡ ( 𝐚 t | 𝐬 t ) ​ T ​ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) . p_{\pi}(\tau)=d_{0}(\mathbf{s}_{0})\prod_{t=0}^{H}\pi(\mathbf{a}_{t}|\mathbf{s}_{t})T(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}). This definition can easily be extended into the partially observed setting by including the observations 𝐨 t \mathbf{o}_{t} and emission function E ⁡ ( 𝐨 t | 𝐬 t ) E(\mathbf{o}_{t}|\mathbf{s}_{t}) . The reinforcement learning objective, J ⁡ ( π ) J(\pi) , can then be written as an expectation under this trajectory distribution: J ⁡ ( π ) = 𝔼 τ ∼ p π ​ ( τ ) ​ [ ∑ t = 0 H γ t ​ r ​ ( 𝐬 t , 𝐚 t ) ] . J(\pi)=\mathbb{E}_{\tau\sim p_{\pi}(\tau)}\left[\sum_{t=0}^{H}\gamma^{t}r(\mathbf{s}_{t},\mathbf{a}_{t})\right]. (1) When H H is infinite, it is sometimes also convenient to assume that the Markov chain on ( 𝐬 t , 𝐚 t ) (\mathbf{s}_{t},\mathbf{a}_{t}) defined by π ⁡ ( 𝐚 t | 𝐬 t ) ​ T ​ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) \pi(\mathbf{a}_{t}|\mathbf{s}_{t})T(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) is ergodic, and define the objective in terms of the expected reward under the stationary distribution of this Markov chain ( Sutton and Barto,, 1998 ) . This definition is somewhat complicated by the role of the discount factor. For a full discussion of this topic, we refer the reader to prior work ( Thomas,, 2014 ) .

In many cases, we will find it convenient to refer to the marginals of the trajectory distribution p π ​ ( τ ) p_{\pi}(\tau) . We will use d π ​ ( 𝐬 ) d^{\pi}(\mathbf{s}) to refer to the overall state visitation frequency, averaged over the time steps, and d t π ​ ( 𝐬 t ) d^{\pi}_{t}(\mathbf{s}_{t}) to refer to the state visitation frequency at time step t t .

In this section, we will briefly summarize different types of reinforcement learning algorithms and present definitions. At a high level, all standard reinforcement learning algorithms follow the same basic learning loop: the agent interacts with the MDP ℳ \mathcal{M} by using some sort of behavior policy , which may or may not match π ⁡ ( 𝐚 | 𝐬 ) \pi(\mathbf{a}|\mathbf{s}) , by observing the current state 𝐬 t \mathbf{s}_{t} , selecting an action 𝐚 t \mathbf{a}_{t} , and then observing the resulting next state 𝐬 t + 1 \mathbf{s}_{t+1} and reward value r t = r ⁡ ( 𝐬 t , 𝐚 t ) r_{t}=r(\mathbf{s}_{t},\mathbf{a}_{t}) . This may repeat for multiple steps, and the agent then uses the observed transitions ( 𝐬 t , 𝐚 t , 𝐬 t + 1 , r t ) (\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{s}_{t+1},r_{t}) to update its policy. This update might also utilize previously observed transitions. We will use 𝒟 = { ( 𝐬 t i , 𝐚 t i , 𝐬 t + 1 i , r t i ) } \mathcal{D}=\{(\mathbf{s}^{i}_{t},\mathbf{a}^{i}_{t},\mathbf{s}^{i}_{t+1},r^{i}_{t})\} to denote the set of transitions that are available for the agent to use for updating the policy (“learning”), which may consist of either all transitions seen so far, or some subset thereof.

#### Policy gradients.