Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems β part 3/12
Cleaned-text code points [29399, 44871). Bibliography omitted here; source.md includes it.
Model-based reinforcement learning is a general term that refers to a broad class of methods that utilize explicit estimates of the transition or dynamics function T β‘ ( π¬ t + 1 | π¬ t , π t ) T(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) , parameterized by a parameter vector Ο \psi , which we will denote T Ο β ( π¬ t + 1 | π¬ t , π t ) T_{\psi}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) . There is no single recipe for a model-based reinforcement learning method. Some commonly used model-based reinforcement learning algorithms learn only the dynamics model T Ο β ( π¬ t + 1 | π¬ t , π t ) T_{\psi}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) , and then utilize it for planning at test time, often by means of model-predictive control (MPC) ( Tassa et al.,, 2012 ) with various trajectory optimization methods ( Nagabandi et al.,, 2018 ; Chua et al.,, 2018 ) . Other model-based reinforcement learning methods utilize a learned policy Ο ΞΈ β ( π t | π¬ t ) \pi_{\theta}(\mathbf{a}_{t}|\mathbf{s}_{t}) in addition to the dynamics model, and employ backpropagation through time to optimize the policy with respect to the expected reward objective ( Deisenroth and Rasmussen,, 2011 ) . Yet another set of algorithms employ the model to generate βsyntheticβ samples to augment the sample set available to model-free reinforcement learning methods. The classic Dyna algorithm uses this recipe in combination with Q-learning and one-step predictions via the model from previously seen states ( Sutton,, 1991 ) , while a variety of recently proposed algorithms employ synthetic model-based rollouts with policy gradients ( Parmas et al.,, 2019 ; Kaiser et al., 2019a, ) and actor-critic algorithms ( Janner et al.,, 2019 ) . Since there are so many variants of model-based reinforcement learning algorithms, we will not go into detail on each of them in this section, but we will discuss some considerations for offline model-based reinforcement learning in Section 5 .
### 2.2 Offline Reinforcement Learning
The offline reinforcement learning problem can be defined as a data-driven formulation of the reinforcement learning problem. The end goal is still to optimize the objective in Equation ( 1 ). However, the agent no longer has the ability to interact with the environment and collect additional transitions using the behavior policy. Instead, the learning algorithm is provided with a static dataset of transitions, π = { ( π¬ t i , π t i , π¬ t + 1 i , r t i ) } \mathcal{D}=\{(\mathbf{s}^{i}_{t},\mathbf{a}^{i}_{t},\mathbf{s}^{i}_{t+1},r^{i}_{t})\} , and must learn the best policy it can using this dataset. This formulation more closely resembles the standard supervised learning problem statement, and we can regard π \mathcal{D} as the training set for the policy. In essence, offline reinforcement learning requires the learning algorithm to derive a sufficient understanding of the dynamical system underlying the MDP β³ \mathcal{M} entirely from a fixed dataset, and then construct a policy Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) that attains the largest possible cumulative reward when it is actually used to interact with the MDP . We will use Ο Ξ² {\pi_{\beta}} to denote the distribution over states and actions in π \mathcal{D} , such that we assume that the state-action tuples ( π¬ , π ) β π (\mathbf{s},\mathbf{a})\in\mathcal{D} are sampled according to π¬ βΌ d Ο Ξ² β ( π¬ ) \mathbf{s}\sim d^{{\pi_{\beta}}}(\mathbf{s}) , and the actions are sampled according to the behavior policy, such that π βΌ Ο Ξ² β ( π | π¬ ) \mathbf{a}\sim{\pi_{\beta}}(\mathbf{a}|\mathbf{s}) .
This problem statement has been presented under a number of different names. The term βoff-policy reinforcement learningβ is typically used as an umbrella term to denote all reinforcement learning algorithms that can employ datasets of transitions π \mathcal{D} where the corresponding actions in each transition were collected with any policy other than the current policy Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) . Q-learning algorithms, actor-critic algorithms that utilize Q-functions, and many model-based reinforcement learning algorithm are off-policy algorithms. However, off-policy algorithms still often employ additional interaction (i.e., online data collection) during the learning process. Therefore, the term βfully off-policyβ is sometimes used to indicate that no additional online data collection is performed. Another commonly used term is βbatch reinforcement learningβ ( Ernst et al.,, 2005 ; Riedmiller,, 2005 ; Lange et al.,, 2012 ) . While this term has been used widely in the literature, it can also cause some amount of confusion, since the use of a βbatchβ in an iterative learning algorithm can also refer to a method that consumes a batch of data, updates a model, and then obtains a different batch, as opposed to a traditional online learning algorithm, which consumes one sample at a time. In fact, Lange et al., (2012) further introduces qualifiers βpureβ and βgrowingβ batch reinforcement learning to clarify this. To avoid this confusion, we will instead use the term βoffline reinforcement learningβ in this tutorial.
The offline reinforcement learning problem can be approached using algorithms from each of the four categories covered in the previous section, and in principle any off-policy RL algorithm could be used as an offline RL algorithm. For example, a simple offline RL method can be obtained simply by using Q-learning without additional online exploration, using π \mathcal{D} to pre-populate the data buffer. This corresponds to changing the initialization of π \mathcal{D} in Algorithm 2 , and setting S = 0 S=0 . However, as we will discuss later, not all such methods are effective in the offline setting.
### 2.3 Example Scenarios
Before delving deeper into the technical questions surrounding offline RL, we will first discuss a few example scenarios where offline RL might be utilized. These scenarios will help us to understand the factors that we must consider when designing offline RL methods that are not only convergent and principled, but also likely to work well in practice. A more complete discussion of actual applications is provided in Section 6 .
#### Decision making in health care.
An example health care scenario might formulate a Markov decision process to model the process of diagnosing and treating a patient, where actions correspond to various available interventions (e.g., diagnostic tests and treatments), and observations correspond to the patientβs symptoms and results of diagnostic tests. A partially observed MDP formulation may be most suitable in such cases. Conventional active reinforcement learning in such scenarios might be prohibitively dangerous β even utilizing a fully trained policy to treat a patient is a difficult prospect for clinicians, and deploying a partially trained policy would be out of the question. Therefore, offline RL might be the only viable path to apply reinforcement learning in such settings. Offline data would then be obtained from treatment histories of real patients, with the βactionsβ that were selected by their physician.
#### Learning goal-directed dialogue policies.
Dialogue can be viewed as interactive sequential decision making problem, which can also be modeled as an MDP, particularly when the dialogue is goal-directed (e.g., a chat bot on an e-commerce website that is offering information about a product to persuade a user to make a purchase). However, since the goal for such agents is to interact successfully with real humans, collecting trials requires interacting with live humans, which may be prohibitively expensive at the scale needed to train effective conversational agents. However, offline data can be collected directly from humans, and may indeed be natural to collect in any application domain where the aim is to partially or completely supplant human operators, by recording the interactions that are already taking place with the human-operated system.
#### Learning robotic manipulation skills.
In a robotic manipulation setting, active reinforcement learning may in fact be quite feasible. However, we might want to learn policies for a variety of robotic skills (e.g., all of the steps necessary to prepare a variety of meals for a home cooking robot) that generalize effectively over different environments and settings. In that case, each skill by itself might require a very large amount of interaction, as we would not only need to collect enough data to learn the skill, but enough data such that this skill generalizes effectively to all the situations (e.g., all the different homes) in which the robot might need to perform it. With offline RL, we could instead imagine including all of the data the robot has ever collected for all of its previously learned skills in the data buffer for each new skill that it learns. In this way, some skills could conceivably be acquired without any new data collection, if they can be assembled from parts of previously learned behaviors (e.g., cooking a soup that includes onions and carrots can likely be learned from experience cooking a soup with onions and meat, as well as another soup with carrots and cucumbers). In this way, offline RL can effectively utilize multi-task data.
### 2.4 What Makes Offline Reinforcement Learning Difficult?
Offline reinforcement learning is a difficult problem for multiple reasons, some of which are reasonably clear, and some of which might be a bit less clear. Arguably the most obvious challenge with offline reinforcement learning is that, because the learning algorithm must rely entirely on the static dataset π \mathcal{D} , there is no possibility of improving exploration: exploration is outside the scope of the algorithm, so if π \mathcal{D} does not contain transitions that illustrate high-reward regions of the state space, it may be impossible to discover those high-reward regions. However, because there is nothing that we can do to address this challenge, we will not spend any more time on it, and will instead assume that π \mathcal{D} adequately covers the space of high-reward transitions to make learning feasible. 2 2 2 It is worth noting that defining this notion formally is itself an open problem, and to the best knowledge of the authors, there are no known non-trivial βsufficiencyβ conditions on π \mathcal{D} that allows us to formulate guarantees that any offline reinforcement learning algorithm will recover an optimal or near-optimal policy.
A more subtle but practically more important challenge with offline reinforcement learning is that, at its core, offline reinforcement learning is about making and answering counterfactual queries. Counterfactual queries are, intuitively, βwhat ifβ questions. Such queries require forming hypotheses about what might happen if the agent were to carry out a course of action different from the one seen in the data. This is a necessity in offline RL, since we if we want the learned policy to perform better than the behavior seen in the dataset π \mathcal{D} , we must execute a sequence of actions that is in some way different. Unfortunately, this strains the capabilities of many of our current machine learning tools, which are designed around the assumption that the data is independent and identically distributed (i.i.d.). That is, in standard supervised learning, the goal is to train a model that attains good performance (e.g., high accuracy) on data coming from the same distribution as the training data. In offline RL, the whole point is to learn a policy that does something differently (presumably better) from the pattern of behavior observed in the dataset π \mathcal{D} .
The fundamental challenge with making such counterfactual queries is distributional shift: while our function approximator (policy, value function, or model) might be trained under one distribution, it will be evaluated on a different distribution, due both to the change in visited states for the new policy and, more subtly, by the act of maximizing the expected return. This latter point is discussed in more detail in Section 4 . Distributional shift issues can be addressed in several ways, with the simplest one being to constrain something about the learning process such that the distributional shift is bounded. For example, by constraining how much the learned policy Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) differs from the behavior policy Ο Ξ² β ( π | π¬ ) {\pi_{\beta}}(\mathbf{a}|\mathbf{s}) , we can bound state distributional shift ( Kakade and Langford,, 2002 ; Schulman et al.,, 2015 ) .
In this section, we will provide a short theoretical illustration of how harmful distributional shift can be on the performance of policies in MDPs. In this example, based on Ross et al., (2011) , we will assume that we are provided with optimal action labels π β \mathbf{a}^{\star} at each state π¬ β π \mathbf{s}\in\mathcal{D} . One might expect that, under such a strong assumption, the performance of our learned policy should be at least as good as the policies that we can learn with reinforcement learning without such optimal action labels. The goal in this analysis will be to bound the number of mistakes made by the learned policy Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) based on this labeled dataset, denoted as β β‘ ( Ο ) = πΌ p Ο β ( Ο ) β [ β t = 0 H Ξ΄ β‘ ( π t β π t β ) ] . \ell(\pi)=\mathbb{E}_{p_{\pi}(\tau)}\left[\sum_{t=0}^{H}\delta(\mathbf{a}_{t}\neq\mathbf{a}_{t}^{\star})\right]. If we train Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) with supervised learning (i.e., standard empirical risk minimization) on this labeled dataset, we have the following result from Ross et al., (2011) :
###### Theorem 2.1 (Behavioral cloning error bound) .
If Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) is trained via empirical risk minimization on π¬ βΌ d Ο Ξ² β ( π¬ ) \mathbf{s}\sim d^{{\pi_{\beta}}}(\mathbf{s}) and optimal labels π β \mathbf{a}^{\star} , and attains generalization error Ο΅ \epsilon on π¬ βΌ d Ο Ξ² β ( π¬ ) \mathbf{s}\sim d^{{\pi_{\beta}}}(\mathbf{s}) , then β β‘ ( Ο ) β€ C + H 2 β Ο΅ \ell(\pi)\leq C+H^{2}\epsilon is the best possible bound on the expected error of the learned policy.
###### Proof.
The proof follows from Theorem 2.1 from Ross et al., (2011) using the 0-1 loss, and the bound is the best possible bound following the example from Ross and Bagnell, (2010) . β
Interestingly, if we allow for additional data collection, where we follow the learned policy Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) to gather additional states π¬ βΌ d Ο β ( π¬ ) \mathbf{s}\sim d^{\pi}(\mathbf{s}) , and then access optimal action labels for these new on-policy states, the best possible bound becomes substantially better:
###### Theorem 2.2 (DAgger error bound) .
If Ο β‘ ( π | π¬ ) \pi(\mathbf{a}|\mathbf{s}) is trained via empirical risk minimization on π¬ βΌ d Ο β ( π¬ ) \mathbf{s}\sim d^{\pi}(\mathbf{s}) and optimal labels π β \mathbf{a}^{\star} , and attains generalization error Ο΅ \epsilon on π¬ βΌ d Ο β ( π¬ ) \mathbf{s}\sim d^{\pi}(\mathbf{s}) , then β β‘ ( Ο ) β€ C + H β Ο΅ \ell(\pi)\leq C+H\epsilon is the best possible bound on the expected error of the learned policy.
###### Proof.
The proof follows from Theorem 3.2 from Ross et al., (2011) . This is the best possible bound, because the probability of a mistake at any time step is at least Ο΅ \epsilon , and β t = 1 H Ο΅ = H β Ο΅ \sum_{t=1}^{H}\epsilon=H\epsilon . β