Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems — part 11/12
Cleaned-text code points [149234, 164822). Bibliography omitted here; source.md includes it.
A number of recent works have also demonstrated effective offline learning of predictive models and their application to control in complex and high-dimensional domains, including settings with image observations. Some of these methods have directly utilized high-capacity predictive models on high-dimensional observations, such as images, directly employing for online trajectory optimization (i.e., MPC). Action-conditional convolutional neural networks ( Oh et al.,, 2015 ) have been used to provide accurate, long-term prediction of behavior in Atari games and have been combined with RL to improve sample-efficiency over model-free methods ( Kaiser et al., 2019b, ) . The visual foresight method involves training a video prediction model to predict future image observations for a robot, conditioned on the current image and future sequence of actions. The model is represented with a recurrent neural network, and trained on large amounts of offline data, typically collected with an uninformative randomized policy ( Finn and Levine,, 2017 ; Ebert et al.,, 2018 ) . More recent work has demonstrated this approach on very large and diverse datasets, collected from multiple viewpoints, many objects, and multiple robots, suggesting a high degree of generalization, though the particular behaviors are comparatively simple, typically relocating individual objects by pushing or grasping Dasari et al., (2019) . A number of prior works have also employed “hybrid” methods that combine elements of model-free and model-based algorithms, predicting future rewards or reward features conditioned on a sequence of future actions, but avoiding direct prediction of future observations. A number of such methods have been explored in the conventional online RL setting ( Tamar et al.,, 2016 ; Dosovitskiy and Koltun,, 2016 ; Oh et al.,, 2017 ) , and in the offline RL setting, such techniques have been applied effectively to learning navigational policies for mobile robots from previously collected data ( Kahn et al.,, 2018 ; Kahn et al.,, 2020 ) .
#### Off-policy evaluation with models.
Model learning has also been explored considerably in the domain of off-policy evaluation (OPE), as a way to reduce the variance of importance sampled estimators of the sort discussed in Section 3.1 . Here, the model is used to provide a sort of baseline for the expected return inside of an importance sampled estimator, as illustrated in Equation ( 6 ) in Section 3.1 . In these settings, the model is typically trained from offline data ( Jiang and Li,, 2015 ; Thomas and Brunskill,, 2016 ; Farajtabar et al.,, 2018 ; Wang et al.,, 2017 ) .
#### Distribution and safe region constraints.
As with the policy constraint methods in Section 4.3 , model-based RL algorithms can also utilize constraints imposed on the policy or planner with the learned model. Methods that guarantee Lyapunov stability ( Berkenkamp et al.,, 2017 ) of the learned model can be used to constrain policies within “safe” regions of the state space where the chance of failure is small. Another example of such an approach is provided by deep imitative models (DIMs) ( Rhinehart et al.,, 2018 ) , which learn a normalizing flow model over future trajectories from offline data, conditioned on a high-dimensional observation. Like the hybrid methods described above, DIMs avoid direct prediction of observations. The learned distribution over dataset trajectories can then be used to both provide predictions for a planner, and provide a distributional constraint, preventing the planner from planning trajectories that deviate significantly from the training data, thus limiting distributional shift.
#### Conservative model-based RL algorithms.
More recently, two concurrent methods, MoREL ( Kidambi et al.,, 2020 ) and MOPO ( Yu et al.,, 2020 ) , have proposed offline model-based RL algorithms that aim to utilize conservative value estimates to provide analytic bounds on performance. Conceptually, these methods resemble the conservative estimation approach described in Section 4.5 , but instead of regularizing a value function, they modify the MDP model learned from data to induce conservative behavior. The basic principle is to provide the policy with a penalty for visiting states under the model where the model is likely to be incorrect. If the learned policy takes actions that remain in regions where the model is accurate, then the model-based estimate of the policy’s value (and its gradient) will likely be accurate also. Let u ( 𝐬 , 𝐚 ) u(\mathbf{s},\mathbf{a}) denote an error oracle that provides a consistent estimate of the accuracy of the model at state-action tuple ( 𝐬 , 𝐚 ) (\mathbf{s},\mathbf{a}) . For example, as proposed by Yu et al., (2020) , this oracle might satisfy the property that D ( T ψ ( s t + 1 | s t , a t ) ∥ T ( s t + 1 | s t , a t ) ) ≤ u ( 𝐬 , 𝐚 ) D(T_{\psi}(s_{t+1}|s_{t},a_{t})\|T(s_{t+1}|s_{t},a_{t}))\leq u(\mathbf{s},\mathbf{a}) for some divergence measure D ( ⋅ , ⋅ ) D(\cdot,\cdot) . Then, conservative behavior can be induced either by modifying the reward function to obtain a conservative reward of the form r ~ ( 𝐬 , 𝐚 ) = r ( 𝐬 , 𝐚 ) − λ u ( 𝐬 , 𝐚 ) \tilde{r}(\mathbf{s},\mathbf{a})=r(\mathbf{s},\mathbf{a})-\lambda u(\mathbf{s},\mathbf{a}) , as in MOPO ( Yu et al.,, 2020 ) , or by modifying the MDP under the learned model so that the agent enters an absorbing state with a low reward value when u ( 𝐬 , 𝐚 ) u(\mathbf{s},\mathbf{a}) is below some threshold, as in MoREL ( Kidambi et al.,, 2020 ) . In both cases, it is possible to show that the model-based estimate of the policy’s performance under the model the modified reward function or MDP structure bounds that policy’s true performance in the real MDP, providing appealing theoretical guarantees on performance. Note, however, that such approaches still require a consistent estimator for the error oracle u ( 𝐬 , 𝐚 ) u(\mathbf{s},\mathbf{a}) . Prior work has used disagreement in a bootstrap ensemble to provide this estimate, but it is not guaranteed to be consistent under sampling error, and resolving this limitation is likely to be an important direction for future work.
### 5.3 Challenges and Open Problems
Although model-based reinforcement learning appears to be a natural fit for the offline RL problem setting, current methods for fully offline model-based reinforcement learning generally rely on explicit uncertainty estimation for the model to detect and quantify distributional shift, for example by using a bootstrap ensemble. Although uncertainty estimation for models is in some ways more straightforward than uncertainty estimation for value functions, current uncertainty estimation methods still leave much to be desired, and it seems likely that offline performance of model-based RL methods can be improved substantially by carefully accounting for distributional shift.
Model-based RL methods also present their own set of challenges: while some MDPs are easy to model accurately, others can be exceedingly difficult. Modeling MDPs with very high-dimensional image observations and long horizons is a major open problem, and current predictive modeling methods generally struggle with long-horizon prediction. Hybrid methods that combine model-based and model-free learning, for example by utilizing short rollouts ( Sutton,, 1991 ; Janner et al.,, 2019 ) or avoiding prediction of full observations ( Dosovitskiy and Koltun,, 2016 ; Oh et al.,, 2017 ; Kahn et al.,, 2020 ) offer some promise in this area.
It is also still an open theoretical question as to whether model-based RL methods even in theory can improve over model-free dynamic programming algorithms. The reasoning behind this question is that, although dynamic programming methods do not explicitly learn a model, they essentially utilize the dataset as a “non-parametric” model. Fundamentally, both dynamic programming methods and model-based RL methods are solving prediction problems, with the former predicting future returns, and the latter predicting future states. Moreover, model-free methods can be modified to predict even more general quantities ( Sutton et al.,, 2011 ) , such as return values with different discount factors, return values for different number of steps into the future, etc. In the linear function approximation case, it is known that model-based updates and fitted value iteration updates actually produce identical iterates ( Vanseijen and Sutton,, 2015 ; Parr et al.,, 2008 ) , though it is unknown whether this relationship holds under non-linear function approximation. Therefore, exploring the theoretical bounds on the optimal performance of offline model-based RL methods under non-linear function approximation, as compared to offline dynamic programming methods, remains an open problem.
## 6 Applications and Evaluation
In this section, we survey and discuss evaluation methods, benchmarks, and applications for offline reinforcement learning methods. As discussed in Section 1 , and as we will discuss further in Section 7 , it is very likely that the full potential of offline reinforcement learning methods has yet to be fully realized, and perhaps the most exciting applications of such methods are still ahead of us. Nonetheless, a number of prior works have applied offline reinforcement learning in a range of challenging domains, from safety-critical real-world physical systems to large-scale learning from logged data for recommender systems. We first discuss how offline reinforcement learning algorithms have been evaluated in prior work, and then discuss specific application domains where such methods have already made an impact.
### 6.1 Evaluation and Benchmarks
While individual application domains, such as recommender systems and healthcare, discussed below, have developed particular domain-specific evaluations, the general state of benchmarking for modern offline reinforcement learning research is less well established. In the absence of well-developed evaluation protocols, one approach employed in recent work is to utilize training data collected via a standard online reinforcement learning algorithm, using either the entire replay buffer for an off-policy algorithm for training ( Kumar et al.,, 2019 ; Agarwal et al.,, 2019 ; Fujimoto et al.,, 2018 ) , or even data from the optimal policy. However, this evaluation setting is rather unrealistic, since the entire point of utilizing offline reinforcement learning algorithms in the real world is to obtain a policy that is better than the best behavior in the dataset, potentially in settings where running reinforcement learning online is impractical due to cost or safety concerns. A simple compromise solution is to utilize data from a “suboptimal” online reinforcement learning run, for example by stopping the online process early, saving out the buffer, and using this buffer as the dataset for offline RL ( Kumar et al.,, 2019 ) . However, even this formulation does not fully evaluate capabilities of offline reinforcement learning methods, and the statistics of the training data have a considerable effect on the difficult of offline RL ( Fu et al.,, 2020 ) , including how concentrated the data distribution is around a specific set of trajectories, and how multi-modal the data is. Broader data distributions (i.e., ones where π β ( 𝐚 | 𝐬 ) {\pi_{\beta}}(\mathbf{a}|\mathbf{s}) has higher entropy) are generally easier to learn from, since there are fewer out-of-distribution actions. On the other hand, highly multi-modal behavior policies can be extremely difficult to learn from for methods that require explicit estimation of π β ( 𝐚 | 𝐬 ) {\pi_{\beta}}(\mathbf{a}|\mathbf{s}) , as discussed in Section 4.3 and 4.6 . Our recently proposed set of offline reinforcement learning benchmarks aims to provide standardized datasets and simulations that cover such difficult cases ( Fu et al.,, 2020 ) .
A reasonable question we might ask in regard to datasets for offline RL is: in which situations might we actually expect offline RL to yield a policy that is significantly better than any trajectory in the training set? While we cannot expect offline RL to discover actions that are better than any action illustrated in the data, we can expect it to effectively utilize the compositional structure inherent in any temporal process. This idea is illustrated in Figure 4 : if the dataset contains a subsequence illustrating a way to arrive at state 2 2 from state 1 1 , as well as a separate subsequence illustrating how to arrive at state 3 3 from state 2 2 , then an effective offline RL method should be able to learn how to arrive at state 3 3 from state 1 1 , which might provide for a substantially higher final reward than any of the subsequences in the dataset. When we also consider the capacity of neural networks to generalize, we could imagine this sort of “transitive induction” taking place on a portion of the state variables, effectively inferring potentially optimal behavior from highly suboptimal components. This capability can be evaluated with benchmarks that explicitly provide data containing this structure, and the D4RL benchmark suite provides a range of tasks that exercise this capability ( Fu et al.,, 2020 ) .
Accurately evaluating the performance of offline RL algorithms can be difficult, because we are typically interested in maximizing the online performance of an algorithm. When simulators are available, online evaluations can be cheaply performed within the simulator order to benchmark the performance of algorithms. Off-policy evaluation (OPE) can also be used to estimate the performance of policies without explicit online interaction, but it is an active area of research as discussed in Section 3.1 . Nevertheless, OPE is a popular tool in areas such as online advertising ( Li et al.,, 2010 ) or healthcare ( Murphy et al.,, 2001 ) where online evaluation can have significant financial or safety consequences. In certain domains, human experts can be used to assess the quality of the decision-making system. For example, Jaques et al., (2019) uses crowd-sourced human labeling to judge whether dialogue generated by an offline RL agent is fluent and amicable, and Raghu et al., (2017) evaluates using a qualitative analysis based one domain experts for sepsis treatment.
### 6.2 Applications in Robotics
Robotics is an appealing application domain for offline reinforcement learning, since RL has the potential to automate the acquisition of complex behavioral skills for robots – particularly with raw sensory observations, such as camera images – but conducting online data collection for each robotic control policy can be expensive and impractical. This is especially true for robots that must learn to act intelligently in complex open-world environments, since the challenge of robust visual perception alone already necessitates large training sets. The ImageNet Large-Scale Visual Recognition Challenge ( Russakovsky et al.,, 2015 ) stipulates a training set of 1.5 million images for open-world object recognition, and it seems reasonable that the sample complexity for a robotic RL algorithm that must act in similar real-world settings should be at least of comparable size. For this reason, utilizing previous collected data can be of critical importance in robotics.
Several prior works have explored offline RL methods for learning robotic grasping, which is a particularly appealing task offline RL methods, since it requires the ability to generalize to a wide range of