Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems — part 12/12

Cleaned-text code points [164822, 180098). Bibliography omitted here; source.md includes it.

objects ( Pinto and Gupta,, 2016 ; Levine et al.,, 2018 ; Kalashnikov et al.,, 2018 ; Zeng et al.,, 2018 ) . Such methods have utilized approximate dynamic programming ( Kalashnikov et al.,, 2018 ) (see Figure 5 ), as well as more domain-specific algorithms, such as a single-step bandit formulation ( Pinto and Gupta,, 2016 ) . Outside of robotic grasping, Ebert et al., (2018) propose a model-based algorithm based on prediction of future video frames for learning a variety of robotic manipulation skills from offline data, while Dasari et al., (2019) expand on this approach with a large and diverse dataset of robotic interaction. Cabi et al., (2019) propose to use reward learning from human preferences combined with offline RL to provide a user-friendly method for controlling robots for object manipulation tasks. In the domain of robotic navigation, Mo et al., (2018) propose a dataset of visual indoor scenes for reinforcement learning, collected via a camera-mounted-robot, and Mandlekar et al., (2020) proposes a dataset of human demonstrations for robotic manipulation. Kahn et al., (2020) discuss how a method based on prediction of future reward signals, blending elements of model-based and model-free learning, can learn effective navigation policies from data collected in the real world using a random exploration policy. Pure model-based methods in robotics typically involve training a model on real or simulated data, and then planning within the model to produce a policy that is executed on a real system. Approaches have included using Gaussian process models for controlling blimps ( Ko et al.,, 2007 ) , and using linear regression ( Koppejan and Whiteson,, 2009 ) and locally-weighted Bayesian regression ( Bagnell and Schneider,, 2001 ) for helicopter control.

### 6.3 Applications in Healthcare

Using offline reinforcement learning in healthcare poses several unique challenges ( Gottesman et al.,, 2018 ) . Safety is a major concern, and largely precludes any possibility of online exploration. Datasets can also be significantly biased towards serious outcomes ( Gottesman et al.,, 2019 ) , since minor cases rarely require treatment, and can lead naïve agents to erroneous conclusions, for example that any drug treatment may cause death simply because it is not prescribed to otherwise healthy individuals.

The MIMIC-III dataset ( Johnson et al.,, 2016 ) , which contains approximately 60K medical records from ICUs, has been influential in enabling data-driven research in healthcare treatment. Q-learning methods on this dataset has been applied to problems such as the treatment of sepsis ( Raghu et al.,, 2017 ) and optimizing the use of ventilators ( Prasad et al.,, 2017 ) . Wang et al., (2018) apply actor-critic methods on MIMIC-III to determine drug recommendations.

Outside of ICU settings, offline RL applications include learning from recordings of seizure activity in mouse brains in order to determine optimal stimulation frequencies for reducing epileptic seizures ( Guez et al.,, 2008 ) . Offline RL has also been used for optimizing long term treatment plans. Shortreed et al., (2011) uses offline fitted Q-iteration for optimizing schizophrenia treatment, Nie et al., (2019) uses doubly-robust estimators to safely determine proper timings of medical treatments, and Tseng et al., (2017) uses a model-based approach for lung cancer treatment. Careful application of offline RL that can handle such challenges may offer healthcare providers powerful assistive tools for optimizing the care of patients and ultimately improving outcomes.

### 6.4 Applications in Autonomous Driving

As in healthcare, a significant barrier to applying reinforcement learning in the domain of self-driving vehicles is safety. In the online setting, exploratory agents can select actions that lead to catastrophic failure, potentially endangering the lives of the passengers. Thus, offline RL is potentially a promising tool for enabling, safe, effective learning in autonomous driving.

While offline RL has not yet found significant application in actual real-world self-driving vehicles ( Yurtsever et al.,, 2020 ) , learning-based approaches have been gaining in popularity. RobotCar ( Maddern et al.,, 2017 ) and BDD-100K ( Yu et al.,, 2018 ) are both large video datasets containing thousands of hours of real-life driving activity. Imitation learning has been a popular approach towards end-to-end, data-driven methods in autonomous driving ( Bojarski et al.,, 2016 ; Sun et al., 2018a, ; Pan et al.,, 2017 ; Codevilla et al.,, 2018 ) . Reinforcement learning approaches have been applied in both simulation ( Sallab et al.,, 2017 ) and in the real world, with human interventions in case the vehicle violates a safety constraint ( Kendall et al.,, 2019 ) (see Figure 7 ). Model-based reinforcement learning methods that employ constraints to keep the agent close to the training data for the model, so as to avoid out-of-distribution inputs as discussed in Section 5 , can effectively provide elements of imitation learning when training on driving demonstration data, as for example in the case of deep imitative models (DIMs) ( Rhinehart et al.,, 2018 ) . Indeed, with the widespread availability of high-quality demonstration data, it is likely that effective methods for offline RL in the field of autonomous driving will, explicitly or implicitly, combine elements of imitation learning and reinforcement learning.

### 6.5 Applications in Advertising and Recommender Systems

Recommender systems and advertising are particularly suitable domains for offline RL because data collection is easy and efficient, and can be obtained by logging user behavior. However, these domains are also “safety critical,” in the sense that a highly suboptimal policy may result in large monetary losses, thereby making unconstrained online exploration problematic. Thus, offline RL approaches have a long history of application in this area.

Off-policy evaluation is commonly used as a tool for performing A/B tests and estimating the performance of advertising and recommender systems without additional interaction with the environment. In contrast to the other applications discussed, policy evaluation for recommender systems is typically formulated within a contextual bandit problem ( Langford et al.,, 2008 ; Li et al.,, 2010 ) , where states may correspond to user history and actions correspond to recommendations. This approximation removes the need for sequential decision making, but can introduce approximation errors if actions have temporal dependence as in domains such as robotics or healthcare.

Applications of offline RL for recommender systems include slate and whole-page optimization ( Swaminathan et al.,, 2017 ) , applying doubly robust estimation to estimate website visits Dudík et al., (2014) , and A/B testing for click optimization ( Gilotte et al.,, 2018 ) . Policy learning from logged, offline data has included studies on optimizing newspaper article click-through-rates ( Strehl et al.,, 2010 ; Garcin et al.,, 2014 ) , advertisement ranking on search pages ( Bottou et al.,, 2013 ) , and personalized ad recommendation for digital marketing ( Theocharous et al.,, 2015 ; Thomas et al.,, 2017 ) .

### 6.6 Applications in Language and Dialogue

Interaction via natural language is not typically thought of as a reinforcement learning problem, but in fact the formalism of sequential decision making can provide a powerful tool for natural language interaction: when dialogue is modeled as a purposeful interaction, the RL framework can in principle offer an effective mechanism for learning policies for outputting natural language responses to human interlocutors. The most direct way to utilize standard online RL methods for natural language interaction – by having machines engage in dialogue with real humans – can be exceedingly tedious, especially in the early stages of training, when the policy would produce mostly non-sensical dialogue. For this reason, offline RL offers a natural avenue to combine the optimal decision making formalism of RL with the kinds of large datasets of human-to-human conversations available in NLP.

In prior work, offline RL approaches have been applied in the areas of dialogue and language interfaces, where datasets consist of logged interactions, such as agent-customer transcripts ( Zhou et al.,, 2017 ) . An example of an application is dialogue management, which is typically concerned

with accomplishing a specific goal, such as retrieving information. Examples of this have included applications of offline RL to the problem of flight booking ( Henderson et al.,, 2008 ) , restaurant information retrieval Pietquin et al., (2011) , and restaurant recommendation ( Kandasamy et al.,, 2017 ) . Jaques et al., (2019) applies offline RL to the problem of dialogue generation, and focuses on producing natural responses that elicit positive feedback from human users. An example of an interaction with a trained agent is shown in Figure 8 .

## 7 Discussion and Perspectives

Offline reinforcement learning offers the possibility of turning reinforcement learning – which is conventionally viewed as a fundamentally active learning paradigm – into a data-driven discipline, such that it can benefit from the same kind of “blessing of scale” that has proven so effective across a range of supervised learning application areas ( LeCun et al.,, 2015 ) . However, making this possible will require new innovations that bring to bear sophisticated statistical methods and combine them with the fundamentals of sequential decision making that are conventionally studied in reinforcement learning. Standard off-policy reinforcement learning algorithms have conventionally focused on dynamic programming methods that can utilize off-policy data, as discussed in Section 2.1 and Section 4 , and importance sampling methods that can incorporate samples from different sampling distributions, as discussed in Section 3.2 .

However, both of these classes of approaches struggle when scaled up to complex high-dimensional function approximators, such as deep neural networks, high-dimensional state or observation spaces, and temporally extended tasks. As a result, the standard off-policy training methods in these two categories have generally proven unsuitable for the kinds of complex domains typically studied in modern deep reinforcement learning. More recently, a number of improvements for offline RL methods have been proposed that take into account the statistics of distributional shift, via either policy constraints or uncertainty estimation, such as the policy constraint formulations that we discuss in Section 4.3 . These formulations have the potential to mitigate the shortcomings of early methods, by explicitly account for the key challenge in offline RL: distributional shift due to differences between the learned policy and the behavior policy.

More generally, such methods shed light on the fact that offline RL is, at its core, a counter-factual inference problem: given data that resulted from a given set of decisions, infer the consequence of a different set of decisions. Such problems are known to be exceptionally challenging in machine learning, because they require us to step outside of the commonly used i.i.d. framework, which assumes that test-time queries involve the same distribution as the one that produced the training data ( Schölkopf,, 2019 ) . It therefore stands to reason that the initial solutions to this problem, proposed in recent work, should aim to reduce distributional shift, either by constraining the policy’s deviation from the data, or by estimating (epistemic) uncertainty as a measure of distributional shift. Moving forward, we might expect that a variety of tools developed for addressing distributional shift and facilitating generalization may find use in offline RL algorithms, including techniques from causal inference ( Schölkopf,, 2019 ) , uncertainty estimation ( Gal and Ghahramani,, 2016 ; Kendall and Gal,, 2017 ) , density estimation and generative modeling ( Kingma et al.,, 2014 ) , distributional robustness ( Sinha et al.,, 2017 ; Sagawa et al.,, 2019 ) and invariance ( Arjovsky et al.,, 2019 ) . More broadly, methods that aim to estimate and address distributional shift, constrain distributions (e.g., various forms of trust regions), and evaluate distribution support from samples are all potentially relevant to developing improved methods for offline reinforcement learning.

The counter-factual inference perspective becomes especially clear when we consider model-based offline RL algorithms, as discussed briefly in Section 5 . In this case, the model answers the question: “what would the resulting state be if the agent took an action other than the one in the dataset?” Of course, the model suffers from distributional shift in much the same way as the value function, since out-of-distribution state-action tuples can result in inaccurate prediction. Nonetheless, prior work has demonstrated good results with model-based methods, particularly in regard to generalization with real-world data ( Finn and Levine,, 2017 ; Ebert et al.,, 2018 ) , and a range of works on predicting physical phenomena have utilized offline datasets ( Lerer et al.,, 2016 ; Battaglia et al.,, 2016 ) . This suggests that model learning may be an important component of effective future offline reinforcement learning methods.

To conclude our discussion of offline reinforcement learning, we will leave the reader with the following thought. While the machine learning community frequently places considerable value on design of novel algorithms and theory, much of the amazing practical progress that we have witnessed over the past decade has arguably been driven just as much by advances in datasets as by advances in methods. Indeed, widely deployed techniques in computer vision and NLP utilize learning methods that are relatively old and well understood, and although improvements in architectures and models have driven rapid increase in performance, the increasing size and diversity of datasets – particularly in real-world applications – have been an instrumental driver of progress. In real-world applications, collecting large, diverse, representative, and well-labeled datasets is often far more important than utilizing the most advanced methods. In the standard active setting in which most reinforcement learning methods operate, collecting large and diverse datasets is often impractical, and in many applications, including safety-critical domains such as driving, and human-interactive domains such as dialogue systems, it is prohibitively costly in terms of time, money, and safety considerations. Therefore, developing a new generation of data-driven reinforcement learning may usher in a new era of progress in reinforcement learning, both by making it possible to bring it to bear on a range of real-world problems that have previously been unsuited for such methods, and by enabling current applications (e.g., in robotics or autonomous driving) to benefit from much larger, more diverse, and more representative datasets that can be reused effectively across experiments.