Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems β€” part 10/12

Cleaned-text code points [133966, 149234). Bibliography omitted here; source.md includes it.

While such explicit uncertainty-based methods are conceptually attractive, it is often very hard to obtain calibrated uncertainty estimates to evaluate 𝒫 π’Ÿ ​ ( Q ^ Ο€ ) \mathcal{P}_{\mathcal{D}}(\hat{Q}^{\pi}) and Unc in practice, especially with modern high-capacity function approximators, such as deep neural networks. In practice, policy constraint and conservative value function methods so far seem to outperform pure uncertainty-based methods ( Fujimoto et al.,, 2018 ) . This might at first appear surprising, since uncertainty estimation has been a very widely used tool in another subfield of reinforcement learning – exploration. In exploration, acting optimistically with respect to estimated uncertainty, or utilizing posterior sampling, has been shown to be effective in practice ( Osband and Van Roy,, 2017 ) . However, in the setting of offline reinforcement learning, where we instead must act conservatively with respect to the uncertainty set, the demands on the fidelity of the uncertainty estimates are much higher. Exploration algorithms only require the uncertainty set to include good behavior – in addition, potentially, to a lot of bad behavior. However, offline reinforcement learning requires the uncertainty set to directly capture the trustworthiness of the Q-function, which is a much higher bar. In practice, imperfect uncertainty sets can give rise to either overly conservative estimates, which impedes learning, or overly loose estimates, which results in exploitation of out-of-distribution actions. Of course, the relatively performance of policy constraint and uncertainty-based methods may change in the future, as the community develops better epistemic uncertainty estimation techniques or better algorithms to incorporate more suitable distribution metrics into offline RL.

Policy constraint methods do however suffer from a number of challenges. First, most of these methods fit an estimated model of the behavior policy Ο€ Ξ² ( β‹… | 𝐬 ) {\pi_{\beta}}(\cdot|\mathbf{s}) from the dataset and constrain the learned policy Ο€ ( β‹… | 𝐬 ) \pi(\cdot|\mathbf{s}) against this estimated behavior policy. This means that the performance of these algorithms is limited by the accuracy of estimation of the behavior policy, which may be hard in scenarios with highly multimodal behaviors, as is the case in practice with real-world problems. For example, if a unimodal Gaussian policy is used to model a highly multi-modal action distribution, averaging across different modes while fitting this behavior policy may actually be unable to prevent the learned policy, Ο€ \pi , from choosing out-of-distribution actions. Methods that enforce the constraint implicitly , using only samples and without explicit behavior policy estimation, are a promising way to alleviate this limitation ( Peng et al.,, 2019 ; Nair et al.,, 2020 ) .

Even when the behavior policy can be estimated exactly, a number of these methods still suffer from the undesirable effects of function approximation. When neural networks are used to represent Q-functions, undesirable effects of function approximation may prevent the Q-function from learning meaningful values even when out-of-distribution actions are controlled for in the target values. For example, when the size of the dataset is limited, approximate dynamic programming algorithms tend to overfit on the small dataset, and this error is then accumulated via Bellman backups ( Fu et al.,, 2019 ; Kumar et al., 2020a, ) . Moreover, if the dataset state-action distribution is narrow, neural network training may only provide brittle, non-generalizable solutions. Unlike online reinforcement learning, where accidental overestimation errors arising due to function approximation can be corrected via active data collection, these errors cumulatively build up and affect future iterates in an offline RL setting.

Methods that estimate a conservative or pessimistic value function ( Kumar et al., 2020b, ) present a somewhat different set of tradeoffs. While they avoid issues associated with estimating the behavior policy and can effectively avoid overestimation, they can instead suffer from underestimation and a form of overfitting: if the dataset is small, the conservative regularizer can assign values that are too low to actions that are undersampled in the dataset. Indeed, excessive pessimism may be one of the bigger issues preventing better performance on current benchmarks, and an important open question is how to dynamically modulate the degree of conservatism to balance the risks of overestimation against the downside of avoiding any unfamiliar action.

A further issue that afflicts both constraint-based and conservative methods is that, while the Q-function is never evaluated on out-of-distribution states during training, the Q-function is still affected by the training set state distribution d Ο€ Ξ² ​ ( 𝐬 ) d^{{\pi_{\beta}}}(\mathbf{s}) , and this is not accounted for in current offline learning methods. For instance, when function approximation couples the Q-value at two distinct states with very different densities under d Ο€ Ξ² ​ ( 𝐬 ) d^{{\pi_{\beta}}}(\mathbf{s}) , dynamic programming with function approximation may give rise to incorrect solutions on the state that has a lower probability d Ο€ Ξ² ​ ( 𝐬 ) d^{{\pi_{\beta}}}(\mathbf{s}) . A variant of this issue was noted in the standard RL setting, referred to as an absence of β€œcorrective feedback” by Kumar et al., 2020a () (see Kumar and Gupta, (2020) for a short summary), and such a problem may affect offline RL algorithms with function approximation more severely, since they have no control over the data distribution at all.

Another potential challenge for all of these offline approximate dynamic programming methods is that the degree of improvement beyond the behavior policy is restricted by error accumulation. Since the error in policy performance compounds with a factor that depends on 1 / ( 1 βˆ’ Ξ³ ) 2 β‰ˆ H 2 1/(1-\gamma)^{2}\approx H^{2} ( Farahmand et al.,, 2010 ; Kumar et al.,, 2019 ; Kidambi et al.,, 2020 ) , a small divergence from the behavior policy at each step can give rise to a policy that diverges away from the behavior distribution and performs very poorly. Besides impacting training, this issue can also lead to severe state distribution shift at test time, which can lead the policy to perform very poorly. Therefore, policy constraints must be strong, so as to ensure that the overall error is small. This can restrict the amount of policy improvement that can be achieved. We might expect that directly constraining the state-action marginal distribution of the policy, such as the methods explored in recent work ( Nachum et al., 2019b, ) might not suffer from the error accumulation issue, however, Kidambi et al., (2020) proved a lower-bound result showing that quadratic scaling with respect to horizon is inevitable in the worst case for any offline RL method. Moreover, as previously discussed in Section 3.5 , representing and enforcing constraints on the state-action marginal distributions themselves requires Bellman backups, which can themselves suffer from out-of-distribution actions. Besides the worst-case dependence on the horizon, an open question that still remains is is the development of constraints that can effectively trade off error accumulation and suboptimality of the learned policy in most β€œaverage”-case MDP instances, and can be easily enforced and optimized in practice via standard optimization techniques without requiring additional function approximators to fit the behavior policy.

## 5 Offline Model-Based Reinforcement Learning

The use of predictive models can be a powerful tool for enabling effective offline reinforcement learning. Since model-based reinforcement learning algorithms primarily rely on their ability to estimate T ⁑ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) T(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) via a parameterized model T ψ ​ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) T_{\psi}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) , rather than performing dynamic programming or importance sampling, they benefit from convenient and powerful supervised learning methods when fitting the model, allowing them to effectively utilize large and diverse datasets. However, like dynamic programming methods, model-based offline RL methods are not immune to the effects of distribution shift. In this section, we briefly discuss how distributional shift affects model-based reinforcement learning methods, then survey a number of works that utilize models for offline reinforcement learning, and conclude with a brief discussion of open challenges. A complete treatment of all model-based reinforcement learning work, as well as work that learns predictive models of physics but does not utilize them for control (e.g., Lerer et al., (2016) ; Battaglia et al., (2016) ), is outside the scope of this tutorial, and we focus primarily on work that performs both model-fitting and control from offline data.

### 5.1 Model Exploitation and Distribution Shift

As discussed in Section 2.1 , the model in a model-based RL algorithm can be utilized either for planning (including online, via MPC) or for training a policy. In both cases, the model must provide accurate predictions for state-action tuples that are more optimal with respect to the current task. That is, the model will be queried at 𝐬 ∼ d Ο€ ​ ( 𝐬 ) \mathbf{s}\sim d^{\pi}(\mathbf{s}) , where Ο€ \pi is either an explicit policy, or an implicit policy produced by running planning under the model. In general d Ο€ ​ ( 𝐬 ) β‰  d Ο€ Ξ² ​ ( 𝐬 ) d^{\pi}(\mathbf{s})\neq d^{\pi_{\beta}}(\mathbf{s}) , which means that the model is itself susceptible to distribution shift. In fact, the model suffers from distribution shift both in terms of the state distribution d Ο€ ​ ( 𝐬 ) d^{\pi}(\mathbf{s}) , and the action distribution Ο€ ⁑ ( 𝐚 | 𝐬 ) \pi(\mathbf{a}|\mathbf{s}) .

Since the policy (or action sequence, in the case of planning) is optimized to obtain the highest possible expected reward under the current model, this optimization process can lead to the policy exploiting the model, intentionally producing out-of-distribution states and actions at which the model T ψ ​ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) T_{\psi}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) erroneously predicts successor states 𝐬 t + 1 \mathbf{s}_{t+1} that lead to higher returns than the actual successor states that would be obtained in the real MDP. This model exploitation problem can lead to policies that produce substantially worse performance in the real MDP than what was predicted under the model. Prior work in online model-based RL has sought to address this issue primarily via uncertainty estimation, analogously to the uncertainty-based methods discussed in Section 4.4 , but this time modeling the epistemic uncertainty of the learned model T ψ ​ ( 𝐬 t + 1 | 𝐬 t , 𝐚 t ) T_{\psi}(\mathbf{s}_{t+1}|\mathbf{s}_{t},\mathbf{a}_{t}) . In low-dimensional MDPs, such uncertainty estimates can be produced by means of Bayesian models such as Gaussian processes ( Deisenroth and Rasmussen,, 2011 ) , while for higher-dimensional problems, Bayesian neural networks and bootstrap ensembles can be utilized ( Chua et al.,, 2018 ; Janner et al.,, 2019 ) . Effective uncertainty estimation is generally considered an important component of modern model-based reinforcement learning methods, for the purpose of mitigating model exploitation.

Theoretical analysis of model-based policy learning can provide bounds on the error incurred from the distributional shift due to the divergence between the learned policy Ο€ ⁑ ( 𝐚 | 𝐬 ) \pi(\mathbf{a}|\mathbf{s}) and the behavior policy Ο€ Ξ² ​ ( 𝐚 | 𝐬 ) {\pi_{\beta}}(\mathbf{a}|\mathbf{s}) ( Sun et al., 2018b, ; Luo et al.,, 2018 ; Janner et al.,, 2019 ) . This analysis resembles the distributional shift analysis provided in the DAgger example in Section 2.4 , except that now both the policy and the transition probabilities experience distributional shift. Following Janner et al., (2019) , if we assume that the total variation distance (TVD) between the learned model T ψ T_{\psi} and true model T T is bounded by Ο΅ m = max t 𝔼 d t Ο€ D T ​ V ( T ψ ( s t + 1 | s t , a t ) βˆ₯ T ( s t + 1 | s t , a t ) ) \epsilon_{m}=\max_{t}\mathbb{E}_{d_{t}^{\pi}}D_{TV}(T_{\psi}(s_{t+1}|s_{t},a_{t})\|T(s_{t+1}|s_{t},a_{t})) , and the TVD between Ο€ \pi and Ο€ Ξ² {\pi_{\beta}} is likewise bounded on sampled states by Ο΅ Ο€ \epsilon_{\pi} , then the true policy value J ⁑ ( Ο€ ) J(\pi) is related to the policy estimate under the model, J ψ ​ ( Ο€ ) J_{\psi}(\pi) , according to J ⁑ ( Ο€ ) β‰₯ J ψ ​ ( Ο€ ) βˆ’ [ 2 ​ Ξ³ ​ r max ​ ( Ο΅ m + 2 ​ Ο΅ Ο€ ) ( 1 βˆ’ Ξ³ ) 2 + 4 ​ r max ​ Ο΅ Ο€ 1 βˆ’ Ξ³ ] . J(\pi)\geq J_{\psi}(\pi)-\left[\frac{2\gamma r_{\text{max}}(\epsilon_{m}+2\epsilon_{\pi})}{(1-\gamma)^{2}}+\frac{4r_{\text{max}}\epsilon_{\pi}}{1-\gamma}\right]. Intuitively, the second term represents error accumulation due to the distribution shift in the policy, while the first term represents error accumulation due to the distribution shift in the model. The first term also includes a dependence on Ο΅ Ο€ \epsilon_{\pi} , because policies that diverge more from Ο€ Ξ² {\pi_{\beta}} will lead to states that are further outside of the data distribution, which in turn will make errors in the model more likely. Janner et al., (2019) also argue that a modified model-based RL procedure that resembles Dyna Sutton, (1991) , where only short-horizon rollouts from the model are generated by β€œbranching” off of states seen in the data, can mitigate this accumulation of error. This is also intuitively natural: if applying the learned model recursively leads to compounding error, then shorter rollouts should incur substantially less error.

### 5.2 Brief Survey of Model-Based Offline Reinforcement Learning

A natural and straightforward way to utilize model-based reinforcement learning algorithms in the offline setting is to simply train the model from offline data, with minimal modification to the algorithm. As with Q-learning and actor-critic methods, model-based reinforcement learning algorithms can be applied to the offline setting naΓ―vely. Furthermore, many effective model-based reinforcement learning methods already take steps to limit model exploitation via a variety of uncertainty estimation methods ( Deisenroth and Rasmussen,, 2011 ; Chua et al.,, 2018 ) , making them reasonably effective in the offline setting. Many such methods have been known to exhibit excellent performance in conventional off-policy settings, where additional data collection is allowed, but prior data is also utilized ( Sutton,, 1991 ; Watter et al.,, 2015 ; Zhang et al.,, 2018 ; Hafner et al.,, 2018 ; Janner et al.,, 2019 ) . Indeed, Yu et al., (2020) showed that MBPO ( Janner et al.,, 2019 ) actually attains reasonable performance on standard offline RL settings without modification, whereas naΓ―ve dynamic programming methods (e.g., soft actor-critic ( Haarnoja et al.,, 2018 ) ) fail to learn meaningful policies without policy constraints. This suggests that model-based RL algorithms are likely to lead to an effective class of offline reinforcement learning methods.

#### Offline RL with standard model-based RL methods.