A Minimalist Approach to Offline Reinforcement Learning — part 2/3
Cleaned-text code points [15324, 31056). Bibliography omitted here; source.md includes it.
In Figure 1 we examine the percent difference in performance when removing the implementation changes in CQL and Fisher-BRC. There is a significant drop in performance in many of the tasks. This is not necessarily a death knell to these algorithms, as it is certainly possible these changes could be kept without tuning when attempting new datasets and domains. However, since neither paper make mention of a training/validation split, we can only assume these changes were made with their evaluation datasets in mind (D4RL, in this instance), and remark there is insufficient evidence that these changes may be universal. Ultimately, we make this point not to suggest a fundamental flaw with pre-existing algorithms, but to suggest that there should be a preference for making minimal adjustments to the underlying RL algorithm, to reduce the need for hyperparameter tuning.
Extra Computation Requirement . A secondary motivating factor for minimalism is avoiding the additional computational costs associated with modifying the underlying algorithm (in particular architecture) and more complex algorithmic ideas. In Table 3 (contained later in Section 6 ), we examine the run time of offline RL algorithms, as well as the change in cost over the underlying algorithm, and find their is a significant computational cost associated with these modifications. On top of the architecture changes, for CQL this is largely due to the costs of logsumexp over multiple sampled actions, and for Fisher-BRC, the costs associated with training the independent generative model. Since an objective of offline RL is to take advantage of existing, potentially large, datasets, there should be a preference for scalable and efficient solutions. Of course, run time should not come at the cost of performance, but as we will later demonstrate, there exists a simple, and computationally-free, approach for offline RL which matches the performance of current state-of-the-art algorithms.
Instability of Trained Policies . In analyzing the final trained policies of prior offline algorithms, we learned of a tangential, and open, challenge in the form of instability. In online RL, if the current policy is unsatisfactory, we can use checkpoints of previous iterations of the policy, or to simply continue training. However, in offline RL, the evaluation should only occur once by definition, greatly increasing the importance of the single policy at evaluation time. We highlight two versions of instability in offline RL. Figure 2 shows that in contrast to the online-trained policy, which converges to a robust low-variance policy, the offline-trained policy exhibits huge variances in performance during a single evaluation. Therefore, even if the average performance is reasonable, the agent may still perform poorly on some episodes. Figure 3 shows instability over the set of evaluations, which means the performance of the agent may be dependent on the specific stopping point chosen for evaluation. This questions the empirical effectiveness of offline RL for safety-critical real-world use cases ( Mandel et al., 2014 ; Gottesman et al., 2018 ; Gauci et al., 2018 ; Jaques et al., 2019 ; Matsushima et al., 2020 ) as well as the current trend of reporting only the mean-value of the final policy in offline benchmarking ( Fu et al., 2020 ) .
Such variances are not likely caused by high policy entropies (e.g. TD3+BC trains a deterministic policy), but rather our hypothesis is that such a problem is due to distributional shifts issues and poor generalizations across unobserved states caused by offline nature of training ( Ross et al., 2011 ) , where the optimized policy is never allowed to execute in the environment, similarly as in BC. This trait of offline RL algorithms appears to be consistent across all offline algorithms we evaluated, even for our minimalistic TD3+BC that is only a few lines change from TD3. While we could not solve this challenge sufficiently within the scope of this work, the fact that this is reproducible even in the minimalistic variant proves that this a fundamental problem shared by all offline training settings, and is a critical problem for the community to study in the future.
## 5 A Minimalist Offline RL Algorithm
A key problem in offline RL, extrapolation error, can be summarized as the inability to properly evaluate out-of-distribution actions. Consequently, there has been a variety of different approaches to limiting, or regularizing, action selection such that the learned policy is easier to evaluate with the given dataset. We remark that while minimizing say, KL divergence, is a both logical and valid approach for reducing extrapolation error, there is no fundamental argument why minimizing one divergence or distance metric should be better than another. Thus, rather than derive an entirely new approach, we focus on simplicity, and present an offline RL algorithm which requires minimal modifications to a pre-existing deep RL algorithm. As discussed in Section 4 a minimalist approach has a variety of benefits, such as reducing the number of hyperparameters to tune, increasing scalability by reducing computational costs, and providing an avenue for analyzing problems by disentangling algorithmic contributions from implementation details.
We now describe such an approach to offline RL. Our algorithm builds on top of TD3 ( Fujimoto et al., 2018 ) , making only two straightforward changes. Firstly, we add a behavior cloning regularization term to the standard policy update step of TD3, to push the policy towards favoring actions contained in the dataset 𝒟 \mathcal{D} : π = argmax π 𝔼 s ∼ 𝒟 [ Q ( s , π ( s ) ) ] → π = argmax π 𝔼 ( s , a ) ∼ 𝒟 [ λ Q ( s , π ( s ) ) − ( π ( s ) − a ) 2 ] . \pi=\argmax_{\pi}\mathbb{E}_{s\sim\mathcal{D}}\left[Q(s,\pi(s))\right]\rightarrow\pi=\argmax_{\pi}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\hbox{\pagecolor{mine}$\lambda$}Q(s,\pi(s))-\hbox{\pagecolor{mine}$\left(\pi(s)-a\right)^{2}$}\right]. (3) Secondly, we normalize the features of every state in the provided dataset. Let s i s_{i} be the i i th feature of the state s s , let μ i \mu_{i} σ i \sigma_{i} be the mean and standard deviation, respectively, of the i i th feature across the dataset: s i = s i − μ i σ i + ϵ , s_{i}=\frac{s_{i}-\mu_{i}}{\sigma_{i}+\epsilon}, (4) where ϵ \epsilon is a small normalization constant (we use 10 − 3 10^{-3} ). While we remark this is a commonly used implementation detail in many deep RL algorithms ( Raffin et al., 2019 ) , we highlight it as (1) we want complete transparency about all implementation changes and (2) normalizing provides a non-trivial performance benefit in offline RL, where it is particularly well-suited as the dataset remains fixed.
While the choice of λ \lambda in Equation 3 is ultimately just a hyperparameter, we observe that the balance between RL (in maximizing Q Q ) and imitation (in minimizing the BC term), is highly susceptible to the scale of Q Q . If we assume an action range of [ − 1 , 1 ] [-1,1] , the BC term is at most 4 4 , however the range of Q Q will be a function of the scale of the reward. Consequently, we can add a normalization term into λ \lambda . Given the dataset of N N transitions ( s i , a i ) (s_{i},a_{i}) , we define the scalar λ \lambda as: λ = α 1 N ∑ ( s i , a i ) | Q ( s i , a i ) | . \lambda=\frac{\alpha}{\frac{1}{N}\sum_{(s_{i},a_{i})}|Q(s_{i},a_{i})|}. (5) This is simply a normalization term based on the average absolute value of Q Q . In practice, we estimate this mean term over mini-batches, rather than the entire dataset. Although this term includes Q Q , it is not differentiated over, and is simply used to scale the loss. This formulation has the added benefit of normalizing the learning rate across tasks, as the gradient ∇ a Q ( s , a ) \nabla_{a}Q(s,a) will also be dependent on the scale of Q Q . We use α = 2.5 \alpha=2.5 in our experiments.
This completes the description of TD3+BC. The Equations 3 , 4 , and 5 summarizes the entirety of our changes to TD3, and can be implemented by modifying only a handful of lines in most codebases.
## 6 Experiments
We evaluate our proposed approach on the D4RL benchmark of OpenAI gym MuJoCo tasks ( Todorov et al., 2012 ; Brockman et al., 2016 ; Fu et al., 2020 ) , which encompasses a variety of dataset settings and domains. Our offline RL baselines include two state-of-the-art algorithms, CQL ( Kumar et al., 2020 ) and Fisher-BRC ( Kostrikov et al., 2021 ) , as well as BRAC ( Wu et al., 2019 ) and AWAC ( Nair et al., 2020 ) due to their algorithmic simplicity.
BC CQL Fisher-BRC TD3+BC
To ensure a fair and identical experimental evaluation across algorithms, we re-run the state-of-the-art algorithms CQL and Fisher-BRC using the author-provided implementations 2 2 2 https://github.com/aviralkumar2907/CQL 3 3 3 https://github.com/google-research/google-research/tree/master/fisher_brc . We train each algorithm for 1 million time steps and evaluate every 5000 time steps. Each evaluation consists of 10 episodes. Results for BRAC are obtained from the D4RL benchmark and from the CQL paper, while the AWAC results are obtained directly from the paper. BC results were obtained using our own implementation.
D4RL. We report the final performance results in Table 2 and display the learning curves in Figure 4 . Although our method is very simplistic in nature, it surpasses, or matches, the performance of the current state-of-the-art offline RL algorithms in most tasks. Only Fisher-BRC exhibits a comparable performance 4 4 4 As noted by previous papers ( Kostrikov et al., 2021 ) , we remark the author-provided implementation of CQL achieves a lower result than their reported results. . Examining the learning curves, we can see that our approach achieves a similar learning speed and stability, without requiring any pre-training phase.
Run time. We evaluate run time of training each of the offline RL algorithms for 1 million time steps, using the author-provoided implementations. Additionally, for fair comparison, we re-implement Fisher-BRC, allowing each method to be compared in the same framework (PyTorch ( Paszke et al., 2019 ) ). The results are reported in Table 3 . Unsurprisingly, we find our approach compares favorably against previous methods in terms of wall-clock training time, effectively adding no cost to the underlying TD3 algorithm. All run time experiments were run with a single GeForce GTX 1080 GPU and an Intel Core i7-6700K CPU at 4.00GHz.
Ablation. In Figure 5 , we perform an ablation study over the components in our method. As noted in previous work ( Fujimoto et al., 2019b ) , without behavior cloning regularization, the RL algorithm alone is insufficient to achieve a high performance (except on some of the random data sets). We also note that our algorithm never underperforms vanilla behavior cloning, even on the expert tasks. Predictably, the removal of state normalization has the least significant impact, but it still provides some benefit across a range of tasks while being a minor adjustment. In Figure 6 , we evaluate the sensitivity of the algorithm to the hyperparameter α \alpha , where the weighting λ = α ∑ ( s , a ) | Q ( s , a ) | \lambda=\frac{\alpha}{\sum_{(s,a)}|Q(s,a)|} on the value function is determined by α \alpha . On many tasks, our approach is robust to this weighting factor, but note the performance on a subset of tasks begins to decrease as α \alpha begins to more heavily favor imitation ( α = 1 \alpha=1 ) or RL ( α = 4 \alpha=4 ).
## 7 Conclusion
Most recent advances in offline RL center around the idea of regularizing policy actions to be close to the support within batch data, and yet many state-of-the-art algorithms have significant complexities and additional modifications beyond base algorithms that lead to not only much slower run time, but also intractable attributions for sources of performance gains. Instead of complexity we optimize for simplicity, and introduce a minimalistic algorithm that achieves a state-of-the-art performance but is only a few lines of changes from the base TD3 algorithm, uses less than half of the computation time of competing algorithms, and has only one additional hyperparameter.
Additionally, we highlight existing open challenges in offline RL research, including not only the extra implementation, computation, and hyperparameter-tuning complexities that we successfully address in this work, but also call attention to the neglected problem of high episodic variance in offline-trained policies compared to online-trained (see Figures 2 and 3 ) that we as the community should address in future works and benchmarking. Finally, we believe the sheer simplicity of our approach highlights a possible overemphasis on algorithmic complexity made by the community, and we hope to inspire future work to revisit simpler alternatives which may have been overlooked.
## Appendix A Broader Impact
Offline RL will have societal impact by enabling new applications for reinforcement learning which can benefit from offline logged data such as robotics or healthcare applications, where collecting data is difficult, time-gated, expensive, etc. This may include potentially negative applications such as enforcing addictive behavior on social media. Another limitation to offline RL is that it is subject to any biases contained in the data set and can influenced by the data-generating policy.
For our specific algorithm, TD3+BC, given the performance gain over existing state-of-the-art methods is minimal, it would be surprising to see our paper result in significant impact in these contexts. However, where we might see impact is in enabling new users access to offline RL by reducing the computational cost, or burden of implementation, from having a simpler approach to offline RL. In other words, we foresee the impact our work is in accessibility and ease-of-use, and not through changing the scope of possible applications.
## Appendix B Experimental Details
Software . We use the following software versions: • Python 3.6
• Pytorch 1.4.0 [ Paszke et al., 2019 ]
• Tensorflow 2.4.0 [ Abadi et al., 2016 ]
• Gym 0.17.0 [ Brockman et al., 2016 ]
• MuJoCo 1.50 5 5 5 License information: https://www.roboti.us/license.html [ Todorov et al., 2012 ]
• mujoco-py 1.50.1.1
All D4RL datasets [ Fu et al., 2020 ] use the v0 version.
Hyperparameters . Our implementations of TD3 6 6 6 https://github.com/sfujim/TD3 , commit 6a9f76101058d674518018ffbb532f5a652c1d37 [ Fujimoto et al., 2018 ] , CQL 7 7 7 https://github.com/aviralkumar2907/CQL , commit d67dbe9cf5d2b96e3b462b6146f249b3d6569796 [ Kumar et al., 2020 ] , and Fisher-BRC 8 8 8 https://github.com/google-research/google-research/tree/master/fisher_brc , commit 9898bb462a79e727ca5f413dba0ac3c4ee48d6c0 [ Kostrikov et al., 2021 ] are based off of their respective author-provided implementations from GitHub. For TD3+BC, only α \alpha was tuned in the range ( 1 , 2 , 2.5 , 3 , 4 ) (1,2,2.5,3,4) on Hopper-medium-v0 and Hopper-expert-v0 on a single seed which was unused in final reported results. We use default hyperparameters according to each GitHub whenever possible. For CQL we modify the GitHub defaults for the actor learning rate and use a fixed α \alpha rather than the Lagrange variant, matching the hyperparameters defined in their paper (which differs from the GitHub), as we found the original hyperparameters performed better. Our re-implementation of Fisher-BRC (in PyTorch rather than Tensorflow) is used only for run time experiments.
We outline the hyperparameters used by TD3+BC in Table 4 , CQL in Table 5 , and Fisher-BRC in Table 6 .