A Minimalist Approach to Offline Reinforcement Learning — part 1/3
Cleaned-text code points [0, 15324). Bibliography omitted here; source.md includes it.
# A Minimalist Approach to Offline Reinforcement Learning
###### Abstract
Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.
## 1 Introduction
Traditionally, reinforcement learning (RL) is thought of as a paradigm for online learning, where the interaction between the RL agent and its environment is of fundamental concern for how the agent learns. In offline RL (historically known as batch RL), the agent learns from a fixed-sized dataset, collected by some arbitrary and possibly unknown process ( Lange et al., 2012 ) . Eliminating the need to interact with the environment is noteworthy as data collection can often be expensive, risky, or otherwise challenging, particularly in real-world applications. Consequently, offline RL enables the use of previously logged data or leveraging an expert, such as a human operator, without any of the risk associated with an untrained RL agent.
Unfortunately, the main benefit of offline RL, the lack of environment interaction, is also what makes it a challenging task. While most off-policy RL algorithms are applicable in the offline setting, they tend to under-perform due to “extrapolation error”: an error in policy evaluation, where agents tend to poorly estimate the value of state-action pairs not contained in the dataset. This in turn affects policy improvement, where agents learn to prefer out-of-distribution actions whose value has been overestimated, resulting in poor performance ( Fujimoto et al., 2019b ) . The solution class for this problem revolves around the idea that the learned policy should be kept close to the data-generating process (or behavior policy), and has been given a variety of names (such as batch-constrained ( Fujimoto et al., 2019b ) , KL-control ( Jaques et al., 2019 ) , behavior-regularized ( Wu et al., 2019 ) , or policy constraint ( Levine et al., 2020 ) ) depending on how this “closeness” is chosen to be implemented.
While there are many proposed approaches to offline RL, we remark that few are truly “simple”, and even the algorithms which claim to work with minor additions to an underlying online RL algorithm make a significant number of implementation-level adjustments. In other cases, there are unmentioned hyperparameters, or secondary components, such as generative models, which make offline RL algorithms difficult to reproduce, and even more challenging to tune. Additionally, such mixture of details slow down the run times of the algorithms, and make causal attributions of performance gains and transfers of techniques across algorithms difficult, as in the case for many online RL algorithms ( Henderson et al., 2017 ; Tucker et al., 2018 ; Engstrom et al., 2020 ; Andrychowicz et al., 2021 ; Furuta et al., 2021 ) . This motivates the need for more minimalist approaches in offline RL.
In this paper, we ask: can we make a deep RL algorithm work offline with minimal changes? We find that we can match the performance of state-of-the-art offline RL algorithms with a single adjustment to the policy update step of the TD3 algorithm ( Fujimoto et al., 2018 ) . TD3’s policy π \pi is updated with the deterministic policy gradient ( Silver et al., 2014 ) : π = argmax π 𝔼 ( s , a ) ∼ 𝒟 [ Q ( s , π ( s ) ) ] . \pi=\argmax_{\pi}\mathbb{E}_{(s,a)\sim\mathcal{D}}[Q(s,\pi(s))]. (1) Our proposed change, TD3+BC, is to simply add a behavior cloning term to regularize the policy: π = argmax π 𝔼 ( s , a ) ∼ 𝒟 [ λ Q ( s , π ( s ) ) − ( π ( s ) − a ) 2 ] , \pi=\argmax_{\pi}\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\lambda Q(s,\pi(s))-\left(\pi(s)-a\right)^{2}\right], (2) with a single hyperparameter λ \lambda to control the strength of the regularizer. This modification can be made by adjusting only a single line of code. Additionally, we remark that normalizing the states over the dataset, such that they have mean 0 0 and standard deviation 1 1 , improves the stability of the learned policy. Importantly, these are the only changes made to the underlying deep RL algorithm. To accommodate reproduciblity, all of our code is open-sourced 1 1 1 https://github.com/sfujim/TD3_BC .
We evaluate our minimal changes to the TD3 algorithm on the D4RL benchmark of continuous control tasks ( Fu et al., 2020 ) . We find that our algorithm compares favorably against many offline RL algorithms, while being significantly easier to implement and more than halving the required computation cost. The surprising effectiveness of our minimalist approach suggests that in the context of offline RL, simpler approaches have been left underexplored in favor of more elaborate algorithmic contributions.
## 2 Related Work
Although to the best of our knowledge, we are the first to use TD3 with behavior cloning (BC) for the purpose of offline RL, we remark that combining RL with BC, and other imitation learning approaches, has been previously considered by many authors.
RL + BC . With the aim of accelerating reinforcement learning from examples (known as learning from demonstrations ( Atkeson and Schaal, 1997 ) ), BC has been used as a regularization for policy optimization with DDPG ( Lillicrap et al., 2015 ; Nair et al., 2018 ; Goecks et al., 2020 ) and the natural policy gradient ( Kakade, 2001 ; Rajeswaran et al., 2017 ) , but with additional sophistication through modified replay buffers and pre-training. The most similar work to our own is a SAC+BC baseline ( Haarnoja et al., 2018 ) from Nair et al. (2020) and an unpublished course project ( Booher, ) combining PPO ( Schulman et al., 2017 ) with BC.
RL + Imitation . Other than directly using BC with RL, imitation learning has been combined with RL in a variety of manners, such as mixed with adversarial methods ( Zhu et al., 2018 ; Kang et al., 2018 ) , used for pre-training ( Pfeiffer et al., 2018 ) , modifying the replay buffer ( Večerík et al., 2017 ; Gulcehre et al., 2020 ) , adjusting the value function ( Kim et al., 2013 ; Hester et al., 2017 ) , or reward shaping ( Judah et al., 2014 ; Wu et al., 2021 ) . In all cases, these methods use demonstrations as a method for overcoming challenges in exploration or improving the learning speed of the RL agent.
Offline RL . As aforementioned, offline RL methods generally rely on some approach for “staying close” to the data. This may be implemented using an estimate of the behavior policy and then defining an explicit policy parameterization ( Fujimoto et al., 2019b ; Ghasemipour et al., 2020 ) or by using divergence regularization ( Jaques et al., 2019 ; Kumar et al., 2019 ; Wu et al., 2019 ; Siegel et al., 2020 ; Guo et al., 2021 ; Kostrikov et al., 2021 ) . Other approaches use a weighted version of BC to favor high advantage actions ( Wang et al., 2018 ; Peng et al., 2019 ; Siegel et al., 2020 ; Wang et al., 2020 ; Nair et al., 2020 ) or perform BC over an explicit subset of the data ( Chen et al., 2020 ) . Some methods have modified the set of valid actions based on counts ( Laroche et al., 2019 ) or the learned behavior policy ( Fujimoto et al., 2019a ) . Another direction is to implement divergence regularization as a form of pessimism into the value estimate ( Nachum et al., 2019 ; Kumar et al., 2020 ; Buckman et al., 2020 ) .
Meta Analyses of RL Algorithms . There are a substantial amount of meta analysis works on online RL algorithms. While some focus on inadequacies in the experimental protocols ( Henderson et al., 2017 ; Osband et al., 2019 ) , others study the roles of subtle implementation details in algorithms ( Tucker et al., 2018 ; Engstrom et al., 2020 ; Andrychowicz et al., 2021 ; Furuta et al., 2021 ) . For example, Tucker et al. (2018) ; Engstrom et al. (2020) identified that superior performances of certain algorithms were more dependent on, or even accidentally due to, minor implementation rather than algorithmic differences. Furuta et al. (2021) study two broad families of off-policy algorithms, which most offline algorithms are based on, and find that a few subtle implementation details are strongly co-adapted and critical to specific algorithms, making attributions of performance gains difficult. Recent offline research also follows a similar trend, where a number of implementation modifications are necessary for high algorithmic performances (see Table 1 ). In contrast, we derive our algorithm by modifying the existing TD3 with only a few lines of codes. Our results suggest the community could also learn from careful explorations of simpler alternatives, besides emphasizing algorithmic novelties and complexities.
## 3 Background
RL . Reinforcement learning (RL) is a framework aimed to deal with tasks of sequential nature. Typically, the problem is defined by a Markov decision process (MDP) ( 𝒮 , 𝒜 , ℛ , p , γ ) (\mathcal{S},\mathcal{A},\mathcal{R},p,\gamma) , with state space 𝒮 \mathcal{S} , action space 𝒜 \mathcal{A} , scalar reward function ℛ \mathcal{R} , transition dynamics p p , and discount factor γ \gamma ( Sutton and Barto, 1998 ) . The behavior of an RL agent is determined by a policy π \pi which maps states to actions (deterministic policy), or states to a probability distribution over actions (stochastic policy). The objective of an RL agent is to maximize the expected discounted return 𝔼 π [ ∑ t = 0 ∞ γ t r t + 1 ] \mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}] , which is the expected cumulative sum of rewards when following the policy in the MDP, where the importance of the horizon is determined by the discount factor. We measure this objective by a value function, which measures the expected discounted return after taking the action a a in state s s : Q π ( s , a ) = 𝔼 π [ ∑ t = 0 ∞ γ t r t + 1 | s 0 = s , a 0 = a ] Q^{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r_{t+1}|s_{0}=s,a_{0}=a\right] .
BC . Another approach for training policies is through imitation of an expert or behavior policy. Behavior cloning (BC) is an approach for imitation learning ( Pomerleau, 1991 ) , where the policy is trained with supervised learning to directly imitate the actions of a provided dataset. Unlike RL, this process is highly dependent on the performance of the data-collecting process.
Offline RL . Offline RL breaks the assumption that the agent can interact with the environment. Instead, the agent is provided a fixed dataset which has been collected by some unknown data-generating process (such as a collection of behavior policies). This setting may be considered more challenging as the agent loses the opportunity to explore the MDP according to its current beliefs, and instead from infer good behavior from only the provided data.
One challenge for offline RL is the problem of extrapolation error ( Fujimoto et al., 2019b ) , which is generalization error in the approximate value function, induced by selecting actions not contained in the dataset. Simply put, it is difficult to evaluate the expected value of a policy which is sufficiently different from the behavior policy. Consequently, algorithms have taken the approach of constraining or regularizing the policy to stay near to the actions in the dataset ( Levine et al., 2020 ) .
## 4 Challenges in Offline RL
In this section, we identify key open challenges in offline RL through analyzing and evaluating prior algorithms. We believe these challenges highlight the importance of minimalist approaches, where performance can be easily attributed to algorithmic contributions, rather than entangled with the specifics of implementation.
Implementation and Tuning Complexities . RL algorithms are notoriously difficult to implement and tune ( Henderson et al., 2017 ; Engstrom et al., 2020 ; Furuta et al., 2021 ) , where minor code-level optimizations and hyperparameters can have non-trivial impact of performance and stability. This problem may be additionally amplified in the context of offline RL, where evaluating changes to implementation and hyperparameters is counter-intuitive to the nature of offline RL, which explicitly aims to eliminate environment interactions ( Paine et al., 2020 ; Yang et al., 2020 ) .
Most offline RL algorithms are built explicitly on top of an existing off-policy deep RL algorithm, such as TD3 ( Fujimoto et al., 2018 ) or SAC ( Haarnoja et al., 2018 ) , but then further modify the underlying algorithm with “non-algorithmic” implementation changes, such as modifications to network architecture, learning rates, or pre-training the actor network. We remark that a desirable property of an offline RL algorithm would be to minimally modify the underlying algorithm, so as to reduce the space of possible adjustments required to achieve a strong performance.
In Table 1 we examine the particular modifications made by two recent offline RL algorithms, CQL ( Kumar et al., 2020 ) and Fisher-BRC ( Kostrikov et al., 2021 ) . On top of algorithmic changes, CQL also adds a pre-training phase where the actor is only trained with imitation learning and selects the max action over a sampled set of actions from the policy during evaluation. Fisher-BRC adds a constant reward bonus to every transition. Both methods modify SAC by removing the entropy term in the target update and modify the default network architecture. These changes add supplementary hyperparameters which may need to be tuned or increase computational costs.
In the online setting, these changes are relatively inconsequential and could be validated with some simple experimentation. However, in the offline setting, where we cannot interact with the environment, making additional adjustments to the underlying algorithm should be considered as more costly as validating their effectiveness is no longer a trivial additional step. This is also problematic because unlike the algorithmic changes proposed by these papers, these implementation details are not well justified, meaning there is much less intuition as to when to include these details, or how to adjust them with minimal experimentation. In the case of the D4RL benchmark on MuJoCo tasks ( Todorov et al., 2012 ; Fu et al., 2020 ) , we have a strong prior that our base deep RL algorithm performs well, as SAC/TD3 are considered state-of-the-art (or nearly) in these domains. If additional changes are necessary, then it suggests the algorithmic contributions alone are insufficient.