Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — part 1/4
Cleaned-text code points [0, 15512). Bibliography omitted here; source.md includes it.
# Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
###### Abstract
Model-free deep reinforcement learning (RL) algorithms have been demonstrated on a range of challenging decision making and control tasks. However, these methods typically suffer from two major challenges: very high sample complexity and brittle convergence properties, which necessitate meticulous hyperparameter tuning. Both of these challenges severely limit the applicability of such methods to complex, real-world domains. In this paper, we propose soft actor-critic, an off-policy actor-critic deep RL algorithm based on the maximum entropy reinforcement learning framework. In this framework, the actor aims to maximize expected reward while also maximizing entropy. That is, to succeed at the task while acting as randomly as possible. Prior deep RL methods based on this framework have been formulated as Q-learning methods. By combining off-policy updates with a stable stochastic actor-critic formulation, our method achieves state-of-the-art performance on a range of continuous control benchmark tasks, outperforming prior on-policy and off-policy methods. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving very similar performance across different random seeds.
###### Keywords:
## 1 Introduction
Model-free deep reinforcement learning (RL) algorithms have been applied in a range of challenging domains, from games ( Mnih et al., 2013 ; Silver et al., 2016 ) to robotic control ( Schulman et al., 2015 ) . The combination of RL and high-capacity function approximators such as neural networks holds the promise of automating a wide range of decision making and control tasks, but widespread adoption of these methods in real-world domains has been hampered by two major challenges. First, model-free deep RL methods are notoriously expensive in terms of their sample complexity. Even relatively simple tasks can require millions of steps of data collection, and complex behaviors with high-dimensional observations might need substantially more. Second, these methods are often brittle with respect to their hyperparameters: learning rates, exploration constants, and other settings must be set carefully for different problem settings to achieve good results. Both of these challenges severely limit the applicability of model-free deep RL to real-world tasks.
One cause for the poor sample efficiency of deep RL methods is on-policy learning: some of the most commonly used deep RL algorithms, such as TRPO ( Schulman et al., 2015 ) , PPO ( Schulman et al., 2017b ) or A3C ( Mnih et al., 2016 ) , require new samples to be collected for each gradient step. This quickly becomes extravagantly expensive, as the number of gradient steps and samples per step needed to learn an effective policy increases with task complexity. Off-policy algorithms aim to reuse past experience. This is not directly feasible with conventional policy gradient formulations, but is relatively straightforward for Q-learning based methods ( Mnih et al., 2015 ) . Unfortunately, the combination of off-policy learning and high-dimensional, nonlinear function approximation with neural networks presents a major challenge for stability and convergence ( Bhatnagar et al., 2009 ) . This challenge is further exacerbated in continuous state and action spaces, where a separate actor network is often used to perform the maximization in Q-learning. A commonly used algorithm in such settings, deep deterministic policy gradient (DDPG) ( Lillicrap et al., 2015 ) , provides for sample-efficient learning but is notoriously challenging to use due to its extreme brittleness and hyperparameter sensitivity ( Duan et al., 2016 ; Henderson et al., 2017 ) .
We explore how to design an efficient and stable model-free deep RL algorithm for continuous state and action spaces. To that end, we draw on the maximum entropy framework, which augments the standard maximum reward reinforcement learning objective with an entropy maximization term ( Ziebart et al., 2008 ; Toussaint, 2009 ; Rawlik et al., 2012 ; Fox et al., 2016 ; Haarnoja et al., 2017 ) . Maximum entropy reinforcement learning alters the RL objective, though the original objective can be recovered using a temperature parameter ( Haarnoja et al., 2017 ) . More importantly, the maximum entropy formulation provides a substantial improvement in exploration and robustness: as discussed by Ziebart (2010) , maximum entropy policies are robust in the face of model and estimation errors, and as demonstrated by ( Haarnoja et al., 2017 ) , they improve exploration by acquiring diverse behaviors. Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization ( O’Donoghue et al., 2016 ) , as well as off-policy methods based on soft Q-learning and its variants ( Schulman et al., 2017a ; Nachum et al., 2017a ; Haarnoja et al., 2017 ) . However, the on-policy variants suffer from poor sample complexity for the reasons discussed above, while the off-policy variants require complex approximate inference procedures in continuous action spaces.
In this paper, we demonstrate that we can devise an off-policy maximum entropy actor-critic algorithm, which we call soft actor-critic (SAC), which provides for both sample-efficient learning and stability. This algorithm extends readily to very complex, high-dimensional tasks, such as the Humanoid benchmark ( Duan et al., 2016 ) with 21 action dimensions, where off-policy methods such as DDPG typically struggle to obtain good results ( Gu et al., 2016 ) . SAC also avoids the complexity and potential instability associated with approximate inference in prior off-policy maximum entropy algorithms based on soft Q-learning ( Haarnoja et al., 2017 ) . We present a convergence proof for policy iteration in the maximum entropy framework, and then introduce a new algorithm based on an approximation to this procedure that can be practically implemented with deep neural networks, which we call soft actor-critic. We present empirical results that show that soft actor-critic attains a substantial improvement in both performance and sample efficiency over both off-policy and on-policy prior methods. We also compare to twin delayed deep deterministic (TD3) policy gradient algorithm ( Fujimoto et al., 2018 ) , which is a concurrent work that proposes a deterministic algorithm that substantially improves on DDPG.
## 2 Related Work
Our soft actor-critic algorithm incorporates three key ingredients: an actor-critic architecture with separate policy and value function networks, an off-policy formulation that enables reuse of previously collected data for efficiency, and entropy maximization to enable stability and exploration. We review prior works that draw on some of these ideas in this section. Actor-critic algorithms are typically derived starting from policy iteration, which alternates between policy evaluation —computing the value function for a policy—and policy improvement —using the value function to obtain a better policy ( Barto et al., 1983 ; Sutton & Barto, 1998 ) . In large-scale reinforcement learning problems, it is typically impractical to run either of these steps to convergence, and instead the value function and policy are optimized jointly. In this case, the policy is referred to as the actor, and the value function as the critic. Many actor-critic algorithms build on the standard, on-policy policy gradient formulation to update the actor ( Peters & Schaal, 2008 ) , and many of them also consider the entropy of the policy, but instead of maximizing the entropy, they use it as an regularizer ( Schulman et al., 2017b ; Schulman et al., 2015 ; Mnih et al., 2016 ; Gruslys et al., 2017 ) . On-policy training tends to improve stability but results in poor sample complexity.
There have been efforts to increase the sample efficiency while retaining robustness by incorporating off-policy samples and by using higher order variance reduction techniques ( O’Donoghue et al., 2016 ; Gu et al., 2016 ) . However, fully off-policy algorithms still attain better efficiency. A particularly popular off-policy actor-critic method, DDPG ( Lillicrap et al., 2015 ) , which is a deep variant of the deterministic policy gradient ( Silver et al., 2014 ) algorithm, uses a Q-function estimator to enable off-policy learning, and a deterministic actor that maximizes this Q-function. As such, this method can be viewed both as a deterministic actor-critic algorithm and an approximate Q-learning algorithm. Unfortunately, the interplay between the deterministic actor network and the Q-function typically makes DDPG extremely difficult to stabilize and brittle to hyperparameter settings ( Duan et al., 2016 ; Henderson et al., 2017 ) . As a consequence, it is difficult to extend DDPG to complex, high-dimensional tasks, and on-policy policy gradient methods still tend to produce the best results in such settings ( Gu et al., 2016 ) . Our method instead combines off-policy actor-critic training with a stochastic actor, and further aims to maximize the entropy of this actor with an entropy maximization objective. We find that this actually results in a considerably more stable and scalable algorithm that, in practice, exceeds both the efficiency and final performance of DDPG. A similar method can be derived as a zero-step special case of stochastic value gradients (SVG(0)) ( Heess et al., 2015 ) . However, SVG(0) differs from our method in that it optimizes the standard maximum expected return objective, and it does not make use of a separate value network, which we found to make training more stable.
Maximum entropy reinforcement learning optimizes policies to maximize both the expected return and the expected entropy of the policy. This framework has been used in many contexts, from inverse reinforcement learning ( Ziebart et al., 2008 ) to optimal control ( Todorov, 2008 ; Toussaint, 2009 ; Rawlik et al., 2012 ) . In guided policy search ( Levine & Koltun, 2013 ; Levine et al., 2016 ) , the maximum entropy distribution is used to guide policy learning towards high-reward regions. More recently, several papers have noted the connection between Q-learning and policy gradient methods in the framework of maximum entropy learning ( O’Donoghue et al., 2016 ; Haarnoja et al., 2017 ; Nachum et al., 2017a ; Schulman et al., 2017a ) . While most of the prior model-free works assume a discrete action space, Nachum et al. (2017b) approximate the maximum entropy distribution with a Gaussian and Haarnoja et al. (2017) with a sampling network trained to draw samples from the optimal policy. Although the soft Q-learning algorithm proposed by Haarnoja et al. (2017) has a value function and actor network, it is not a true actor-critic algorithm: the Q-function is estimating the optimal Q-function, and the actor does not directly affect the Q-function except through the data distribution. Hence, Haarnoja et al. (2017) motivates the actor network as an approximate sampler, rather than the actor in an actor-critic algorithm. Crucially, the convergence of this method hinges on how well this sampler approximates the true posterior. In contrast, we prove that our method converges to the optimal policy from a given policy class, regardless of the policy parameterization. Furthermore, these prior maximum entropy methods generally do not exceed the performance of state-of-the-art off-policy algorithms, such as DDPG, when learning from scratch, though they may have other benefits, such as improved exploration and ease of fine-tuning. In our experiments, we demonstrate that our soft actor-critic algorithm does in fact exceed the performance of prior state-of-the-art off-policy deep RL methods by a wide margin.
## 3 Preliminaries
We first introduce notation and summarize the standard and maximum entropy reinforcement learning frameworks.
### 3.1 Notation
We address policy learning in continuous action spaces. We consider an infinite-horizon Markov decision process (MDP), defined by the tuple ( 𝒮 , 𝒜 , p , r ) (\mathcal{S},\mathcal{A},p,r) , where the state space 𝒮 \mathcal{S} and the action space 𝒜 \mathcal{A} are continuous, and the unknown state transition probability p : 𝒮 × 𝒮 × 𝒜 → [ 0 , ∞ ) p:\ \mathcal{S}\times\mathcal{S}\times\mathcal{A}\rightarrow[0,\,\infty) represents the probability density of the next state 𝐬 t + 1 ∈ 𝒮 {\mathbf{s}_{t+1}}\in\mathcal{S} given the current state 𝐬 t ∈ 𝒮 {\mathbf{s}_{t}}\in\mathcal{S} and action 𝐚 t ∈ 𝒜 {\mathbf{a}_{t}}\in\mathcal{A} . The environment emits a bounded reward r : 𝒮 × 𝒜 → [ r min , r max ] r:\mathcal{S}\times\mathcal{A}\rightarrow[r_{\mathrm{min}},r_{\mathrm{max}}] on each transition. We will use ρ π ( 𝐬 t ) \rho_{\pi}({\mathbf{s}_{t}}) and ρ π ( 𝐬 t , 𝐚 t ) \rho_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) to denote the state and state-action marginals of the trajectory distribution induced by a policy π ( 𝐚 t | 𝐬 t ) \pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}}) .
### 3.2 Maximum Entropy Reinforcement Learning
Standard RL maximizes the expected sum of rewards ∑ t 𝔼 ( 𝐬 t , 𝐚 t ) ∼ ρ π [ r ( 𝐬 t , 𝐚 t ) ] \sum_{t}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})\right] . We will consider a more general maximum entropy objective (see e.g. Ziebart (2010) ), which favors stochastic policies by augmenting the objective with the expected entropy of the policy over ρ π ( 𝐬 t ) \rho_{\pi}({\mathbf{s}_{t}}) : J ( π ) = ∑ t = 0 T 𝔼 ( 𝐬 t , 𝐚 t ) ∼ ρ π [ r ( 𝐬 t , 𝐚 t ) + α ℋ ( π ( ⋅ | 𝐬 t ) ) ] . \displaystyle J(\pi)=\sum_{t=0}^{T}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\alpha\mathcal{H}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))\right]. (1) The temperature parameter α \alpha determines the relative importance of the entropy term against the reward, and thus controls the stochasticity of the optimal policy. The maximum entropy objective differs from the standard maximum expected reward objective used in conventional reinforcement learning, though the conventional objective can be recovered in the limit as α → 0 \alpha\rightarrow 0 . For the rest of this paper, we will omit writing the temperature explicitly, as it can always be subsumed into the reward by scaling it by α − 1 \alpha^{-1} .
This objective has a number of conceptual and practical advantages. First, the policy is incentivized to explore more widely, while giving up on clearly unpromising avenues. Second, the policy can capture multiple modes of near-optimal behavior. In problem settings where multiple actions seem equally attractive, the policy will commit equal probability mass to those actions. Lastly, prior work has observed improved exploration with this objective ( Haarnoja et al., 2017 ; Schulman et al., 2017a ) , and in our experiments, we observe that it considerably improves learning speed over state-of-art methods that optimize the conventional RL objective function. We can extend the objective to infinite horizon problems by introducing a discount factor γ \gamma to ensure that the sum of expected rewards and entropies is finite. Writing down the maximum entropy objective for the infinite horizon discounted case is more involved ( Thomas, 2014 ) and is deferred to Appendix A .