Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — part 2/4
Cleaned-text code points [15512, 31242). Bibliography omitted here; source.md includes it.
Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered ( Ziebart et al., 2008 ; Fox et al., 2016 ; Haarnoja et al., 2017 ) . We will discuss how we can devise a soft actor-critic algorithm through a policy iteration formulation, where we instead evaluate the Q-function of the current policy and update the policy through an off-policy gradient update. Though such algorithms have previously been proposed for conventional reinforcement learning, our method is, to our knowledge, the first off-policy actor-critic method in the maximum entropy reinforcement learning framework.
## 4 From Soft Policy Iteration to Soft Actor-Critic
Our off-policy soft actor-critic algorithm can be derived starting from a maximum entropy variant of the policy iteration method. We will first present this derivation, verify that the corresponding algorithm converges to the optimal policy from its density class, and then present a practical deep reinforcement learning algorithm based on this theory.
### 4.1 Derivation of Soft Policy Iteration
We will begin by deriving soft policy iteration, a general algorithm for learning optimal maximum entropy policies that alternates between policy evaluation and policy improvement in the maximum entropy framework. Our derivation is based on a tabular setting, to enable theoretical analysis and convergence guarantees, and we extend this method into the general continuous setting in the next section. We will show that soft policy iteration converges to the optimal policy within a set of policies which might correspond, for instance, to a set of parameterized densities.
In the policy evaluation step of soft policy iteration, we wish to compute the value of a policy π \pi according to the maximum entropy objective in Equation 1 . For a fixed policy, the soft Q-value can be computed iteratively, starting from any function Q : 𝒮 × 𝒜 → ℝ Q:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} and repeatedly applying a modified Bellman backup operator 𝒯 π \mathcal{T}^{\pi} given by 𝒯 π Q ( 𝐬 t , 𝐚 t ) ≜ r ( 𝐬 t , 𝐚 t ) + γ 𝔼 𝐬 t + 1 ∼ p [ V ( 𝐬 t + 1 ) ] , \displaystyle\mathcal{T}^{\pi}Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})\triangleq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[V({\mathbf{s}_{t+1}})\right], (2) where V ( 𝐬 t ) = 𝔼 𝐚 t ∼ π [ Q ( 𝐬 t , 𝐚 t ) − log π ( 𝐚 t | 𝐬 t ) ] \displaystyle V({\mathbf{s}_{t}})=\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi}\left[Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log\pi({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right] (3) is the soft state value function. We can obtain the soft value function for any policy π \pi by repeatedly applying 𝒯 π \mathcal{T}^{\pi} as formalized below.
###### Lemma 1 (Soft Policy Evaluation) .
Consider the soft Bellman backup operator 𝒯 π \mathcal{T}^{\pi} in Equation 2 and a mapping Q 0 : 𝒮 × 𝒜 → ℝ Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} with | 𝒜 | < ∞ |\mathcal{A}|<\infty , and define Q k + 1 = 𝒯 π Q k Q^{k+1}=\mathcal{T}^{\pi}Q^{k} . Then the sequence Q k Q^{k} will converge to the soft Q-value of π \pi as k → ∞ k\rightarrow\infty .
###### Proof.
See Appendix B.1 . ∎
In the policy improvement step, we update the policy towards the exponential of the new Q-function. This particular choice of update can be guaranteed to result in an improved policy in terms of its soft value. Since in practice we prefer policies that are tractable, we will additionally restrict the policy to some set of policies Π \Pi , which can correspond, for example, to a parameterized family of distributions such as Gaussians. To account for the constraint that π ∈ Π \pi\in\Pi , we project the improved policy into the desired set of policies. While in principle we could choose any projection, it will turn out to be convenient to use the information projection defined in terms of the Kullback-Leibler divergence. In the other words, in the policy improvement step, for each state, we update the policy according to π new = arg min π ′ ∈ Π D KL ( π ′ ( ⋅ | 𝐬 t ) ∥ exp ( Q π old ( 𝐬 t , ⋅ ) ) Z π old ( 𝐬 t ) ) . \displaystyle\pi_{\mathrm{new}}=\arg\underset{\pi^{\prime}\in\Pi}{\min}\mathrm{D_{KL}}\left(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})\;\middle\|\;\frac{\exp\left(Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\,\cdot\,})\right)}{Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})}\right). (4) The partition function Z π old ( 𝐬 t ) Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}}) normalizes the distribution, and while it is intractable in general, it does not contribute to the gradient with respect to the new policy and can thus be ignored, as noted in the next section. For this projection, we can show that the new, projected policy has a higher value than the old policy with respect to the objective in Equation 1 . We formalize this result in Lemma 2 .
###### Lemma 2 (Soft Policy Improvement) .
Let π old ∈ Π \pi_{\mathrm{old}}\in\Pi and let π new \pi_{\mathrm{new}} be the optimizer of the minimization problem defined in Equation 4 . Then Q π new ( 𝐬 t , 𝐚 t ) ≥ Q π old ( 𝐬 t , 𝐚 t ) Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) for all ( 𝐬 t , 𝐚 t ) ∈ 𝒮 × 𝒜 ({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A} with | 𝒜 | < ∞ |\mathcal{A}|<\infty .
###### Proof.
See Appendix B.2 . ∎
The full soft policy iteration algorithm alternates between the soft policy evaluation and the soft policy improvement steps, and it will provably converge to the optimal maximum entropy policy among the policies in Π \Pi ( Theorem 1 ). Although this algorithm will provably find the optimal solution, we can perform it in its exact form only in the tabular case. Therefore, we will next approximate the algorithm for continuous domains, where we need to rely on a function approximator to represent the Q-values, and running the two steps until convergence would be computationally too expensive. The approximation gives rise to a new practical algorithm, called soft actor-critic.
###### Theorem 1 (Soft Policy Iteration) .
Repeated application of soft policy evaluation and soft policy improvement from any π ∈ Π \pi\in\Pi converges to a policy π ∗ \pi^{*} such that Q π ∗ ( 𝐬 t , 𝐚 t ) ≥ Q π ( 𝐬 t , 𝐚 t ) Q^{\pi^{*}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) for all π ∈ Π \pi\in\Pi and ( 𝐬 t , 𝐚 t ) ∈ 𝒮 × 𝒜 ({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A} , assuming | 𝒜 | < ∞ |\mathcal{A}|<\infty .
###### Proof.
See Appendix B.3 . ∎
### 4.2 Soft Actor-Critic
As discussed above, large continuous domains require us to derive a practical approximation to soft policy iteration. To that end, we will use function approximators for both the Q-function and the policy, and instead of running evaluation and improvement to convergence, alternate between optimizing both networks with stochastic gradient descent. We will consider a parameterized state value function V ψ ( 𝐬 t ) V_{\psi}({\mathbf{s}_{t}}) , soft Q-function Q θ ( 𝐬 t , 𝐚 t ) Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) , and a tractable policy π ϕ ( 𝐚 t | 𝐬 t ) \pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}}) . The parameters of these networks are ψ , θ {\psi},\ \theta , and ϕ {\phi} . For example, the value functions can be modeled as expressive neural networks, and the policy as a Gaussian with mean and covariance given by neural networks. We will next derive update rules for these parameter vectors.
The state value function approximates the soft value. There is no need in principle to include a separate function approximator for the state value, since it is related to the Q-function and policy according to Equation 3 . This quantity can be estimated from a single action sample from the current policy without introducing a bias, but in practice, including a separate function approximator for the soft value can stabilize training and is convenient to train simultaneously with the other networks. The soft value function is trained to minimize the squared residual error J V ( ψ ) = 𝔼 𝐬 t ∼ 𝒟 [ 1 2 ( V ψ ( 𝐬 t ) − 𝔼 𝐚 t ∼ π ϕ [ Q θ ( 𝐬 t , 𝐚 t ) − log π ϕ ( 𝐚 t | 𝐬 t ) ] ) 2 ] J_{V}({\psi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D}}\left[\frac{1}{2}\left(V_{\psi}({\mathbf{s}_{t}})-\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim\pi_{\phi}}\left[Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right]\right)^{2}\right]\,
where 𝒟 \mathcal{D} is the distribution of previously sampled states and actions, or a replay buffer. The gradient of Equation 5 can be estimated with an unbiased estimator
∇ ^ ψ J V ( ψ ) = ∇ ψ V ψ ( 𝐬 t ) ( V ψ ( 𝐬 t ) − Q θ ( 𝐬 t , 𝐚 t ) + log π ϕ ( 𝐚 t | 𝐬 t ) ) , \hat{\nabla}_{\psi}J_{V}({\psi})=\nabla_{\psi}V_{\psi}({\mathbf{s}_{t}})\left(V_{\psi}({\mathbf{s}_{t}})-Q_{\theta}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log\pi_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right),
where the actions are sampled according to the current policy, instead of the replay buffer. The soft Q-function parameters can be trained to minimize the soft Bellman residual
with
which again can be optimized with stochastic gradients
The update makes use of a target value network V ψ ¯ V_{\bar{\psi}} , where ψ ¯ \bar{\psi} can be an exponentially moving average of the value network weights, which has been shown to stabilize training ( Mnih et al., 2015 ) . Alternatively, we can update the target weights to match the current value function weights periodically (see Appendix E ). Finally, the policy parameters can be learned by directly minimizing the expected KL-divergence in Equation 4 :
There are several options for minimizing J π J_{\pi} . A typical solution for policy gradient methods is to use the likelihood ratio gradient estimator ( Williams, 1992 ) , which does not require backpropagating the gradient through the policy and the target density networks. However, in our case, the target density is the Q-function, which is represented by a neural network an can be differentiated, and it is thus convenient to apply the reparameterization trick instead, resulting in a lower variance estimator. To that end, we reparameterize the policy using a neural network transformation
where ϵ t \epsilon_{t} is an input noise vector, sampled from some fixed distribution, such as a spherical Gaussian. We can now rewrite the objective in Equation 10 as
J π ( ϕ ) = 𝔼 𝐬 t ∼ 𝒟 , ϵ t ∼ 𝒩 [ log π ϕ ( f ϕ ( ϵ t , 𝐬 t ) | 𝐬 t ) − Q θ ( 𝐬 t , f ϕ ( ϵ t , 𝐬 t ) ) ] , J_{\pi}({\phi})=\operatorname{\mathbb{E}}_{{\mathbf{s}_{t}}\sim\mathcal{D},\epsilon_{t}\sim\mathcal{N}}\left[\log\pi_{\phi}(f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}})|{\mathbf{s}_{t}})-Q_{\theta}({\mathbf{s}_{t}},f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}))\right],
where π ϕ \pi_{\phi} is defined implicitly in terms of f ϕ f_{\phi} , and we have noted that the partition function is independent of ϕ {\phi} and can thus be omitted. We can approximate the gradient of Equation 12 with
where 𝐚 t {\mathbf{a}_{t}} is evaluated at f ϕ ( ϵ t , 𝐬 t ) f_{\phi}(\epsilon_{t};{\mathbf{s}_{t}}) . This unbiased gradient estimator extends the DDPG style policy gradients ( Lillicrap et al., 2015 ) to any tractable stochastic policy.
Our algorithm also makes use of two Q-functions to mitigate positive bias in the policy improvement step that is known to degrade performance of value based methods ( Hasselt, 2010 ; Fujimoto et al., 2018 ) . In particular, we parameterize two Q-functions, with parameters θ i \theta_{i} , and train them independently to optimize J Q ( θ i ) J_{Q}(\theta_{i}) . We then use the minimum of the Q-functions for the value gradient in Equation 6 and policy gradient in Equation 13 , as proposed by Fujimoto et al. (2018) . Although our algorithm can learn challenging tasks, including a 21-dimensional Humanoid, using just a single Q-function, we found two Q-functions significantly speed up training, especially on harder tasks. The complete algorithm is described in Algorithm 1 . The method alternates between collecting experience from the environment with the current policy and updating the function approximators using the stochastic gradients from batches sampled from a replay buffer. In practice, we take a single environment step followed by one or several gradient steps (see Appendix D for all hyperparameter). Using off-policy data from a replay buffer is feasible because both value estimators and the policy can be trained entirely on off-policy data. The algorithm is agnostic to the parameterization of the policy, as long as it can be evaluated for any arbitrary state-action tuple.
## 5 Experiments
The goal of our experimental evaluation is to understand how the sample complexity and stability of our method compares with prior off-policy and on-policy deep reinforcement learning algorithms. We compare our method to prior techniques on a range of challenging continuous control tasks from the OpenAI gym benchmark suite ( Brockman et al., 2016 ) and also on the rllab implementation of the Humanoid task ( Duan et al., 2016 ) . Although the easier tasks can be solved by a wide range of different algorithms, the more complex benchmarks, such as the 21-dimensional Humanoid (rllab), are exceptionally difficult to solve with off-policy algorithms ( Duan et al., 2016 ) . The stability of the algorithm also plays a large role in performance: easier tasks make it more practical to tune hyperparameters to achieve good results, while the already narrow basins of effective hyperparameters become prohibitively small for the more sensitive algorithms on the hardest benchmarks, leading to poor performance ( Gu et al., 2016 ) .
We compare our method to deep deterministic policy gradient (DDPG) ( Lillicrap et al., 2015 ) , an algorithm that is regarded as one of the more efficient off-policy deep RL methods ( Duan et al., 2016 ) ; proximal policy optimization (PPO) ( Schulman et al., 2017b ) , a stable and effective on-policy policy gradient algorithm; and soft Q-learning (SQL) ( Haarnoja et al., 2017 ) , a recent off-policy algorithm for learning maximum entropy policies. Our SQL implementation also includes two Q-functions, which we found to improve its performance in most environments. We additionally compare to twin delayed deep deterministic policy gradient algorithm (TD3) ( Fujimoto et al., 2018 ) , using the author-provided implementation. This is an extension to DDPG, proposed concurrently to our method, that first applied the double Q-learning trick to continuous control along with other improvements. We have included trust region path consistency learning (Trust-PCL) ( Nachum et al., 2017b ) and two other variants of SAC in Appendix E . We turned off the exploration noise for evaluation for DDPG and PPO. For maximum entropy algorithms, which do not explicitly inject exploration noise, we either evaluated with the exploration noise (SQL) or use the mean action (SAC). The source code of our SAC implementation 1 1 1 github.com/haarnoja/sac and videos 2 2 2 sites.google.com/view/soft-actor-critic are available online.
### 5.1 Comparative Evaluation
Figure 1 shows the total average return of evaluation rollouts during training for DDPG, PPO, and TD3. We train five different instances of each algorithm with different random seeds, with each performing one evaluation rollout every 1000 environment steps. The solid curves corresponds to the mean and the shaded region to the minimum and maximum returns over the five trials.