Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — part 3/4
Cleaned-text code points [31242, 47106). Bibliography omitted here; source.md includes it.
The results show that, overall, SAC performs comparably to the baseline methods on the easier tasks and outperforms them on the harder tasks with a large margin, both in terms of learning speed and the final performance. For example, DDPG fails to make any progress on Ant-v1, Humanoid-v1, and Humanoid (rllab), a result that is corroborated by prior work ( Gu et al., 2016 ; Duan et al., 2016 ) . SAC also learns considerably faster than PPO as a consequence of the large batch sizes PPO needs to learn stably on more high-dimensional and complex tasks. Another maximum entropy RL algorithm, SQL, can also learn all tasks, but it is slower than SAC and has worse asymptotic performance. The quantitative results attained by SAC in our experiments also compare very favorably to results reported by other methods in prior work ( Duan et al., 2016 ; Gu et al., 2016 ; Henderson et al., 2017 ) , indicating that both the sample efficiency and final performance of SAC on these benchmark tasks exceeds the state of the art. All hyperparameters used in this experiment for SAC are listed in Appendix D .
### 5.2 Ablation Study
The results in the previous section suggest that algorithms based on the maximum entropy principle can outperform conventional RL methods on challenging tasks such as the humanoid tasks. In this section, we further examine which particular components of SAC are important for good performance. We also examine how sensitive SAC is to some of the most important hyperparameters, namely reward scaling and target value update smoothing constant.
#### Stochastic vs. deterministic policy.
Soft actor-critic learns stochastic policies via a maximum entropy objective. The entropy appears in both the policy and value function. In the policy, it prevents premature convergence of the policy variance ( Equation 10 ). In the value function, it encourages exploration by increasing the value of regions of state space that lead to high-entropy behavior ( Equation 5 ). To compare how the stochasticity of the policy and entropy maximization affects the performance, we compare to a deterministic variant of SAC that does not maximize the entropy and that closely resembles DDPG, with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using fixed rather than learned exploration noise. Figure 2 compares five individual runs with both variants, initialized with different random seeds. Soft actor-critic performs much more consistently, while the deterministic variant exhibits very high variability across seeds, indicating substantially worse stability. As evident from the figure, learning a stochastic policy with entropy maximization can drastically stabilize training. This becomes especially important with harder tasks, where tuning hyperparameters is challenging. In this comparison, we updated the target value network weights with hard updates, by periodically overwriting the target network parameters to match the current value network (see Appendix E for a comparison of average performance on all benchmark tasks).
#### Policy evaluation.
Since SAC converges to stochastic policies, it is often beneficial to make the final policy deterministic at the end for best performance. For evaluation, we approximate the maximum a posteriori action by choosing the mean of the policy distribution. Figure 3(a) compares training returns to evaluation returns obtained with this strategy indicating that deterministic evaluation can yield better performance. It should be noted that all of the training curves depict the sum of rewards, which is different from the objective optimized by SAC and other maximum entropy RL algorithms, including SQL and Trust-PCL, which maximize also the entropy of the policy.
#### Reward scale.
Soft actor-critic is particularly sensitive to the scaling of the reward signal, because it serves the role of the temperature of the energy-based optimal policy and thus controls its stochasticity. Larger reward magnitudes correspond to lower entries. Figure 3(b) shows how learning performance changes when the reward scale is varied: For small reward magnitudes, the policy becomes nearly uniform, and consequently fails to exploit the reward signal, resulting in substantial degradation of performance. For large reward magnitudes, the model learns quickly at first, but the policy then becomes nearly deterministic, leading to poor local minima due to lack of adequate exploration. With the right reward scaling, the model balances exploration and exploitation, leading to faster learning and better asymptotic performance. In practice, we found reward scale to be the only hyperparameter that requires tuning, and its natural interpretation as the inverse of the temperature in the maximum entropy framework provides good intuition for how to adjust this parameter.
#### Target network update.
It is common to use a separate target value network that slowly tracks the actual value function to improve stability. We use an exponentially moving average, with a smoothing constant τ \tau , to update the target value network weights as common in the prior work ( Lillicrap et al., 2015 ; Mnih et al., 2015 ) . A value of one corresponds to a hard update where the weights are copied directly at every iteration and zero to not updating the target at all. In Figure 3(c) , we compare the performance of SAC when τ \tau varies. Large τ \tau can lead to instabilities while small τ \tau can make training slower. However, we found the range of suitable values of τ \tau to be relatively wide and we used the same value (0.005) across all of the tasks. In Figure 4 ( Appendix E ) we also compare to another variant of SAC, where instead of using exponentially moving average, we copy over the current network weights directly into the target network every 1000 gradient steps. We found this variant to benefit from taking more than one gradient step between the environment steps, which can improve performance but also increases the computational cost.
## 6 Conclusion
We present soft actor-critic (SAC), an off-policy maximum entropy deep reinforcement learning algorithm that provides sample-efficient learning while retaining the benefits of entropy maximization and stability. Our theoretical results derive soft policy iteration, which we show to converge to the optimal policy. From this result, we can formulate a soft actor-critic algorithm, and we empirically show that it outperforms state-of-the-art model-free deep RL methods, including the off-policy DDPG algorithm and the on-policy PPO algorithm. In fact, the sample efficiency of this approach actually exceeds that of DDPG by a substantial margin. Our results suggest that stochastic, entropy maximizing reinforcement learning algorithms can provide a promising avenue for improved robustness and stability, and further exploration of maximum entropy methods, including methods that incorporate second order information (e.g., trust regions ( Schulman et al., 2015 ) ) or more expressive policy classes is an exciting avenue for future work.
## Acknowledgments
We would like to thank Vitchyr Pong for insightful discussions and help in implementing our algorithm as well as providing the DDPG baseline code; Ofir Nachum for offering support in running Trust-PCL experiments; and George Tucker for his valuable feedback on an early version of this paper. This work was supported by Siemens and Berkeley DeepDrive.
## Appendix A Maximum Entropy Objective
The exact definition of the discounted maximum entropy objective is complicated by the fact that, when using a discount factor for policy gradient methods, we typically do not discount the state distribution, only the rewards. In that sense, discounted policy gradients typically do not optimize the true discounted objective. Instead, they optimize average reward, with the discount serving to reduce variance, as discussed by Thomas (2014) . However, we can define the objective that is optimized under a discount factor as J ( π ) = ∑ t = 0 ∞ 𝔼 ( 𝐬 t , 𝐚 t ) ∼ ρ π [ ∑ l = t ∞ γ l − t 𝔼 𝐬 l ∼ p , 𝐚 l ∼ π [ r ( 𝐬 t , 𝐚 t ) + α ℋ ( π ( ⋅ | 𝐬 t ) ) | 𝐬 t , 𝐚 t ] ] . \displaystyle J(\pi)=\sum_{t=0}^{\infty}\operatorname{\mathbb{E}}_{({\mathbf{s}_{t}},{\mathbf{a}_{t}})\sim\rho_{\pi}}\left[\sum_{l=t}^{\infty}\gamma^{l-t}\operatorname{\mathbb{E}}_{\mathbf{s}_{l}\sim p,\mathbf{a}_{l}\sim\pi}\left[r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\alpha\mathcal{H}(\pi({\,\cdot\,}|{\mathbf{s}_{t}}))|{\mathbf{s}_{t}},{\mathbf{a}_{t}}\right]\right]. (14) This objective corresponds to maximizing the discounted expected reward and entropy for future states originating from every state-action tuple ( 𝐬 t , 𝐚 t ) ({\mathbf{s}_{t}},{\mathbf{a}_{t}}) weighted by its probability ρ π \rho_{\pi} under the current policy.
## Appendix B Proofs
### B.1 Lemma 1
Lemma 1 (Soft Policy Evaluation). Consider the soft Bellman backup operator 𝒯 π \mathcal{T}^{\pi} in Equation 2 and a mapping Q 0 : 𝒮 × 𝒜 → ℝ Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} with | 𝒜 | < ∞ |\mathcal{A}|<\infty , and define Q k + 1 = 𝒯 π Q k Q^{k+1}=\mathcal{T}^{\pi}Q^{k} . Then the sequence Q k Q^{k} will converge to the soft Q-value of π \pi as k → ∞ k\rightarrow\infty .
###### Proof.
Define the entropy augmented reward as r π ( 𝐬 t , 𝐚 t ) ≜ r ( 𝐬 t , 𝐚 t ) + 𝔼 𝐬 t + 1 ∼ p [ ℋ ( π ( ⋅ | 𝐬 t + 1 ) ) ] r_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\triangleq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[\mathcal{H}\left(\pi({\,\cdot\,}|{\mathbf{s}_{t+1}})\right)\right] and rewrite the update rule as Q ( 𝐬 t , 𝐚 t ) ← r π ( 𝐬 t , 𝐚 t ) + γ 𝔼 𝐬 t + 1 ∼ p , 𝐚 t + 1 ∼ π [ Q ( 𝐬 t + 1 , 𝐚 t + 1 ) ] \displaystyle Q({\mathbf{s}_{t}},{\mathbf{a}_{t}})\leftarrow r_{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p,{\mathbf{a}_{t+1}}\sim\pi}\left[Q({\mathbf{s}_{t+1}},{\mathbf{a}_{t+1}})\right] (15) and apply the standard convergence results for policy evaluation ( Sutton & Barto, 1998 ) . The assumption | 𝒜 | < ∞ |\mathcal{A}|<\infty is required to guarantee that the entropy augmented reward is bounded. ∎
### B.2 Lemma 2
Lemma 2 (Soft Policy Improvement). Let π old ∈ Π \pi_{\mathrm{old}}\in\Pi and let π new \pi_{\mathrm{new}} be the optimizer of the minimization problem defined in Equation 4 . Then Q π new ( 𝐬 t , 𝐚 t ) ≥ Q π old ( 𝐬 t , 𝐚 t ) Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) for all ( 𝐬 t , 𝐚 t ) ∈ 𝒮 × 𝒜 ({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A} with | 𝒜 | < ∞ |\mathcal{A}|<\infty .
###### Proof.
Let π old ∈ Π {\pi_{\mathrm{old}}}\in\Pi and let Q π old Q^{\pi_{\mathrm{old}}} and V π old V^{\pi_{\mathrm{old}}} be the corresponding soft state-action value and soft state value, and let π new {\pi_{\mathrm{new}}} be defined as π new ( ⋅ | 𝐬 t ) \displaystyle{\pi_{\mathrm{new}}}({\,\cdot\,}|{\mathbf{s}_{t}}) = arg min π ′ ∈ Π D KL ( π ′ ( ⋅ | 𝐬 t ) ∥ exp ( Q π old ( 𝐬 t , ⋅ ) − log Z π old ( 𝐬 t ) ) ) \displaystyle=\arg\min_{\pi^{\prime}\in\Pi}\mathrm{D_{KL}}\left(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})\;\middle\|\;\exp\left(Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\,\cdot\,})-\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right)\right) = arg min π ′ ∈ Π J π old ( π ′ ( ⋅ | 𝐬 t ) ) . \displaystyle=\arg\min_{\pi^{\prime}\in\Pi}J_{\pi_{\mathrm{old}}}(\pi^{\prime}({\,\cdot\,}|{\mathbf{s}_{t}})). (16) It must be the case that J π old ( π new ( ⋅ | 𝐬 t ) ) ≤ J π old ( π old ( ⋅ | 𝐬 t ) ) J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{new}}}({\,\cdot\,}|{\mathbf{s}_{t}}))\leq J_{\pi_{\mathrm{old}}}({\pi_{\mathrm{old}}}({\,\cdot\,}|{\mathbf{s}_{t}})) , since we can always choose π new = π old ∈ Π {\pi_{\mathrm{new}}}={\pi_{\mathrm{old}}}\in\Pi . Hence 𝔼 𝐚 t ∼ π new [ log π new ( 𝐚 t | 𝐬 t ) − Q π old ( 𝐬 t , 𝐚 t ) + log Z π old ( 𝐬 t ) ] ≤ 𝔼 𝐚 t ∼ π old [ log π old ( 𝐚 t | 𝐬 t ) − Q π old ( 𝐬 t , 𝐚 t ) + log Z π old ( 𝐬 t ) ] , \displaystyle\resizebox{22609920}{}{$\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{new}}}}\left[\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right]\leq\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{old}}}}\left[\log{\pi_{\mathrm{old}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})-Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\log Z^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}})\right]$}, (17) and since partition function Z π old Z^{\pi_{\mathrm{old}}} depends only on the state, the inequality reduces to 𝔼 𝐚 t ∼ π new [ Q π old ( 𝐬 t , 𝐚 t ) − log π new ( 𝐚 t | 𝐬 t ) ] ≥ V π old ( 𝐬 t ) . \displaystyle\operatorname{\mathbb{E}}_{{\mathbf{a}_{t}}\sim{\pi_{\mathrm{new}}}}\left[Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})-\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})\right]\geq V^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}}). (18) Next, consider the soft Bellman equation: Q π old ( 𝐬 t , 𝐚 t ) \displaystyle Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) = r ( 𝐬 t , 𝐚 t ) + γ 𝔼 𝐬 t + 1 ∼ p [ V π old ( 𝐬 t + 1 ) ] \displaystyle=r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[V^{\pi_{\mathrm{old}}}({\mathbf{s}_{t+1}})\right] ≤ r ( 𝐬 t , 𝐚 t ) + γ 𝔼 𝐬 t + 1 ∼ p [ 𝔼 𝐚 t + 1 ∼ π new [ Q π old ( 𝐬 t + 1 , 𝐚 t + 1 ) − log π new ( 𝐚 t + 1 | 𝐬 t + 1 ) ] ] \displaystyle\leq r({\mathbf{s}_{t}},{\mathbf{a}_{t}})+\gamma\operatorname{\mathbb{E}}_{{\mathbf{s}_{t+1}}\sim p}\left[\operatorname{\mathbb{E}}_{{\mathbf{a}_{t+1}}\sim{\pi_{\mathrm{new}}}}\left[Q^{\pi_{\mathrm{old}}}({\mathbf{s}_{t+1}},{\mathbf{a}_{t+1}})-\log{\pi_{\mathrm{new}}}({\mathbf{a}_{t+1}}|{\mathbf{s}_{t+1}})\right]\right] ⋮ \displaystyle\ \ \vdots ≤ Q π new ( 𝐬 t , 𝐚 t ) , \displaystyle\leq Q^{\pi_{\mathrm{new}}}({\mathbf{s}_{t}},{\mathbf{a}_{t}}), (19) where we have repeatedly expanded Q π old Q^{\pi_{\mathrm{old}}} on the RHS by applying the soft Bellman equation and the bound in Equation 18 . Convergence to Q π new Q^{\pi_{\mathrm{new}}} follows from Lemma 1 . ∎
### B.3 Theorem 1
Theorem 1 (Soft Policy Iteration). Repeated application of soft policy evaluation and soft policy improvement to any π ∈ Π \pi\in\Pi converges to a policy π ∗ \pi^{*} such that Q π ∗ ( 𝐬 t , 𝐚 t ) ≥ Q π ( 𝐬 t , 𝐚 t ) Q^{\pi^{*}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})\geq Q^{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) for all π ∈ Π \pi\in\Pi and ( 𝐬 t , 𝐚 t ) ∈ 𝒮 × 𝒜 ({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A} , assuming | 𝒜 | < ∞ |\mathcal{A}|<\infty .
###### Proof.
Let π i \pi_{i} be the policy at iteration i i . By Lemma 2 , the sequence Q π i Q^{\pi_{i}} is monotonically increasing. Since Q π Q^{\pi} is bounded above for π ∈ Π \pi\in\Pi (both the reward and entropy are bounded), the sequence converges to some π ∗ \pi^{*} . We will still need to show that π ∗ \pi^{*} is indeed optimal. At convergence, it must be case that J π ∗ ( π ∗ ( ⋅ | 𝐬 t ) ) < J π ∗ ( π ( ⋅ | 𝐬 t ) ) J_{\pi^{*}}({\pi^{*}}({\,\cdot\,}|{\mathbf{s}_{t}}))<J_{\pi^{*}}(\pi({\,\cdot\,}|{\mathbf{s}_{t}})) for all π ∈ Π \pi\in\Pi , π ≠ π ∗ \pi\neq\pi^{*} . Using the same iterative argument as in the proof of Lemma 2 , we get Q π ∗ ( 𝐬 t , 𝐚 t ) > Q π ( 𝐬 t , 𝐚 t ) Q^{\pi^{*}}({\mathbf{s}_{t}},{\mathbf{a}_{t}})>Q^{\pi}({\mathbf{s}_{t}},{\mathbf{a}_{t}}) for all ( 𝐬 t , 𝐚 t ) ∈ 𝒮 × 𝒜 ({\mathbf{s}_{t}},{\mathbf{a}_{t}})\in\mathcal{S}\times\mathcal{A} , that is, the soft value of any other policy in Π \Pi is lower than that of the converged policy. Hence π ∗ {\pi^{*}} is optimal in Π \Pi . ∎
## Appendix C Enforcing Action Bounds