Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor — part 4/4

Cleaned-text code points [47106, 50624). Bibliography omitted here; source.md includes it.

We use an unbounded Gaussian as the action distribution. However, in practice, the actions needs to be bounded to a finite interval. To that end, we apply an invertible squashing function ( tanh \tanh ) to the Gaussian samples, and employ the change of variables formula to compute the likelihoods of the bounded actions. In the other words, let 𝐮 ∈ ℝ D \mathbf{u}\in\mathbb{R}^{D} be a random variable and μ ⁡ ( 𝐮 | 𝐬 ) \mu(\mathbf{u}|\mathbf{s}) the corresponding density with infinite support. Then 𝐚 = tanh ⁡ ( 𝐮 ) \mathbf{a}=\tanh(\mathbf{u}) , where tanh \tanh is applied elementwise, is a random variable with support in ( − 1 , 1 ) (-1,1) with a density given by π ⁡ ( 𝐚 | 𝐬 ) \displaystyle\pi(\mathbf{a}|\mathbf{s}) = μ ⁡ ( 𝐮 | 𝐬 ) ​ | det ( d ​ 𝐚 d ​ 𝐮 ) | − 1 . \displaystyle=\mu(\mathbf{u}|\mathbf{s})\left|\det\left(\frac{\mathrm{d}\mathbf{a}}{\mathrm{d}\mathbf{u}}\right)\right|^{-1}. (20) Since the Jacobian d ​ 𝐚 / d ​ 𝐮 = diag ⁡ ( 1 − tanh 2 ⁡ ( 𝐮 ) ) \nicefrac{{\mathrm{d}\mathbf{a}}}{{\mathrm{d}\mathbf{u}}}=\mathrm{diag}(1-\tanh^{2}(\mathbf{u})) is diagonal, the log-likelihood has a simple form log ⁡ π ⁡ ( 𝐚 | 𝐬 ) \displaystyle\log\pi(\mathbf{a}|\mathbf{s}) = log ⁡ μ ⁡ ( 𝐮 | 𝐬 ) − ∑ i = 1 D log ⁡ ( 1 − tanh 2 ⁡ ( u i ) ) , \displaystyle=\log\mu(\mathbf{u}|\mathbf{s})-\sum_{i=1}^{D}\log\left(1-\tanh^{2}(u_{i})\right), (21) where u i u_{i} is the i th i^{\mathrm{th}} element of 𝐮 \mathbf{u} .

## Appendix D Hyperparameters

Table 1 lists the common SAC parameters used in the comparative evaluation in Figure 1 and Figure 4 . Table 2 lists the reward scale parameter that was tuned for each environment.

## Appendix E Additional Baseline Results

Figure 4 compares SAC to Trust-PCL ( Figure 4 . Trust-PC fails to solve most of the task within the given number of environment steps, although it can eventually solve the easier tasks ( Nachum et al., 2017b ) if ran longer. The figure also includes two variants of SAC: a variant that periodically copies the target value network weights directly instead of using exponentially moving average, and a deterministic ablation which assumes a deterministic policy in the value update ( Equation 6 ) and the policy update ( Equation 13 ), and thus strongly resembles DDPG with the exception of having two Q-functions, using hard target updates, not having a separate target actor, and using fixed exploration noise rather than learned. Both of these methods can learn all of the tasks and they perform comparably to SAC on all but Humanoid (rllab) task, on which SAC is the fastest.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .