A Distributional Perspective on Reinforcement Learning — part 6/6

Cleaned-text code points [77024, 82745). Bibliography omitted here; source.md includes it.

While our training regime closely follows that of DQN ( Mnih et al., 2015 ) , we use Adam ( Kingma & Ba, 2015 ) instead of RMSProp ( Tieleman & Hinton, 2012 ) for gradient rescaling. We also performed some hyperparameter tuning for our final results. Specifically, we evaluated two hyperparameters over our five training games and choose the values that performed best. The hyperparameter values we considered were V max ∈ { 3 , 10 , 100 } V_{\textsc{max}}\in\{3,10,100\} and ϵ a ​ d ​ a ​ m ∈ { 1 / L , 0.1 / L , 0.01 / L , 0.001 / L , 0.0001 / L } \epsilon_{adam}\in\{1/L,0.1/L,0.01/L,0.001/L,0.0001/L\} , where L = 32 L=32 is the minibatch size. We found V max = 10 V_{\textsc{max}}=10 and ϵ a ​ d ​ a ​ m = 0.01 / L \epsilon_{adam}=0.01/L performed best. We used the same step-size value as DQN ( α = 0.00025 \alpha=0.00025 ).

Pseudo-code for the categorical algorithm is given in Algorithm 1 . We apply the Bellman update to each atom separately, and then project it into the two nearest atoms in the original support. Transitions to a terminal state are handled with γ t = 0 \gamma_{t}=0 .

## Appendix D Comparison of Sampled Wasserstein Loss and Categorical Projection

Lemma 3 proves that for a fixed policy π \pi the distributional Bellman operator is a γ \gamma -contraction in d ¯ p \bar{d}_{p} , and therefore that 𝒯 π \mathcal{T}^{\pi} will converge in distribution to the true distribution of returns Z π Z^{\pi} . In this section, we empirically validate these results on the CliffWalk domain shown in Figure 11 . The dynamics of the problem match those given by Sutton & Barto (1998) . We also study the convergence of the distributional Bellman operator under the sampled Wasserstein loss and the categorical projection (Equation 7 ) while following a policy that tries to take the safe path but has a 10% chance of taking another action uniformly at random.

We compute a ground-truth distribution of returns Z π Z^{\pi} using 10000 10000 Monte-Carlo (MC) rollouts from each state. We then perform two experiments, approximating the value distribution at each state with our discrete distributions.

In the first experiment, we perform supervised learning using either the Wasserstein loss or categorical projection (Equation 7 ) with cross-entropy loss. We use Z π Z^{\pi} as the supervised target and perform 5000 5000 sweeps over all states to ensure both approaches have converged. In the second experiment, we use the same loss functions, but the training target comes from the one-step distributional Bellman operator with sampled transitions. We use V min = − 100 V_{\textsc{min}}=-100 and V max = − 1 V_{\textsc{max}}=-1 . 4 4 4 Because there is a small probability of larger negative returns, some approximation error is unavoidable. However, this effect is relatively negligible in our experiments. For the sample updates we perform 10 times as many sweeps over the state space. Fundamentally, these experiments investigate how well the two training regimes (minimizing the Wasserstein or categorical loss) minimize the Wasserstein metric under both ideal (supervised target) and practical (sampled one-step Bellman target) conditions.

In Figure 10 a we show the final Wasserstein distance d 1 ​ ( Z π , Z θ ) d_{1}(Z^{\pi},Z_{\theta}) between the learned distributions and the ground-truth distribution as we vary the number of atoms. The graph shows that the categorical algorithm does indeed minimize the Wasserstein metric in both the supervised and sample Bellman setting. It also highlights that minimizing the Wasserstein loss with stochastic gradient descent is in general flawed, confirming the intuition given by Proposition 5 . In repeat experiments the process converged to different values of d 1 ​ ( Z π , Z θ ) d_{1}(Z^{\pi},Z_{\theta}) , suggesting the presence of local minima (more prevalent with fewer atoms).

Figure 10 provides additional insight into why the sampled Wasserstein distance may perform poorly. Here, we see the cumulative densities for the approximations learned under these two losses for five different states along the safe path in CliffWalk. The Wasserstein has converged to a fixed-point distribution, but not one that captures the true (Monte Carlo) distribution very well. By comparison, the categorical algorithm captures the variance of the true distribution much more accurately.

## Appendix E Supplemental Videos and Results

In Figure 13 we provide links to supplemental videos showing the C51 agent during training on various Atari 2600 games. Figure 12 shows the relative performance of C51 over the course of training. Figure 14 provides a table of evaluation results, comparing C51 to other state-of-the-art agents. Figures 15 – 18 depict particularly interesting frames.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .