Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research — part 3/3

Cleaned-text code points [31746, 43723). Bibliography omitted here; source.md includes it.

### A.4 MountainCar-v0

MountainCar is a one dimensional track between two mountains. The goal is to drive up the mountain to the right. The agent receives a -1 reward for every time step it does not reach the top. The episode terminates when it reaches 0.5 position, or if 200 iterations are reached. The objective is for the agent to learn to drive back and forth to build momentum that will be enough to push the car up the hill. The observation consists of the car’s position and velocity and the action consists of pushing left, pushing right and no push.

### A.5 MinAtar Environments

Some of the best reinforcement learning algorithms require tens or hundreds of millions of timesteps to learn to play Atari games, the equivalent of several weeks of training in real time. MinAtar reduces the complexity of the representation learning problem for 5 Atari games, while maintaining the mechanics of the original games as much as possible. We use these MinAtar games to measure the impact of using some extensions to the DQN algorithm. MinAtar provides analogues to five Atari games which play out on a 10x10 grid. The environments provide a 10x10xn state representation, where each of the n channels correspond to a game-specific object, such as ball, paddle and brick in the game Breakout. Detailed descriptions of each game are available in https://github.com/kenjyoung/MinAtar .

## Appendix B Detailed description of the revisting rainbow components

In this section we present the enhancements to DQN that were combined for the revisting rainbow agent.

### B.1 Double Q-learning

van Hasselt et al. [2016] added double Q-learning [ Hasselt, 2010 ] to mitigate overestimation bias in the Q Q -estimates by decoupling the maximization of the action from its selection in the target bootstrap. The loss from Equation 2 is replaced with L ⁡ ( θ ) = 𝔼 ( s , a , r , s ′ ) ∼ U ⁡ ( D ) ​ [ ( r + γ ​ Q θ ¯ ​ ( s ′ , arg ⁡ max a ′ ∈ 𝒜 ​ Q θ ​ ( s ′ , a ′ ) ) − Q θ ​ ( s , a ) ) 2 ] \displaystyle L(\theta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim U(D)}\left[\left(r+\gamma Q_{\bar{\theta}}(s^{\prime},\arg\max_{a^{\prime}\in\mathcal{A}}Q_{\theta}(s^{\prime},a^{\prime}))-Q_{\theta}(s,a)\right)^{2}\right]

### B.2 Prioritized experience replay

Instead of sampling uniformly from the replay buffer ( U ⁡ ( D ) U(D) ), prioritized experience replay [ Schaul et al., 2016 ] proposed to sample a trajectory t = ( s , a , r , s ′ ) t=(s,a,r,s^{\prime}) with probability p t p_{t} proportional to the temporal difference error: p t ∝ | r + γ ​ max a ′ ∈ 𝒜 ​ Q θ ¯ ​ ( s ′ , a ′ ) − Q θ ​ ( s , a ) | ω p_{t}\propto\left|r+\gamma\max_{a^{\prime}\in\mathcal{A}}Q_{\bar{\theta}}(s^{\prime},a^{\prime})-Q_{\theta}(s,a)\right|^{\omega} where ω \omega is a hyper-parameter for the sampling distribution.

### B.3 Dueling networks

Wang et al. [2016] introduced dueling networks by modifying the DQN network architecture. Specifically, two streams share the initial convolutional layers, separately estimating V ∗ ​ ( s ) V^{*}(s) , and the advantages for each action: A ⁡ ( s , a ) := Q ∗ ​ ( s , a ) − V ∗ ​ ( s ) A(s,a):=Q^{*}(s,a)-V^{*}(s) . The output of the full network is defined by: Q θ ​ ( s , a ) = V η ​ ( f η ​ ( s ) ) + A ψ ​ ( f η ​ ( s ) , a ) − ∑ a ′ ∈ 𝒜 A ψ ​ ( f ξ ​ ( s ) , a ′ ) | 𝒜 | Q_{\theta}(s,a)=V_{\eta}(f_{\eta}(s))+A_{\psi}(f_{\eta}(s),a)-\frac{\sum_{a^{\prime}\in\mathcal{A}}A_{\psi}(f_{\xi}(s),a^{\prime})}{|\mathcal{A}|} Where ξ \xi denotes the parameters for the shared convolutional layers, η \eta denotes the parameters for the value estimator stream, ψ \psi denots the parameters for the advantage estimator stream, and θ := ξ ∪ η ∪ ψ \theta:=\xi\cup\eta\cup\psi .

### B.4 Multi-step learning

Instead of computing the temporal difference error using a single-step transition, one can use multi-step targets instead [ Sutton, 1988 ] , where for a trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , ⋯ ) (s_{0},a_{0},r_{0},s_{1},a_{1},\cdots) and update horizon n n : R t ( n ) := ∑ k = 0 n − 1 γ k ​ r t + k + 1 R_{t}^{(n)}:=\sum_{k=0}^{n-1}\gamma^{k}r_{t+k+1} yielding the multi-step temporal difference: R t ( n ) + γ n ​ max a ′ ∈ 𝒜 ​ Q θ ¯ ​ ( s t + n , a ′ ) − Q θ ​ ( s t , a t ) R_{t}^{(n)}+\gamma^{n}\max_{a^{\prime}\in\mathcal{A}}Q_{\bar{\theta}}(s_{t+n},a^{\prime})-Q_{\theta}(s_{t},a_{t})

### B.5 Distributional RL

Bellemare et al. [2017] demonstrated that the Bellman recurrence also holds for value distributions : Z ⁡ ( x , a ) ​ = 𝐷 ​ R ​ ( s , a ) + γ ​ Z ​ ( X ′ , A ′ ) Z(x,a)\overset{D}{=}R(s,a)+\gamma Z(X^{\prime},A^{\prime}) where Z Z , R R , and ( X ′ , A ′ ) (X^{\prime},A^{\prime}) are random variables representing the return, immediate reward, and next state-action, respectively. The authors present an algorithm (C51) to maintain an estimate Z θ Z_{\theta} of the return distribution Z Z by use of a parameterized categorical distribution with 51 atoms.

### B.6 Quantile Regression for Distributional RL

Dabney et al. [2017] computes the return quantile values for N N fixed, uniform probabilities. The distribution of the random return is approximated by a uniform mixture of N N Diracs with each θ i \theta_{i} assigned a quantile value trained with quantile regression. Formally, a quantile distribution Z θ ∈ 𝒵 Q Z_{\theta}\in\mathcal{Z}_{Q} maps each state-action pair ( x , a ) (x,a) to a uniform probability distribution is, Z θ ​ ( x , a ) := 1 N ​ ∑ i = 1 N δ θ i ​ ( x , a ) Z_{\theta}(x,a):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\theta_{i}(x,a)} These quantile estimates are trained using the Huber [ Huber, 1964 ] quantile regression loss.

### B.7 Implicit quantile networks

Dabney et al. [2018b] extend the approach of Dabney et al.(2018), from learning a discrete set of quantiles to learningthe full quantile function, a continuous map from probabilities to returns. Moreover, extended the fixed quantile fractions to uniform samples. The approximation of the quantile function Z Z is,

Z τ ​ ( x , a ) ≈ f ​ ( ψ ⁡ ( x ) ⊙ ϕ ⁡ ( τ ) ) a Z_{\tau}(x,a)\approx f(\psi(x)\odot\phi(\tau))_{a}

where ⊙ \odot denotes the element-wise (Hadamard) product.

### B.8 Noisy nets

Fortunato et al. [2018] propose replacing the simple ϵ \epsilon -greedy exploration strategy used by DQN with noisy linear layers that include a noisy stream. Specifically, the standard linear layers defined by 𝐲 = 𝐛 + 𝐖𝐱 \mathbf{y}=\mathbf{b}+\mathbf{Wx} are replaced by: 𝐲 = ( 𝐛 + 𝐖𝐱 ) + ( 𝐛 𝐧𝐨𝐢𝐬𝐲 ⊙ ϵ b + ( 𝐖 𝐧𝐨𝐢𝐬𝐲 ⊙ ϵ w ) ​ 𝐱 ) \mathbf{y}=(\mathbf{b}+\mathbf{Wx})+(\mathbf{b_{noisy}}\odot\epsilon^{b}+(\mathbf{W_{noisy}}\odot\epsilon^{w})\mathbf{x}) where ϵ b \epsilon^{b} and ϵ w \epsilon^{w} are noise variables ( Hessel et al. [2018] use factorised Gaussian noise) and ⊙ \odot is the Hadamard product.

### B.9 Munchausen Reinforcement Learning

Vieillard et al. [2020] modify the regression target adding the scaled log-policy to the immediate reward: Y M − D ​ Q ​ N = r t + α ​ τ ​ ln ⁡ π θ ¯ ​ ( a t ∣ s t ) + γ ​ ∑ a ′ ∈ 𝒜 π θ ¯ ​ ( a ′ ∣ s t + 1 ) ​ ( Q θ ¯ ​ ( s t + 1 , a ′ ) − τ ​ ln ⁡ π θ ¯ ​ ( a ′ ∣ s t + 1 ) ) Y^{M-DQN}=r_{t}+\alpha\tau\ln\pi_{\bar{\theta}}\left(a_{t}\mid s_{t}\right)+\gamma\sum_{a^{\prime}\in\mathcal{A}}\pi_{\bar{\theta}}\left(a^{\prime}\mid s_{t+1}\right)\left(Q_{\bar{\theta}}\left(s_{t+1},a^{\prime}\right)-\tau\ln\pi_{\bar{\theta}}\left(a^{\prime}\mid s_{t+1}\right)\right)

with π θ = sm ⁡ ( Q θ ¯ τ ) \pi_{\theta}=\operatorname{sm}\left(\frac{Q_{\bar{\theta}}}{\tau}\right) and a scaling factor α ∈ [ 0 , 1 ] \alpha\in[0,1] .

### B.10 Loss functions

We ran experiments using Mean Square Error and Huber loss [ Huber, 1964 ] , therefore we consider it important to introduce these concepts in this paper. MSE is the sum of squared distances between the target variable and predicted values. MSE gives relatively higher weight (penalty) to large errors/outliers, while smoothening the gradient for smaller errors. The MSE is formally defined by the following equation, MSE ⁡ ( y i , y ^ i ) = 1 2 ​ ( y i − y ^ i ) 2 \mathrm{MSE}(y_{i},\hat{y}_{i})=\frac{1}{2}\left(y_{i}-\hat{y}_{i}\right)^{2}

On the other hand, Huber loss is less sensitive to outliers in data than the squared error loss and it is defined by the following equation,

Huber ⁡ ( y i , y ^ i ) = { 1 2 ​ ( y i − y ^ i ) 2 for ​ | y i − y ^ i | ≤ δ , δ ​ | y i − y ^ i | − 1 2 ​ δ 2 otherwise \mathrm{Huber}\left(y_{i},\hat{y}_{i}\right)=\left\{\begin{array}[]{lr}\frac{1}{2}\left(y_{i}-\hat{y}_{i}\right)^{2}&\text{ for }\left|y_{i}-\hat{y}_{i}\right|\leq\delta,\\ \delta\left|y_{i}-\hat{y}_{i}\right|-\frac{1}{2}\delta^{2}&\text{ otherwise }\end{array}\right.

## Appendix C Hyperparameters settings for classic control environments

## Appendix D Hyperparameters settings for MinAtar environments

## Appendix E Average runtime for environments used in the experiments

## Appendix F Impact of algorithmic components

In the following tables we list the algorithmic components that had the most positive effect on the various agents and environments. The following abbreviations are used in this section: Dist: Distributional, Noi: Noisy, Prio: Prioritized, Mult: Multi-step, Dou: Double , Due: Dueling.

## Appendix G Examining network architectures and batch sizes

In this section we evaluate the effect of varying the number of layers and number of hidden units with DQN ( Figure 10 ) and Rainbow ( Figure 11 ), as well as the effect of batch sizes on both ( Figure 12 ). For all the results in this section we ran 10 independent runs for each, and the shaded areas represent 95% confidence intervals.

## Appendix H Reevaluating the Huber loss, complete results

In this section we present complete results comparing the various combinations possible when using either Adam or RMSProp optimizers, and Huber or MSE losses.

### H.1 Classic control and MinAtar results

We first evaluated the various combinations on all the classic control and MinAtar environments and found that Adam+MSE worked best. Full results displayed in Figure 13 .

### H.2 Atari results

Given that the standard practice when training DQN is to use RMSPRop with the Huber loss, yet our results on the classic control and MinAtar environments suggest that MSE would work better, we performed a thorough comparison of the various combinations, which we present in this section.

We compare Adam vs RMSProp when both use the Huber loss ( Figure 14 ) and MSE loss ( Figure 15 ); we also compare the MSE vs Huber loss when using RMSProp ( Figure 16 ) and using Adam ( Figure 17 ). In Figure 18 we compare the various combinations using human normalized scores across all games, and we provide complete training curves for all games in Figure 19 . As can be seen, the best results are obtained when using Adam and the MSE loss, in contrast to the standard practice of using RMSProp and the Huber loss.

## Appendix I Rainbow flavours, full results

In this section we present the complete results for all the environments when comparing DQN against the various Rainbow flavours ( Figure 20 ).

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .