Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research — part 2/3

Cleaned-text code points [15747, 31746). Bibliography omitted here; source.md includes it.

Under constant hyper-parameter settings (see appendix for details), we evaluate both the addition of each algorithmic component to DQN, as well as their removal from the full Rainbow agent on the classic control environments ( Figure 1 ) and on Minatar ( Figure 2 ).

We analyze our results in the context of the first two questions posed in Section 3 : Would Hessel et al. (2018) have arrived at the same qualitative conclusions, had they run their experiments on a set of smaller-scale experiments? Do the results of Hessel et al. (2018) generalize well to non-ALE environments, or are their results overly-specific to the chosen benchmark?

What we find is that the performance of the different components is not uniform throughout all environments; a finding which is consistent with the results observed by Hessel et al. (2018) . However, if we were to suggest a single agent that balances the tradeoffs of the different algorithmic components, our analysis would be consistent with Hessel et al. (2018) : combining all components produces a better overall agent.

Nevertheless, there are important details in the variations of the different algorithmic components that merit a more thorough investigation. An important finding of our work is that distributional RL, when added on its own to DQN, may actually hurt performance (e.g. Acrobot and Freeway); similarly, performance can sometimes increase when distributional RL is removed from Rainbow (e.g. MountainCar and Seaquest); this is in contrast to what was found by Hessel et al. (2018) on the ALE experiments and warrants further investigation. As Lyle et al. (2019) noted, under non-linear function approximators (as we are using in these experiments), using distributional RL generally produces different outcomes than the non-distributional variant, but these differences are not always positive.

## 5 Beyond the Rainbow

In this section we seek to answer the third question posed in section 3 : Is there scientific value in conducting empirical research in small- to mid-scale environments? We leverage the low cost of the small-scale environments to conduct a thorough investigation into some of the algorithmic components studied. Unless otherwise specified, the classic control and MinAtar environments are averaged over 100 and 10 independent runs, respectively; in both cases shaded areas report 95% confidence intervals.

### 5.1 Examining network architectures and batch sizes

We investigated the interaction of the best per-game hyper-parameters with the number of layers and units per layer. Due to space constraints we omit the figures from the main paper, but include them in the appendix. We found that in general using 2-3 layers with at least 256 units each yielded the best performance. Further, aside from Cartpole, the algorithms were generally robust to varying network architecture dimensions.

Another often overlooked hyper-parameter in training RL agents is the batch size. We investigated the sensitivity of DQN and Rainbow to varying batch sizes and found that while for DQN it is sub-optimal to use a batch size below 64, Rainbow seems fairly robust to both small and large batch sizes.

### 5.2 Examining distribution parameterizations

Although distributional RL is an important component of the Rainbow agent, at the time of its development Rainbow was only evaluated with the C51 parameterization of the distribution, as originally proposed by Bellemare et al. (2017) . Since then there have been a few new proposals for parameterizing the return distribution, notably quantile regression ( Dabney et al., 2017 ; Dabney et al., 2018a ) and implicit quantile networks ( Dabney et al., 2018b ) . In this section we investigate the interaction of these parameterizations with the other Rainbow components.

Quantile Regression for Distributional RL

In contrast to C51, QR-DQN ( Dabney et al., 2017 ; Dabney et al., 2018a ) computes the return quantile values for N N fixed, uniform probabilities. Compared to C51, QR-DQN has no restrictions or bound for value, as the distribution of the random return is approximated by a uniform mixture of N N Diracs: Z θ ​ ( x , a ) := 1 N ​ ∑ i = 1 N δ θ i ​ ( x , a ) Z_{\theta}(x,a):=\frac{1}{N}\sum_{i=1}^{N}\delta_{\theta_{i}(x,a)} , with each θ i \theta_{i} assigned a quantile value trained with quantile regression. In Figure 3 we evaluate the interaction of the different Rainbow components with Quantile and find that, in general, QR-DQN responds favourably when augmented with each of the components. We also evaluate a new agent, QRainbow , which is the same as Rainbow but with the QR-DQN parameterization. It is interesting to observe that in the classic control environments Rainbow outperforms QRainbow, but QRainbow tends to perform better than Rainbow on Minatar (with the notable exception of Freeway), suggesting that perhaps the quantile parameterization of the return distribution has greater benefits when used with networks that include convolutional layers.

Implicit quantile networks

We continue by investigating using implicit quantile networks (IQN) as the parameterization of the return distribution Dabney et al. (2018b) . IQN learns to transform a base distribution (typically a uniform distribution in [ 0 , 1 ] [0,1] ) to the quantile values of the return distribution. This can result in greater representation power in comparison to QR-DQN, as well as the ability to incorporate distortion risk measures . We repeat the “Rainbow experiment” with IQN and report the results in Figure 4 . In contrast to QR-DQN, in the classic control environments the effect on performance of various Rainbow components is rather mixed and, as with QR-DQN IRainbow underperforms Rainbow. In Minatar we observe a similar trend as with QR-DQN: IRainbow outperforms Rainbow on all the games except Freeway.

### 5.3 Munchausen Reinforcement Learning

Vieillard et al. (2020) introduced Munchausen RL as a simple variant to any temporal difference learning agent consisting of two main components: the use of stochastic policies and augmenting the reward with the scaled log-policy. Integrating their proposal to DQN yields M-DQN with performance superior to that of C51; the integration of Munchausen-RL to IQN produced M-IQN, a new state-of-the art agent on the ALE.

In Figure 5 we report the results when repeating the Rainbow experiment on M-DQN and M-IQN. In the classic control environments neither of the Munchausen variants seem to yield much of an improvement over their base agents. In Minatar, while M-DQN does seem to improve over DQN, the same cannot be said of M-IQN. We explored combining all the Rainbow components 4 4 4 We were unable to successfully integrate M-DQN with C51 nor double-DQN, so our M-Rainbow agent is compared against Rainbow without distributional RL and without double-DQN. with the Munchausen agents and found that, in the classic control environments, while M-Rainbow underperforms relative to its non-Munchausen counterpart, M-IRainbow can provide gains. In Minatar, the results vary from game to game, but it appears that the Munchausen agents yield an advantage on the same games (Asterix, Breakout, and SpaceInvaders).

.

### 5.4 Reevaluating the Huber loss

The Huber loss is what is usually used to train DQN agents as it is meant to be less sensitive to outliers. Based on recent anecdotal evidence, we decided to evaluate training DQN using the mean-squared error (MSE) loss and found the surprising result that on all environments considered using the MSE loss yielded much better results than using the Huber loss, sometimes even surpassing the performance of the full Rainbow agent (full classic control and Minatar results are provided in the appendix). This begs the question as to whether the Huber loss is truly the best loss to use for DQN, especially considering that reward clipping is typically used for most ALE experiments, mitigating the occurence of outlier observations. Given that we used Adam ( Kingma and Ba, 2015 ) for all our experiments while the original DQN algorithm used RMSProp, it is important to consider the choice of optimizer in answering this question.

To obtain an answer, we compared the performance of the Huber versus the MSE loss when used with both the Adam and RMSProp optimizers on all 60 Atari 2600 games. In Figure 6 we present the improvement obtained when using the Adam optimizer with the MSE loss over using the RMSProp optimizer with the Huber loss and find that, overwhelmingly, Adam+MSE is a superior combination than RMSProp+Huber. In the appendix we provide complete comparisons of the various optimizer-loss combinations that confirm our finding. Our analyses also show that, when using RMSProp, the Huber loss tends to perform better than MSE, which in retrospect explains why ( Mnih et al., 2015 ) chose the Huber over the simpler MSE loss when introducing DQN.

Our findings highlight the importance in properly evaluating the interaction of the various components used when training RL agents, as was also argued by Fujimoto et al. (2020) with regards to loss functions and non-uniform sampling from the replay buffer; as well as by Hessel et al. (2019) with regards to inductive biases used in training RL agents.

## 6 Putting it all together

### 6.1 Rainbow flavours

We compare the performance of DQN against all of the Rainbow variants and show the results of two environments in Figure 7 (full comparisons in the appendix). These two environments highlight the fact that, although Rainbow does outperform DQN, there are important differences amongst the various flavours that invite further investigation.

### 6.2 Environment properties

Our exhaustive experimentation on the four classic control and five MinAtar environments grant us some insight into their differing properties. We believe these environments pose a variety of interesting challenges for RL research and present a summary of our insights here, with a more thorough analysis in the appendix, in the hope that it may stimulate future investigations.

Classic control

Although CartPole is a relatively simple task, DQN can be quite sensitive to learning rates and as such, can prove an interesting testbed for optimizer stability. We found the use of Noisy networks and MSE loss to dramatically help with this sensitivity. It seems that distributional RL is required for obtaining good performance on LunarLander (see the poor results with DQN, M-DQN, and M-Rainbow in comparison with the others), suggesting this would be a good environment to investigate the differences in expectational and distributional RL as proposed by Lyle et al. (2019) . Both Acrobot and MountainCar are sparse reward environments, which are typically good environments for investigating exploration strategies; indeed, we observe that Noisy networks tend to give a performance gain in both these environments. MountainCar appears to be the more difficult of the two, a fact also observed by Tang et al. (2017) , Colas et al. (2018a) , and Declan et al. (2020) .

MinAtar

The value of MinAtar environments has already been argued by Young and Tian (2019) and recently exemplified by Ghiassian et al. (2020) and Buckman et al. (2021) . It is worth highlighting that both Seaquest and Freeway appear to lend themselves well for research on exploration methods, due to their partial observability and reward sparsity. We would like to stress that these environments enable researchers to investigate the importance and effect of using convolutional networks in RL without the prohibitive expense of the ALE benchmark.

## 7 Conclusion

On a limited computational budget we were able to reproduce, at a high-level, the findings of Hessel et al. (2018) and uncover new and interesting phenomena. Evidently it is much easier to revisit something than to discover it in the first place; however, our intent with this work was to argue for the relevance and significance of empirical research on small- and medium-scale environments. We believe that these less computationally intensive environments lend themselves well to a more critical and thorough analysis of the performance, behaviours, and intricacies of new algorithms (a point also argued by Osband et al. (2020) ). It is worth remarking that when we initially ran 10 independent trials for the classic control environments, the confidence intervals were very wide and inconclusive; boosting the independent trials to 100 gave us tighter confidence intervals with small amounts of extra compute. This would be impractical for most large-scale environments. Ensuring statistical significance when comparing algorithms, a point made by Colas et al. (2018b) and Jordan et al. (2020) , is facilitated with the ability to run a large number of independent trials.

We are by no means calling for less emphasis to be placed on large-scale benchmarks. We are simply urging researchers to consider smaller-scale environments as a valuable tool in their investigations, and reviewers to avoid dismissing empirical work that focuses on these smaller problems. By doing so, we believe, we will get both a clearer picture of the research landscape and will reduce the barriers for newcomers from diverse, and often underprivileged, communities. These two points can only help make our community and our scientific advances stronger.

## 8 Acknowledgements

The authors would like to thank Marlos C. Machado, Sara Hooker, Matthieu Geist, Nino Vieillard, Hado van Hasselt, Eleni Triantafillou, and Brian Tanner for their insightful comments on our work.

## Appendix A Environments

For OpenAI environments, data is summarized from https://github.com/openai/gym and information provided on the wiki https://github.com/openai/gym/wiki .

### A.1 CartPole-v0

CartPole is a task of balancing a pole on top of the cart. The cart has access to its position and velocity as state, and can only go left or right for each action. The task is over when the pole falls over (less than ± 12 \pm 12 deg), the cart goes out of the boundaries ( ± 2.4 \pm 2.4 units off the center), or 200 time steps are reached, with each step returning 1 reward. The agent is given a continuous 4-dimensional space describing the environment, and can respond by returning one of two values, pushing the cart either right or left.

### A.2 Acrobot-v1

In the Acrobot environment, the agent is given rewards for swinging a double-jointed pendulum up from a stationary position. The agent can actuate the second joint by returning one of three actions, corresponding to left, right, or no torque. The agent is given a six dimensional vector describing the environment’s angles and velocities. The episode ends when the end of the second pole is more than the length of a pole above the base. For each timestep that the agent does not reach this state, it is given a -1 reward. The episode length is 500 timesteps.

### A.3 LunarLander-v2

In the LunarLander environment, the agent attempts to land a lander on a particular location on a simulated 2D world. If the lander hits the ground going too fast, the lander will explode, or if the lander runs out of fuel, the lander will plummet toward the surface. The agent is given a continuous vector describing the state, and can turn its engine on or off. The landing pad is placed in the center of the screen, and if the lander lands on the pad, it is given a reward (100-140 points). The agent also receives a variable amount of reward when coming to rest, or contacting the ground with a leg (10 points). The agent loses a small amount of reward by firing the engine (-0.3 points), and loses a large amount of reward if it crashes (-100 points). The observation consists of the x and y coordinates, the x and y velocities, angle, angular velocity, and ground contact information of the lander (left and right leg) and the action consists of do nothing, fire left orientation engine, fire down engine and fire right orientation engine.