Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC — part 2/4

Cleaned-text code points [15319, 31160). Bibliography omitted here; source.md includes it.

Vectorization - RL agents typically take multiple steps in a single environment, followed by a gradient update with a small batch size (Rainbow DQN took 4 environment steps, followed by a batch of 32). However, taking multiple steps in parallel and performing updates on larger batches can significantly reduce walltime. We follow Schmidt & Schmied (2021) , taking 1 step in 64 parallel environments with one gradient update with batch size 256 ( Schmidt & Schmied (2021) took two gradient updates). This results in a replay ratio (ratio of gradient updates to environment steps) of 1 64 \frac{1}{64} . Higher replay ratios have been shown to improve performance ( D’Oro et al., 2022 ) , however we opt to keep this value low to reduce walltime.

### 3.2 Hyperparameters

Hyperparameters have repeatedly shown to have a very large impact on performance in RL ( Ceron et al., 2024 ) , thus we perform a small amount of tuning to improve performance. Firstly, how frequently the target network is updated is closely intertwined with batch size and replay ratio. We found that updating the target network every 500 gradient steps 4 4 4 This equates to 32,000 environment steps (128,000 frames), compared to Rainbow DQN’s 8,000 steps. performed best. Given our high batch size, we additionally performed minor hyperparameter tests using different learning rates finding that a slightly higher learning rate of 1 × 10 − 4 1\text{\times}{10}^{-4} performed best, compared to 6.25 × 10 − 5 6.25\text{\times}{10}^{-5} in Rainbow DQN. In Appendix D.2 , we clarify the meaning of the terms frames, steps and transitions.

For many years, RL algorithms have used a discount rate of 0.99 0.99 , however, when reaching high performance, lower discount rates alter the optimal policy, causing even optimally performing agents to not collect the maximum cumulative rewards. To prevent this, we follow MuZero Reanalyse ( Schrittwieser et al., 2021 ) using γ = 0.997 \gamma=0.997 . For our Prioritized Experience Replay, we use the lower value of α = 0.2 \alpha=0.2 , the parameter used to determine sample priority, recommended by Toromanoff et al. (2019) when using IQN. Lastly, many previous experiments used only noisy networks or ϵ \epsilon -greedy exploration, however, we opt to use both until 100M frames, then set ϵ \epsilon to zero, effectively disabling it. We elaborate on this decision in Appendix F .

## 4 Evaluation

To assess BTR, we test on two standard RL benchmarks, Atari ( Bellemare et al., 2013 ) and Procgen ( Cobbe et al., 2020 ) in Section 4.1 . Secondly, we train agents for three modern games (Super Mario Galaxy, Mario Kart Wii, and Mortal Kombat) with complex 3D graphics and physics in Section 4.2 , never before shown to be trainable with RL.

### 4.1 Atari and Procgen Performance

We evaluate BTR on the Atari-60 benchmark following ( Machado et al., 2018 ) and without life information (see Appendix I for the impact), evaluating every million frames on 100 episodes. Figure 1 plots BTR against DQN, Rainbow DQN and Dreamer-v3, showing BTR’s competitive performance despite using significantly less walltime. Figure 2 shows a box plot comparison of final performance. In comparison to human expert performance, BTR equals or exceeds them in 52 of 60. Importantly, we find that BTR appears to continue increasing performance beyond 200 million frames, indicating that higher performance is still possible with more time and data. Results tables and graphs can be found in Appendices A and B , respectively.

To further confirm BTR’s performance, we benchmark on Procgen ( Cobbe et al., 2020 ) , a procedurally generated set of environments aiming to prevent overfitting to specific tasks, a prevalent problem in RL ( Justesen et al., 2018 ; Juliani et al., 2019 ) . The results are shown in Figure 3 with individual games in Appendix B . BTR is able to exceed Rainbow DQN + Impala’s performance, despite using significantly fewer convolutional filters (which Cobbe et al. (2020) found to significantly improve performance) and using 8 hours of walltime compared to 41. While BTR provides an improvement over Rainbow DQN in Procgen, we did not target procedurally generated environments thus it does not currently compete with the state-of-the-art ( Cobbe et al., 2021 ; Hafner et al., 2023 ) . There are numerous ways performance can be improved ( Jesson & Jiang, 2024 ; Cobbe et al., 2020 ) which we leave to future work.

### 4.2 Applying BTR to Modern Games

To demonstrate BTR’s capabilities beyond standard RL benchmarks, we utilized Dolphin ( Dolphin-Emulator, 2024 ) , a Nintendo Wii emulator, to train agents for a range of modern 3D games: Super Mario Galaxy, Mario Kart Wii and Mortal Kombat. Using a desktop PC, we were able to train the agent to complete some of the most difficult tasks within each game. Namely, the final level in Super Mario Galaxy, Rainbow Road (a notoriously difficult track in Mario Kart Wii), and defeating all opponents in Mortal Kombat Endurance mode. For details about the environments and setup, see Appendix J . To achieve this, BTR required minimal adjustments: changing the input image resolution to 140x114 (from Atari’s 84x84) due to the game’s higher resolution and aspect ratio, and to reduce the number of vectorized environments to 4 due to the games’ memory and CPU requirements.

BTR was able to solve all three games, including consistently finishing first place in Mario Kart. In contrast, Rainbow DQN’s performance plateaued before completing any of the games. We provide videos of our agent playing all three Wii games and all games in the Atari-5 benchmark 5 5 5 https://www.youtube.com/playlist?list=PL4geUsKi0NN-sjbuZP_fU28AmAPQunLoI .

## 5 Analysis

Given BTR’s performance demonstrated in Section 4 , in this section, we ablate each component to evaluate their performance impact (Section 5.1 ). Using the ablated agents, we measure numerous attributes during and after training to assess each component’s impact (Section 5.2 ).

### 5.1 Ablations Studies

BTR amalgamates independently evaluated components into a single algorithm. To understand and verify each component’s contribution, Figure 5 plots BTR’s performance without each component on the Atari-5 benchmark. 6 6 6 Due to the resources required to evaluate on all environments, Aitchison et al. (2023) proposes a subset of 5 games that closely correlate with the performance across all of them.

We find that Impala had the largest effect on performance (+142% IQM), with the other components generally causing a less significant effect. Despite this, simply using Rainbow with Impala does not produce similar results (6.3 IQM compared to 7.7 on Atari-5). Munchausen and IQN have a strong impact on environments requiring fine-grained control such as Phoenix , as explored in Section 5.2 .

For vectorization and maxpooling, while their inclusion reduces performance, we find their secondary effects crucial to keep BTR computationally accessible. Omitting vectorization increases walltime by 328% (Figure 6 ) by processing environment steps in parallel and taking fewer gradient steps (781,000 compared to Rainbow DQN’s 12.5 million). 7 7 7 A result of removing vectorization is using smaller batches, which Obando Ceron et al. (2024) finds improves exploration. We find that maxpooling decreases the model’s parameters by 77%, and makes using wider convolutional layers possible without causing the total number of parameters to increase drastically.

### 5.2 What are the effects of BTR’s components?

To help interpret the results in Section 5.1 , Table 2 measures seven different attributes of the agent either during or after training: action gaps and action swaps (linked to causing approximation errors ( Bellemare et al., 2016 ) ); policy churn (which can cause excessive off-policyness and instability ( Schaul et al., 2022 ) ) and score with additional noise (indicating robustness of the policies).

While it is clear that Impala strongly contributes to performance, we find that without BTR’s other components the learned policy is highly noisy and unstable. Table 2 , demonstrates that without IQN and Munchausen the agent experiences very low action gaps (absolute Q-value difference between the highest two valued actions), causing the agent to swap its argmax action almost every other step. This is likely to result in approximation errors altering the policy and causing a high degree of off-policyness in the replay buffer. This is particularly detrimental in games requiring fine-grained control, such as Phoenix where the agent needs to narrowly dodge many projectiles, reflected in BTR’s performance without these components.

Furthermore, we find that maxpooling produces a more robust policy. To test this, we evaluate the performance of BTR’s ablations when taking different quantities of ϵ \epsilon -actions and with altered observations and find maxpooling alleviates some of the performance loss (Table 2 ). Lastly, we find Munchausen and IQN to have a significant impact on Policy Churn ( Schaul et al., 2022 ) , with Munchausen reducing it by 6.4% and IQN increasing it by 3.3%. As a result, when these components are used together, they appear to reach a level of churn which does not harm learning and potentially provides some exploratory benefits.

Lastly, Figure 7 shows an analysis of the trained model weights across the Atari-5 benchmark. We find little difference between trained models other than when removing Impala, which decreases dormant neurons and increases the L2 Norm of different layers, which have been linked with plasticity loss ( Lyle et al., 2024 ) .

## 6 Related Work

The most similar work to BTR, developing a computationally-limited non-distributed RL algorithm, is “Fast and Efficient Rainbow” ( Schmidt & Schmied, 2021 ) . They optimized Rainbow DQN to maximize performance for 10 million frames through parallelizing the environments and dropping C51 along with hyperparameter optimizations. This differs from our goals of producing an algorithm that scales across training regimes (up to 200 million frames) and domains (Atari, Procgen, Super Mario Galaxy, Mario Kart and Mortal Kombat), resulting in different design decisions.

For less computation-limited approaches, Ape-X ( Horgan et al., 2018 ) was the first to explore highly distributed training, allowing agents to be trained on a billion frames in 120 hours through using > > 100 CPUs. Following this, Kapturowski et al. (2018) proposed R2D2 using a recurrent neural network, increasing sample efficiency but slowing down gradient updates by 38%. Agent57 ( Badia et al., 2020a ) was the first RL agent to achieve superhuman performance across 57 Atari games, though required 90 billion frames. MEME ( Kapturowski et al., 2023 ) , Agent57’s successor, focused on achieving superhuman performance within the standard 200 million frames limit, achieved by using a significantly higher replay ratio and larger network architecture. Most recently, Dreamer-v3 ( Hafner et al., 2023 ) used a 200 million parameter model requiring over a week of training, achieving similar results as MEME. We detail some key differences between BTR, MEME and Dreamer-v3 in Table 3 . While these approaches perform equally or better than BTR, all are inaccessible to smaller research labs or hobbyists due to their required computational resources and walltime. Therefore, while these algorithms have important research value demonstrating the possible performance of RL agents, performative algorithms with a lower cost of entry, like BTR, are necessary for RL to become widely applicable and accessible.

## 7 Conclusions

We have demonstrated that, once again, independent improvements from across Deep Reinforcement Learning can be combined into a single algorithm capable of pushing the state-of-the-art far beyond what any single improvement is capable of. Importantly, we find that this can be accomplished on desktop PCs, increasing the accessibility of RL for smaller research labs and hobbyists.

We acknowledge there exists many more promising improvements we could not include in BTR, leaving room for more future work to create stronger integrated agents in a few years. For example, BTR does not add an explicit exploration component, resulting in it struggling in hard-exploration tasks such as Montezuma’s Revenge ; therefore, mechanisms used in Never Give Up ( Badia et al., 2020b ) , etc may prove useful. Section 5.1 found that the neural network’s core architecture, Impala, had the largest impact on performance, an area we believe is generally underappreciated in RL. Previous work ( Kapturowski et al., 2018 ) has incorporated recurrent models enhancing performance, though we are uncertain how this can be incorporated into BTR without affecting its computational accessibility, a question which warrants future research.

## Impact Statement

This paper presents work whose goal is to advance the field of Reinforcement Learning, particularly to improve accessibility to those with limited computational resources. As with any work increasing accessibility, this has potential for misuse by bad actors. However, we believe these concerns are offset by the field’s potential to tackle key societal problems.

## Acknowledgments

This work was supported by the UK Research and Innovation (UKRI) Centre for Doctoral Training in Machine Intelligence for Nano-electronic Devices and Systems [EP/S024298/1] and the Engineering and Physical Sciences Research Council (EPSRC) ActivATOR project [EP/W017466/1]. The authors acknowledge the use of the IRIDIS X High Performance Computing Facility, and the Southampton-Wolfson AI Research Machine (SWARM) GPU cluster generously funded by the Wolfson Foundation, together with the associated support services at the University of Southampton in the completion of this work. The authors dedicate this work to the memory of George Morton-Fallows, whose passion for computer science inspired this research.

## Appendix A Full Results Tables

## Appendix B Full Results Graphs

## Appendix C Additional Ablations

## Appendix D Hyperparameters

### D.1 Environment Details

### D.2 Algorithm Hyperparameters

### D.3 Clarity of the terms Frames, Steps and Transitions

Throughout the Arcade Learning Environment’s history (ALE) ( Bellemare et al., 2013 ; Machado et al., 2018 ) , there have been many ambiguities around the terms: frames, steps and transitions, which are sometimes used interchangeably. Frames refer to the number of individual frames the agent plays, including those within repeated actions (also called frame skipping). This is notably different from the number of steps the agent takes, which does not include these skipped frames. When using the standard Atari wrapper, training for 200M frames is equivalent to training for 50M steps. Lastly, transitions refer to the standard tuple ( s t , a t , r t , s t + 1 ) (s_{t},a_{t},r_{t},s_{t+1}) , where the timestep t t refers to a steps, not frames. We encourage researchers to make this clear when publishing work, including when mentioning the values of different hyperparameters.

## Appendix E Beyond The Rainbow Architecture & Loss Function

### E.1 Architecture

Figure E7 shows the neural network architecture of the BTR algorithm. The architecture is highly similar to the Impala architecture ( Espeholt et al., 2018 ) , with notable exceptions:

∙ \bullet Spectral Normalization Within each Impala CNN block, each residual layer (containing two Conv 3x3 + ReLu) has spectral normalization applied, as discussed in Section 3.1 .

Following the CNN blocks, a 6x6 adaptive maxpooling layer is added.

In order to use IQN, it is required to draw Tau samples, which are multiplied by the output of the CNN layers, as shown by the section ‘IQN Samples’ in figure E7 .