Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC — part 3/4

Cleaned-text code points [31160, 47074). Bibliography omitted here; source.md includes it.

Dueling (as included in the original Rainbow DQN) splits the fully connected layers into value and advantage streams, where the advantage stream output has a mean of 0, and is then added to the value stream.

As included in Rainbow DQN, Noisy Networks replace the linear layers with noisy layers.

Lastly, the sizes of many of the layers given in Figure E7 are dependent upon the Impala width scale, of which we use the value 2. For example, the Impala CNN blocks have [16 × \times width, 32 × \times width, 32 × \times width] channels respectively. The output size of the convolutional layers (including the maxpooling layer) is 6 × \times 6 × \times 32 × \times width, as a 6x6 maxpooling layer is used. Lastly, the cos embedding layer, after generating IQN samples, requires the same size as the output of the convolutional layers. Hence, the size is selected accordingly. Another benefit of the 6x6 maxpooling layer is following the product of the convolutional layers and IQN samples, the number of parameters is fixed, regardless of the input size. Figure E8 shows the number of parameters the ablated versions of BTR have.

### E.2 Loss Function

The resulting loss function for the BTR algorithm remains the same as that defined in the appendix of the Munchausen paper, which gave a loss function for Munchausen-IQN. As the other components in BTR do not affect the loss, the resulting temporal-difference loss function is the same. For self-containment, we include this loss function below: T D B ​ T ​ R = r t + α [ τ ln π ( a t | s t ) ] l 0 0 + γ ∑ a ∈ A π ( a | s t + 1 ) ( z σ ′ ( s t + 1 , a ) − τ ln π ( a | s t + 1 ) ) − z σ ( s t , a t ) TD_{BTR}=r_{t}+\alpha[\tau\ln\pi(a_{t}|s_{t})]_{l_{0}}^{0}+\gamma\sum_{a\in A}\pi(a|s_{t+1})(z_{\sigma^{{}^{\prime}}}(s_{t+1},a)-\tau\ln\pi(a|s_{t+1}))-z_{\sigma}(s_{t},a_{t}) (E1) with π ( ⋅ | s ) = s m ( q ~ ​ ( s , ⋅ ) τ ) \pi(\cdot|s)=sm(\frac{\tilde{q}(s,\cdot)}{\tau}) (that is, the policy is softmax with q˜, the quantity with respect to which the original policy of IQN is greedy). It is also worth noting here that due to the character conflict of both Munchausen and IQN using τ \tau (Munchausen as a temperature parameter, and IQN for drawing samples), we replace IQN’s τ \tau with σ \sigma . l 0 , τ l_{0},\tau and α \alpha are hyperparameters set by Munchausen. We use the same values in BTR, also shown in our hyperparameter table in Appendix D.2 .

### E.3 Analysis Confidence Intervals

Due to space constraints, we was unable to include confidence intervals for Table 2 in the main paper. A repeat of the main paper Table can be found in Table E7 , with the associated confidence intervals in Table E8 .

## Appendix F BTR with and without Epsilon Greedy

One of the first observations we made early in the testing process was that the inclusion of using ϵ \epsilon -greedy in addition to NoisyNetworks benefited some environments but not others. Specifically, performance was reduced on BattleZone and Phoenix , both games where the agent reached very high levels of performance with extremely precise control. However, DoubleDunk performed significantly worse, only reaching a score of 0, rather than the score of 23 the final BTR algorithm achieved. Similar findings were also found in the full version of Rainbow DQN, which used only NoisyNetworks, which achieved a best score of -0.3 (Dopamine’s “compact” Rainbow DQN, however, which did not use NoisyNetworks achieved 22). From this, we conclude that NoisyNetworks alone failed to sufficiently explore the environment, whereas ϵ \epsilon -greedy did not. From these results, we eventually decided to use both methods, but disable ϵ \epsilon -greedy halfway through training to reap the best of both techniques.

## Appendix G Experiment Compute Resources

### G.1 Our Compute Resources

For running our experiments, we used a mixture of desktop computers and internal clusters. The desktop PCs used an GPU Nvidia RTX4090, CPU intel i9-14900k and 64GB of DDR5 6000mhz RAM. When using internal clusters, we used a mixture of GPUs, including Nvidia A100s, Nvidia Volta V100 and Nvidia Quadro RTX 8000. As for CPUs, we used 2 x 2.4 GHz Intel(R) Xeon(R) Gold 6336Y, 48 Cores. Lastly, we saved the models used to produce our analysis, totalling around 300gb across all of our ablations on the Atari-5 benchmark, saving a model every 1 million frames.

As most of our experiments were performed on desktop PC, in the main body of our paper we reference these speeds. We found that desktop PCs actually outperformed internal clusters, likely due to desktop CPUs being more suited to performing environment steps, outlined in the next subsection.

When testing ideas originally (those mentioned in Appendix H ), we only tested them using a single run of the games BattleZone , NameThisGame and Phoenix unless otherwise stated. Whilst this method of evaluation is not statistically significant, for preliminary purposes with computational restrictions, we deemed this the best option.

### G.2 BTR with Different Hardware

In this work, we look to make high-performance RL more accessible to those with fewer computing resources, especially those only with access to desktop computers. Most of our experiments were performed with an RTX4090, we also provide some walltimes for 200M Atari frames for lower-end machines and provide a brief comparison of desktop PCs against internal clusters:

Desktops:

Original: RTX 4090, Intel i9-13900k (2023), 64GB RAM - 11.5 Hours

RTX 3070, Ryzen 9 3900X (2019), 64GB RAM - 52 Hours

RTX 2080 ti, Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz (2018), 128GB RAM - 32 Hours

Internal Clusters:

Nvidia H100, 48 Core Intel(R) Xeon(R) Platinum 8468 (2023), 2TB RAM - 15 Hours

Nvidia A100, 24 Core Intel(R) Xeon(R) Gold 6336Y (2021), 512GB RAM - 22 Hours

We note that there is significant variability in hardware (processors, memory bus speeds, etc), but the results still show reasonable times compared to not using BTR. Overall, we found that training BTR was very capable of running on lower end machines, with the agent (excluding the environments) using around 15GB of RAM. The main performance bottleneck was running the environment in parallel, making the number of CPU cores and processor speed most important. BTR also provides strong performance long before 200M frames, thus providing practical utility for lower-end machines.

## Appendix H Other Things We Tried

Throughout the development of the BTR algorithm, we experimented with many different components and hyperparameters. A brief list of ideas we tried that performed worse or equivalent to the final algorithm includes:

• Using Exponential Moving Average networks rather than using fixed target networks (this was both computationally slower and performed worse).

• Varying the frequency of updating the target network (we tested 250, 500 and 1000, finding 500 to perform best).

• Changing the size of maxpool layer following the convolutional layers (we tested 4 and 8, however 6 performed significantly better).

• Decaying the learning rate from 1 × 10 − 4 1\text{\times}{10}^{-4} to 0 0 over the course of training (this made no significant difference).

• Different learning rates, finding 1 × 10 − 4 1\text{\times}{10}^{-4} to perform best, however 5 × 10 − 5 5\text{\times}{10}^{-5} also performed similarly as was used in Implicit Quantile Networks (IQN).

• Using the AdamW optimizer ( Loshchilov & Hutter, 2019 ) which uses weight decay with the decay parameter 1 ​ e − 4 1e-4 , however found this made no significant difference.

• Using the GeLu activation instead of ReLu, which drastically reduced performance.

Only testing on a single environment ( BattleZone ), we also tried: • Annealing the discount rate from 0.97 to 0.997 throughout training, but found no significant difference.

• Applying spectral normalization to the linear layers (dramatically worse performance).

• Increasing the number of cos’ from IQN (no significant difference on performance).

• Using Dopamine’s Prioritized Experience Replay buffer which doesn’t include a α \alpha value (moderately worse performance).

• As discussed in F , we also tried not using ϵ \epsilon -greedy when using noisy nets.

Lastly we also tried removing some of the original components from Rainbow DQN on Atari BattleZone , including Dueling, Prioritized Experience Replay and Noisy Networks. Prioritized Experience Replay and Noisy Networks both proved beneficial, so were kept in the algorithm. Dueling did not seem to make any significant difference, however we did not choose to remove it for a clearer continuation of Rainbow DQN, in addition to potentially being useful in other Atari environments.

Shortly after the submission of this work, we tested BTR with addition of Layer Normalization, and found positive results. Layer Normalization can improve the robustness to a variety of pathologies that cause loss of plasticity ( Lyle et al., 2024 ) , and helps to improve the conditioning of the network’s gradients in RL ( Ball et al., 2023 ) . Below in Figure H9 we show the impact of including layer normalization into BTR.

## Appendix I Altered Atari Environment Settings

In order to investigate the impact of the environmental sticky actions parameter and to compare against other works, we include results for it on the Atari-5 benchmark in Figure I10 .

Some prior works choose to pass life information to the agent ( Schmidt & Schmied, 2021 ) . To clarify, this is different to terminal on loss of life. Life information does not reset the episode upon losing a life, but does pass a terminal to the buffer, allowing the agent to experience further into episodes while also giving the agent a negative signal for losing a life. This setting is not recommended in Machado et al. (2018) , and works which use it are not comparable to those which don’t. To emphasize this point, we take the three games from the Atari-5 and perform a comparison.

## Appendix J BTR for Wii Games

BTR interfaces with different Wii Games via the Dolphin Emulator. Specifically, we use a forked repository of Dolphin Emulator to allow Python scripts to interact with the emulator. This includes loading savestates (used to reset episodes), grabbing the screen as a PIL image at the Wii’s internal resolution of 480p (downsampled to 140x114 and grey-scaled, used for all observations), reading the Wii’s RAM (used for reward functions and termination conditions) and allowed programmatic input into the emulator (for setting actions). Using Dolphin’s portable setting, we are able to run multiple Dolphin Emulators simultaneously on the same machine. Each instance runs as a unique process, and communicates with the agent via Python’s multiprocessing library. Similarly to the Atari benchmark, for all games we used a frameskip of 4.

### J.1 Super Mario Galaxy

This environment used Super Mario Galaxy’s final level, The Center of the Universe , and had to make it to the final fight at the end of the game. The agent had 6 actions, including None, moving in each direction and jumping. Additionally, if the jump action was performed following a movement, the agent would continue to move in that direction.

Rewards were given via finding many values in the Wii’s memory that resembled progress in the level. The agent was then rewarded for this progress value increasing from the last frame. If the agent’s position entered a set region, the progress variable would be moved. Additionally, the game uses a life system, where the player has a maximum of 3 lives and can lose or gain lives in many different ways. The agent was given a reward of +1 for gaining a life, and -1 for losing a life. Lastly, episode termination occurred if the agent reached 0 lives, or if the agent made it to the end of the level. For this task, we also allowed the agent to start episodes at many points throughout the level, which rapidly sped up training since the agent could easily experience different areas of the level.

Whilst a difficult task, once the agent first completed the level, it did not take long to start consistently completing it due to the deterministic nature of the game.

### J.2 Mario Kart Wii

The Mario Kart Wii environment had the agent play against the game’s internal opponents (on hard mode, with 12 racers including the agent), on the course Rainbow Road (with items on the 150cc speed setting). The agent had to complete 4 laps of the course to finish the race. The agent had just 4 action, including accelerate, drifting left or right, and using its item. While this limited the agent’s potential actions substantially, we found using fewer actions to dramatically accelerate training.

Rewards of +1 were given via reaching checkpoints that were scattered throughout the course (100 in total per lap). Additionally, if the agent’s speed dropped below a set threshold (65 km/h), the agent would receive a reward of -0.01 per frame. The agent would be terminated with a reward of -10 if its speed dropped below the threshold for over 80 frames, or with a reward of +10 for finishing the race, with a bonus based on the position the agent finished in. Lastly, the agent was rewarded with a +1 for using its item. Without this reward, we found the agent to often neglect using its item, likely due to many of the items only providing rewards in the long term, such as slowing down other racers or blocking incoming items far in the future. Similarly to Super Mario Galaxy, we had the agent start the episode in multiple positions around the first lap, allowing it to experience the whole track early in training.

This agent took the longest to train, taking around 160M frames to reach consistent completion. In particular, the agent took a long time to consistently complete the race due to the other racers and randomized items making the environment highly stochastic, with many rare scenarios which could cause the episode to terminate.

### J.3 Mortal Kombat

The Mortal Kombat environment put the agent in the game’s endurance mode, where the agent would sequentially fight 15 different opponents, but keep retain its health between fights, and only gain health after defeating every 3 opponents. We provided the agent with 14 actions, including: None, Left, Right, Up, Down, Axe Kick, Punch, Snap Kick, Grab, Block, Toggle Weapon, Jump Left, Jump Right, and Crouch. These actions were far from the game’s total action space, and limited the agent’s ability to perform some of the combos within the game. We limited the agent’s actions as the full action space is extremely large.

The agent was positively rewarded for damaging the opponent, and negatively rewarded for taking damage, with one taking one tenth of the health bar equating to +1 reward respectively. The episode was terminated with a reward of -10 for reaching 0 health, and +10 for defeating the 15th and final enemy.

The Mortal Kombat agent learned considerably faster than Super Mario Galaxy and Mario Kart Wii, first completing the environment in 50M frames, and getting progressively more consistent until training was stopped at 90M frames. The agent quickly learned how to dodge enemy hits, and relied heavily upon this strategy.

## Appendix K Atari-5 Regression Procedure

In our main paper ablation figure (Figure 5 ), we used the regression procedure recommended in Atari-5 ( Aitchison et al., 2023 ) . This procedure is typically used to predict the Median score across the entire 57 game Atari suite, while only needing to use 5 games. While we believe this procedure produces a valid and useful plot, we find that BTR did differ significantly from the predicted value. We opted to use both the predicted median and the IQM across the 5 games to give two easy to interpret averages. Figure K12 shows the 57 game suite’s true median, compared to the median predicted by Atari-5.

## Instructions for reporting errors