Bigger, Better, Faster: Human-level Atari with human-level efficiency — part 2/3

Cleaned-text code points [15973, 31924). Bibliography omitted here; source.md includes it.

Scaling network capacity is one of the motivating factors for our work. As such, we adopt the Impala-CNN ( Espeholt et al., 2018 ) network, a 15-layer ResNet, which has previously led to substantial performance gains over the standard 3-layer CNN architecture in Atari tasks where large amounts of data are available ( Agarwal et al., 2022 ; Schmidt & Schmied, 2021 ) . Additionally, BBF scales the width of each layer in Impala-CNN by 4 × \times . In Figure 3 , we examine how the performance of SPR, SR-SPR and BBF varies with different choices of scaling width, for both the ResNet and original CNN architectures. Interestingly, although the CNN has roughly 50% more parameters than the ResNet at each scale level, the ResNet yields better performance at all scaling levels for both SR-SPR and BBF.

What stands out from Figure 3 is that BBF’s performance continues to grow as width is increased, whereas SR-SPR seems to peak at 1-2 × \times (for both architectures). Given that ResNet BBF performs comparably at 4 × \times and 8 × \times , we chose 4 × \times to reduce the computational burden. While reducing widths beyond this could further reduce computational costs, this comes at the cost of increasingly sharp reductions in performance for all methods tested.

### Receding update horizon.

One of the surprising components of BBF is the use of an update horizon ( n n -step) that decreases exponentially from 10 to 3 over the first 10K gradient steps following each network reset. Given that we follow the schedule of D’Oro et al. (2023) and reset every 40k gradient steps, the annealing phase is always 25% of training, regardless of the replay ratio. As can be seen in Figure 5 , this yields a much stronger agent than using a fixed value of n = 3 n=3 , which is default for Rainbow, or n = 10 n=10 , which is typically used by Atari 100K agents like SR-SPR.

Our n n -step schedule is motivated by the theoretical results of Kearns & Singh (2000) – larger values of n n -step leads to faster convergence but to higher asymptotic errors with respect to the optimal value function. Thus, selecting a fixed value of n n corresponds to a choice between having either rapid convergence to a worse asymptote, or slower convergence to a better asymptote. As such, our exponential annealing schedule closely resembles the optimal decreasing schedule for n n -step derived by Kearns & Singh (2000) .

### Increasing discount factor.

Motivated by findings that increasing the discount factor γ \gamma during learning improves performance ( François-Lavet et al., 2015 ) , we increase γ \gamma from γ 1 \gamma_{1} to γ 2 \gamma_{2} , following the same exponential schedule as for the update horizon. Note that increasing γ \gamma has the effect of progressively giving more weights to delayed rewards. We choose γ 1 = 0.97 \gamma_{1}=0.97 , slightly lower than the typical discount used for Atari, and γ 2 = 0.997 \gamma_{2}=0.997 as it is used by MuZero ( Schrittwieser et al., 2021 ) and EfficientZero ( Ye et al., 2021 ) . As with the update horizon, Figure 5 demonstrates that this strategy outperforms using a fixed value.

### Weight decay.

We incorporate weight decay in our agent to curb statistical overfitting, as BBF is likely to overfit with its high replay ratio. To do so, we use the AdamW optimizer ( Loshchilov & Hutter, 2019 ) with a weight decay value of 0.1 0.1 . Figure 5 suggests the gains from adding weight decay are significant and increase with replay ratio, indicating that the regularizing effects of weight decay enhance replay ratio scaling with large networks.

### Removing noisy nets.

Finally, we found that NoisyNets ( Fortunato et al., 2018a ) , used in the original SPR ( Schwarzer et al., 2021 ) and SR-SPR, did not improve performance. This could be due to NoisyNets causing over-exploration due to increased policy churn ( Schaul et al., 2022 ) from added noise during training, or due to added variance in optimization, and we leave investigation to future work. Removing NoisyNets results in large computational and memory savings, as NoisyNets creates duplicate copies of the weight matrices for the final two linear layers in the network, which contain the vast majority of all parameters: turning on NoisyNets increases the FLOPs per forward pass and the memory footprint by a factor of 2.5 × \times and 1.6 × \times , respectively, which both increases runtime and reduces the number of training runs that can be run in parallel on a single GPU. Removing NoisyNets is thus critical to allowing BBF to achieve reasonable compute efficiency despite its larger networks. We found that this decision had no significant impact on task performance (see Figure A.2 in appendix).

## 5 Analysis

In light of the importance of BBF’s components, we discuss possible consequences of our findings for other algorithms.

### The importance of self-supervision.

One unifying aspect of the methods compared in Figure 2 is that they all use some form of self-supervised objective. In sample-constrained scenarios, like the one considered here, relying on more than the temporal-difference backups is likely to improve learning speed, provided the self-supervised losses are consistent with the task at hand. We test this by removing the SPR objective (inherited from SR-SPR) from BBF, and observe a substantial performance degredation (see Figure 5 ). It is worth noting that EfficientZero uses a self-supervised objective that is extremely similar to SPR, a striking commonality between BBF and EfficientZero.

### Sample efficiency via more gradient steps.

The original DQN agent ( Mnih et al., 2015b ) has a replay ratio of 0.25 0.25 , which means a gradient update is performed only after every 4 environment steps. In low-data regimes, it is more beneficial to perform more gradient steps, although many algorithms cannot benefit from this without additional regularization ( D’Oro et al., 2023 ) . As Figure 6 confirms, performance of BBF grows with increasing replay ratio in the same manner as its base algorithm, SR-SPR. More strikingly, we observe a linear relationship between the performance of BBF and SR-SPR across all replay ratios, with BBF performing roughly 0.45 IQM above SR-SPR. While the direction of this relationship is intuitive given the network scaling introduced by BBF, its linearity is unexpected, and further investigation is needed to understand the nature of the interaction between replay ratio and network scaling.

One interesting comparison to note is that, although EfficientZero uses a replay ratio of 1.2, they train with a batch size that is 8 times larger than ours. Thus, their effective replay ratio is comparable to ours.

### The surprising importance of target networks

Many prior works on Atari 100k, such as DrQ and SPR ( Kostrikov et al., 2020 ; Schwarzer et al., 2021 ) chose not to use target networks, seeing them unnecessary or an impediment to sample efficiency. Later, D’Oro et al. (2023) re-introduced an exponential moving average target network, used both for training and action selection, and found that it improved performance somewhat, especially at high replay ratios. With network scaling, however, using a target network becomes a critical, but easy-to-overlook, component of the algorithm at all replay ratios (see Figure 7 ).

### Reset Strength

Increasing the replay ratio is in general challenging, as explored by Fedus et al. (2020b) and Kumar et al. (2020) . Periodic resetting, as suggested by Nikishin et al. (2022) and D’Oro et al. (2023) , has proven effective to enable scaling to larger replay ratios, quite possibly a result of reduced overfitting. This is confirmed in Figure 5 , where the importance of resets is clear. Further, Figure 5 and Figure 8 demonstrate the added benefit of more aggressive perturbations, relative to SR-SPR.

### Scale is not enough on its own.

The naïve approach of simply scaling the capacity of the CNN used by SR-SPR turns out to be insufficient to improve performance. Instead, as Figure 3 shows, the performance of SR-SPR collapses as network size increases. As discussed in section 4 , it is interesting to observe that the smaller Impala-CNN ResNet (as measured by number of parameters and FLOPs) yields stronger performance at all width scales.

### Computational efficiency.

As machine learning methods become more sophisticated, an often overlooked metric is their computational efficiency. Although EfficientZero trains in around 8.5 hours, it requires about 512 CPU cores and 4 distributed GPUs. IRIS uses half of an A100 GPU for a week per run. SR-SPR, at its highest replay ratio of 16, uses 25% of an A100 GPU and a single CPU for roughly 24 hours. Our BBF agent at replay ratio 8 takes only 10 hours with a single CPU and half of an A100 GPU. Thus, measured by GPU-hours, BBF provides the best trade-off between performance and compute (see Figure 2 ).

## 6 Revisiting the Atari 100k benchmark

A natural question is whether there is any value in continuing to use the Atari 100K benchmark, given that both EfficientZero and BBF are able to achieve human-level performance (IQM ≥ 1.0 \geq 1.0 ) in just 100K steps. When considering this, it is important to remember that IQM is an aggregate measure. Indeed, in the left panel of Figure 4 we can see there is still room for improvement with regards to the optimality gap , which measures the amount by which each algorithm fails to meet a minimum score of 1.0 1.0 ( Agarwal et al., 2021b ) . Specifically, despite monotonic progress over the years, no agent is yet able to achieve human-level performance on all 26 games, which would yield an optimality gap of zero, without using dramatically more than two hours of data ( Kapturowski et al., 2022 ) .

Overfitting on Atari 100K . Another important consideration is that the Atari 100K benchmark uses only 26 of the 55 games from the full ALE suite, and it does not include sticky actions 3 3 3 With 25% probability, the environment will execute the previous action again, instead of the agent’s executed action. ( Machado et al., 2018 ) , which may make tasks significantly harder. Since we extensively benchmark BBF on Atari 100K, this raises the question of whether BBF works well on unseen Atari games and with sticky actions.

Fortunately, it does. In Figure 9 , we compare the performance of BBF on all 55 games with sticky actions, and show that sticky action do not significantly harm performance. We do observe that the held-out games not included in the Atari 100k set are significantly more challenging than the 26 Atari 100k games (see Figure 11 ) – but this is even more true for baselines such as DQN (Nature) that did not use Atari 100k. Furthermore, as shown in Figure 8 , we find that BBF’s design choices generally provide even more benefit on these held-out games, possibly due to their increased difficulty.

### New Frontiers

In fact, BBF works so well on the standard Atari setting that it is able to roughly match DQN’s performance at 256 hours with only two hours of gameplay time ( Figure 10 ). This suggests a clear new milestone for the community: can we match Rainbow’s final performance with just two hours of gameplay? To facilitate future research toward this, we release scores on the set of 55 games with sticky actions, at various scales and replay ratios.

### Data Scaling

Prior works have indicated that many sample-efficient RL algorithms plateau in performance when trained for longer than they were originally designed for ( Agarwal et al., 2022 , e.g.,) . To examine this phenomenon, we train BBF, SPR and SR-SPR at replay ratio 2 out to one million environment steps ( Figure 12 ), keeping all parameters unchanged (including conducting resets as normal past 100k steps). We observe that SPR and SR-SPR experience stagnating performance, with SR-SPR’s advantage over SPR fading by 1M steps. BBF, however, remains consistently ahead of both, matching DQN’s final performance before 200k environment steps and matching Rainbow’s performance at 20M environment steps by 1M steps. We note that this experiment costs only 2.5 times more than training at replay ratio 8 to 100k steps, so we encourage other researchers to run similar experiments.

Additionally, we note in Figure 13 that it is possible to compare algorithms even with extremely small amounts of data, such as 20k or 50k steps, by which point BBF at replay ratio 2 (even with sticky actions enabled) outperforms most recently proposed algorithms ( Robine et al., 2023 ; Micheli et al., 2023 ; Hafner et al., 2023 ) , which did not use sticky actions. We thus suggest that compute-constrained groups consider this setting, as training BBF at replay ratio 2 for 40k environment steps takes only half of an A100 for 1 hour.

## 7 Discussion and Future Work

We introduced BBF, an algorithm that is able to achieve super-human level performance on the ALE with only 2-hours of gameplay. Although BBF is not the first to achieve this milestone, it is able to do so in a computationally efficient manner. Furthermore, BBF is able to better handle the scaling of networks and replay ratios, which are crucial for network expressivity and learning efficiency. Indeed, Figure 3 suggests that BFF is better-able to use over-parameterized networks than prior agents.

The techniques necessary to achieve this result invite a number of research questions for future work. Large replay ratios are a key element of BFF’s performance, and the ability to scale them is due to the periodic resets incorporated into the algorithm. These resets are likely striking a favourable balance between catastrophic forgetting and network plasticity. An interesting avenue for future research is whether there are other mechanisms for striking this balance that perhaps are more targeted (e.g. not requiring resetting the full network, as was recently explored by Sokar et al. (2023) ). We remarked on the fact that all the methods compared in Figure 2 use a form of self-supervision. Would other self-supervised losses (e.g. ( Mazoure et al., 2020 ; Castro et al., 2021 ; Agarwal et al., 2021a ) ) produce similar results? Surprisingly, Li et al. (2022) argue that self-supervision from pixels does not improve performance; our results seem to contradict this finding.

Recent attention has shifted towards more realistic benchmarks ( Fan et al., 2022 ) but such benchmarks exclude the majority of researchers outside certain resource-rich labs, and may require an alternative paradigm ( Agarwal et al., 2022 ) . One advantage of the Atari 100k benchmark is that, while still a challenging benchmark, it is relatively cheap compared to other benchmarks of similar complexity. However, despite its apparent saturation, scientific progress can still be made on this benchmark if we expand its scope. We hope our work provides a solid starting point for this.

Overall, we hope that our work inspires other researchers to continue pushing the frontier of sample efficiency in deep RL forward, to ultimately reach human-level performance across all tasks with human-level or superhuman efficiency.

## Appendix A Additional Results

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.