Bigger, Better, Faster: Human-level Atari with human-level efficiency — part 1/3

Cleaned-text code points [0, 15973). Bibliography omitted here; source.md includes it.

# Bigger, Better, Faster: Human-level Atari with human-level efficiency

###### Abstract

We introduce a value-based RL agent, which we call BBF, that achieves super-human performance in the Atari 100K benchmark. BBF relies on scaling the neural networks used for value estimation, as well as a number of other design choices that enable this scaling in a sample-efficient manner. We conduct extensive analyses of these design choices and provide insights for future work. We end with a discussion about updating the goalposts for sample-efficient RL research on the ALE. We make our code and data publicly available.

###### Keywords:

## 1 Introduction

Deep reinforcement learning (RL) has been central to a number of successes including playing complex games at a human or super-human level, such as OpenAI Five ( Berner et al., 2019 ) , AlphaGo ( Silver et al., 2016 ) , and AlphaStar ( Vinyals et al., 2019 ) , controlling nuclear fusion plasma in a tokomak ( Degrave et al., 2022 ) , and integrating human feedback for conversational agents ( Ouyang et al., 2022 ) . The success of these RL methods has relied on large neural networks and an enormous number of environment samples to learn from – a human player would require tens of thousands of years of game play to gather the same amount of experience as OpenAI Five or AlphaGo. It is plausible that such large networks are necessary for the agent’s value estimation and/or policy to be expressive enough for the environment’s complexity, while large number of samples might be needed to gather enough experience so as to determine the long-term effect of different action choices as well as train such large networks effectively. As such, obtaining human-level sample efficiency with deep RL remains an outstanding goal.

Although advances in modern hardware enable using large networks, in many environments it may be challenging to scale up the number of environment samples, especially for real-world domains such as healthcare or robotics. While approaches such as offline RL leverage existing datasets to reduce the need for environment samples ( Agarwal et al., 2020 ) , the learned policies may be unable to handle distribution shifts when interacting with the real environment ( Levine et al., 2020 ) or may simply be limited in performance without online interactions ( Ostrovski et al., 2021 ) .

Thus, as RL continues to be used in increasingly challenging and sample-scarce scenarios, the need for scalable yet sample-efficient online RL methods becomes more pressing. Despite the variability in problem characteristics making a one-size-fits-all solution unrealistic, there are many insights that may transfer across problem domains. As such, methods that achieve “state-of-the-art” performance on established benchmarks can provide guidance and insights for others wishing to integrate their techniques.

In this vein, we focus on the Atari 100K benchmark ( Kaiser et al., 2020 ) , a well-known benchmark where agents are constrained to roughly 2 hours of game play, which is the amount of practice time the professional tester was given before human score evaluation. While human-level efficiency has been obtained by the model-based EfficientZero agent ( Ye et al., 2021 ) , it has remained elusive for model-free RL agents. To this end, we introduce BBF, a model-free RL agent that achieves super-human performance – interquartile mean ( Agarwal et al., 2021b ) human-normalized score above 1.0 1.0 – while being much more computationally efficient than EfficientZero (Figure 2 ). Achieving this level of performance required a larger network than the decade-old 3-layer CNN architecture ( Mnih et al., 2013 ) , but as we will discuss below, scaling network size is not sufficient on its own. We discuss and analyze the various techniques and components that are necessary to train BBF successfully and provide guidance for future work to build on our findings. Finally, we propose moving the goalpost for sample-efficient RL research on the ALE.

## 2 Background

The RL problem is generally described as a Markov Decision Proces (MDP) ( Puterman, 2014 ) , defined by the tuple ⟨ 𝒮 , 𝒜 , 𝒫 , ℛ ⟩ \langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\rangle , where 𝒮 \mathcal{S} is the set of states, 𝒜 \mathcal{A} is the set of available actions, 𝒫 : 𝒮 × 𝒜 → Δ ⁡ ( 𝒮 ) \mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S}) 1 1 1 Δ ⁡ ( S ) \Delta(S) denotes a distribution over the set S S . is the transition function, and ℛ : 𝒮 × 𝒜 → ℝ \mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function. Agent behavior in RL can be formalized by a policy π : 𝒮 → Δ ⁡ ( 𝒜 ) \pi:\mathcal{S}\rightarrow\Delta(\mathcal{A}) , which maps states to a distribution of actions. The value of π \pi when starting from s ∈ 𝒮 s\in\mathcal{S} is defined as the discounted sum of expected rewards: V π ​ ( s ) := 𝔼 π , 𝒫 ​ [ ∑ t = 0 ∞ γ t ​ r ​ ( s t , a t ) ] V^{\pi}(s):=\mathbb{E}_{\pi,\mathcal{P}}\left[\sum_{t=0}^{\infty}\gamma^{t}r\left(s_{t},a_{t}\right)\right] , where γ ∈ [ 0 , 1 ) \gamma\in[0,1) is a discount factor that encourages the agent to accumulate rewards sooner rather than later. The goal of an RL agent is to find a policy π ∗ \pi^{*} that maximizes this sum: V π ∗ ≥ V π V^{\pi^{*}}\geq V^{\pi} for all π \pi .

While there are a number of valid approaches ( Sutton & Barto, 1998 ) , in this paper we focus on model-free value-based methods. Common value-based algorithms approximate the Q ∗ Q^{*} -values, defined via the Bellman recurrence: Q ∗ ​ ( s , a ) := R ⁡ ( s , a ) + γ ​ 𝔼 s ′ ∼ 𝒫 ⁡ ( s , a ) ​ [ max a ′ ∈ 𝒜 ⁡ Q ∗ ​ ( s ′ , a ′ ) ] Q^{*}(s,a):=R(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim\mathcal{P}(s,a)}[\max_{a^{\prime}\in\mathcal{A}}Q^{*}(s^{\prime},a^{\prime})] . The optimal policy π ∗ \pi^{*} can then be obtained from the optimal state-action value function Q ∗ Q^{*} as π ∗ ​ ( x ) := max a ∈ 𝒜 ⁡ Q ∗ ​ ( s , a ) \pi^{*}(x):=\max_{a\in\mathcal{A}}Q^{*}(s,a) . A common approach for learning Q ∗ Q^{*} is the method of temporal differences, optimizing the Bellman temporal difference: ( r ⁡ ( s t , a t ) + γ ​ max a t + 1 ⁡ Q ⁡ ( s t + 1 , a t + 1 ) ) − Q ⁡ ( s t , a t ) . \left(r\left({s}_{t},{a}_{t}\right)+\gamma\max_{{a}_{t+1}}Q\left({s}_{t+1},{a}_{t+1}\right)\right)-Q\left({s}_{t},{a}_{t}\right). We often refer to ( r ⁡ ( s t , a t ) + γ ​ max a t + 1 ⁡ Q ⁡ ( s t + 1 , a t + 1 ) ) \left(r\left({s}_{t},{a}_{t}\right)+\gamma\max_{{a}_{t+1}}Q\left({s}_{t+1},{a}_{t+1}\right)\right) as the Bellman target .

Mnih et al. (2015a) introduced the agent DQN by combining temporal-difference learning with deep networks, and demonstrated its capabilities in achieving human-level performance on the Arcade Learning Environment (ALE) ( Bellemare et al., 2013 ) . They used a network consisting of 3 convolutional layers and 2 fully connected layers, parameterized by θ \theta , to approximate Q Q (denoted as Q θ Q_{\theta} ). We will refer to this architecture as the CNN architecture. Most of the work in value-based agents is built on the original DQN agent, and we discuss a few of these advances below which are relevant to our work.

Hessel et al. (2018) combined six components into a single agent they called Rainbow : prioritized experience ( Schaul et al., 2016 ) , n n -step learning ( Sutton, 1988 ) , distributional RL ( Bellemare et al., 2017 ) , double Q-learning ( van Hasselt et al., 2016 ) , dueling architecture ( Wang et al., 2016 ) and NoisyNets ( Fortunato et al., 2018b ) . Hessel et al. (2018) and Ceron & Castro (2021) both showed that Multi-step learning is one of the most crucial components of Rainbow, in that removing it caused a large drop in performance.

In n n -step learning, instead of computing the temporal difference error using a single-step transition, one can use n n -step targets instead ( Sutton, 1988 ) , where for a trajectory ( s 0 , a 0 , r 0 , s 1 , a 1 , ⋯ ) (s_{0},a_{0},r_{0},s_{1},a_{1},\cdots) and update horizon n n : R t ( n ) := ∑ k = 0 n − 1 γ k ​ r t + k + 1 R_{t}^{(n)}:=\sum_{k=0}^{n-1}\gamma^{k}r_{t+k+1} , yielding the multi-step temporal difference: R t ( n ) + γ n ​ max a ′ ​ Q θ ​ ( s t + n , a ′ ) − Q θ ​ ( s t , a t ) R_{t}^{(n)}+\gamma^{n}\max_{a^{\prime}}Q_{{\theta}}(s_{t+n},a^{\prime})-Q_{\theta}(s_{t},a_{t}) .

Most modern RL algorithms store past experiences in a replay buffer that increases sample efficiency by allowing the agent to use samples multiple times during learning, and to leverage modern hardware such as GPUs and TPUs by training on sampled mini-batches. An important design parameter is the replay ratio , the ratio of learning updates to online experience collected ( Fedus et al., 2020a ) . It is worth noting that DQN uses a replay ratio of 0.25 0.25 (4 environment interactions for every learning update), while some sample-efficient agents based on Rainbow use a value of 1 1 .

Nikishin et al. (2022) showed that the networks used by deep RL agents have a tendency to overfit to early experience, which can result in sub-optimal performance. They proposed a simple strategy consisting of periodically resetting the parameters of the final layers of DQN-based agents to counteract this. Building on this promising work, D’Oro et al. (2023) added a shrink-and-perturb technique for the parameters of the convolutional layers, and showed that this allowed them to scale the replay ratio to values as high as 16, with no performance degradation.

## 3 Related Work

Sample-Efficient RL on ALE: Sample efficiency has always been an import aspect of evaluation in RL, as it can often be expensive to interact with an environment. Kaiser et al. (2020) introduced the Atari 100K benchmark, which has proven to be useful for evaluating sample-efficiency, and has led to a number of recent advances.

Kostrikov et al. (2020) use data augmentation to design a sample-efficient RL method, DrQ, which outperformed prior methods on Atari 100K. Data-Efficient Rainbow (DER) ( Van Hasselt et al., 2019 ) and DrQ( ϵ \epsilon ) ( Agarwal et al., 2021b ) simply modified the hyperparameters of existing model-free algorithms to exceed the performance of existing methods without any algorithmic innovation.

Schwarzer et al. (2021) introduced SPR, which builds on Rainbow ( Hessel et al., 2017 ) and uses a self-supervised temporal consistency loss based on BYOL ( Grill et al., 2020 ) combined with data augmentation. SR-SPR ( Schwarzer et al., 2021 ) combines SPR with periodic network resets to achieve state-of-the-art performance on the 100K benchmark. Ye et al. (2021) used a self-supervised consistency loss similar to SPR ( Chen & He, 2021 ) .

EfficientZero ( Ye et al., 2021 ) , an efficient variant of MuZero ( Schrittwieser et al., 2020 ) , learns a discrete-action latent dynamics model from environment interactions, and selects actions via lookahead MCTS in the latent space of the model. Micheli et al. (2023) introduce IRIS, a data-efficient agent that learns in a world model composed of an autoencoder and an auto-regressive Transformer.

Scaling in Deep RL: Deep neural networks are useful for extracting features from data relevant for various downstream tasks. Recently, there has been interest in the scaling properties of neural network architectures, as scaling model size has led to commensurate performance gains in applications ranging from language modelling to computer vision.

Based on those promising gains, the deep RL community has begun to investigate the effect of increasing the model size of the function approximator. Sinha et al. (2020) and Ota et al. (2021) explore the interplay between the size, structure, and performance of deep RL agents to provide intuition and guidelines for using larger networks. Kumar et al. (2022) find that with ResNets (up to 80 million parameter networks) combined with distributional RL and feature normalization, offline RL can exhibit strong performance that scales with model capacity. Taiga et al. (2023) show that generalization capabilities on the ALE benefit from higher capacity networks, such as ResNets. Cobbe et al. (2020) and Farebrother et al. (2023) demonstrate benefits when scaling the number of features in each layer of the ResNet architecture used by Impala ( Espeholt et al., 2018 ) , which motivated the choice of feature width scaling in this work. Different from these works, our work focus on improving sample-efficiency in RL as opposed to offline RL or improving generalization in RL.

In the context of online RL, Hafner et al. (2023) demonstrate that increased dynamics model size, trained via supervised learning objectives, leads to monotonic improvements in the agent’s final performance. Recently, AdA ( Team et al., 2023 ) scales transformer encoder for a Muesli agent up to 265M parameters. Interestingly, AdA required distillation from smaller models to bigger models to achieve this scaling, in the spirit of reincarnating RL ( Agarwal et al., 2022 ) . However, it is unclear whether findings from above papers generalize to scaling typical value-based deep RL methods in sample-constraint settings, which we study in this work.

## 4 Method

The question driving this work is: How does one scale networks for deep RL when samples are scarce? To investigate this, we focus on the well-known Atari 100K benchmark ( Kaiser et al., 2020 ) , which includes 26 Atari 2600 games of diverse characteristics, where the agent may perform only 100K environment steps, roughly equivalent to two hours of human gameplay 2 2 2 100k steps (400k frames) at 60 FPS is 111 minutes. . As we will see, näively scaling networks can rarely maintain performance, let alone improve it.

The culmination of our investigation is the Bigger, Better, Faster agent, or BBF in short, which achieves super-human performance on Atari 100K with about 6 hours on single GPU. Figure 2 demonstrates the strong performance of BBF relative to some of the best-performing Atari 100K agents: EfficientZero ( Ye et al., 2021 ) , SR-SPR ( D’Oro et al., 2023 ) , and IRIS ( Micheli et al., 2023 ) . BBF consists of a number of components, which we discuss in detail below.

Our implementation is based on the Dopamine framework ( Castro et al., 2018 ) and uses mostly already previously-released components. For evaluation, we use rliable ( Agarwal et al., 2021b ) and in particular, the interquartile mean (IQM) metric, which is the average score of the middle 50% runs combined across all games and seeds.

### Base agent.

BBF uses a modified version of the recently introduced SR-SPR agent ( D’Oro et al., 2023 ) . Through the use of periodic network resets, SR-SPR is able to scale up its replay ratio (RR) to values as high as 16, yielding better sample efficiency. For BBF, we use RR=8 in order to balance the increased computation arising from our large network. Note that this is still very high relative to existing Atari agents – Rainbow and its data-efficient variant DER ( Van Hasselt et al., 2019 ) use RR=0.25 and 1, respectively.

As we expect that many users will not wish to pay the computational costs of running at replay ratio 8, we also present results for BBF and ablations at replay ratio 2 (matching SPR). For all experiments we state which replay ratio is being used in the captions.

### Harder resets.

The original SR-SPR agent ( D’Oro et al., 2023 ) used a shrink-and-perturb method for the convolutional layers where parameters were only perturbed 20% of the way towards a random target, while later layers were fully reset to a random initialization. An interesting result of our investigation is that using harder resets of the convolutional layers yields better performance. In our work, we move them 50% towards the random target, resulting in a stronger perturbation and improving results (see Figure 5 ). This may be because larger networks need more regularization, as we find that they reduce loss faster ( Figure A.1 ).

### Larger network.