Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 1/6

Cleaned-text code points [0, 12290). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
SAMPLE -E FFICIENT REINFORCEMENT LEARNING
BY BREAKING THE REPLAY RATIO BARRIER
Pierluca D’Oro∗
Mila, Universit´e de Montr´eal
Max Schwarzer∗
Google Brain
Mila, Universit´e de Montr´eal
Evgenii Nikishin
Mila, Universit´e de Montr´eal
Pierre-Luc Bacon
Mila, Universit´e de Montr´eal
Marc G. Bellemare
Google Brain, Mila
Aaron Courville
Mila, Universit´e de Montr´eal
ABSTRACT
Increasing the replay ratio, the number of updates of an agent’s parameters per
environment interaction, is an appealing strategy for improving the sample effi-
ciency of deep reinforcement learning algorithms. In this work, we show that
fully or partially resetting the parameters of deep reinforcement learning agents
causes better replay ratio scaling capabilities to emerge. We push the limits of the
sample efficiency of carefully-modified algorithms by training them using an order
of magnitude more updates than usual, significantly improving their performance
in the Atari 100k and DeepMind Control Suite benchmarks. We then provide an
analysis of the design choices required for favorable replay ratio scaling to be
possible and discuss inherent limits and tradeoffs.
1 I NTRODUCTION
In many real world scenarios, each interaction with the environment comes at a cost, and it is
desirable for deep reinforcement learning (RL) algorithms to learn with a minimal amount of
samples (Franc ¸ois-Lavet et al., 2018). This can be naturally achieved if an algorithm is able to
leverage more computational resources during training to improve its performance. Given the online
nature of deep RL, there is a peculiar way to aim at having such behavior: to train the agent for longer,
given a dataset of experiences, before interacting with the environment again.
1 2 4 8 16 32 64 128
Replay Ratio
200
400
600
800IQM
SR-SAC
SAC
(a) DeepMind Control Suite (DMC15-500k)
1 2 4 8 16
Replay Ratio
0.2
0.4
0.6IQM
SR-SPR
SPR (b) Atari 100k
Figure 1: Scaling behavior of SAC and SR-SAC in the DeepMind Control Suite (DMC15-500k)
benchmark, and of SPR and SR-SPR in the Atari 100k benchmark (5 seeds for point for SAC and
SR-SAC, at least 20 seeds for point for SPR and SR-SPR, 95% bootstrapped C.I.).
∗Equal contribution. Correspondance to{pierluca.doro, schwarzm}@mila.quebec.
1

Published as a conference paper at ICLR 2023
A method based on this idea can be said to be scaling the replay ratio, the number of updates of an
agent’s parameters for each environment interaction. Despite generally providing limited benefit
when applied to standard baselines (Fedus et al., 2020; Kumar et al., 2021), replay ratio scaling
has been shown to bring performance improvements to well-tuned algorithms. Recent approaches
were able to achieve better sample efficiency by increasing it to higher values, up to 8 for discrete
control (Kielak, 2019) or 20 for continuous control (Chen et al., 2021; Smith et al., 2022).
In this paper, we show that it is possible, with minimal but careful modifications to model-free
algorithms mostly based on parameter resets (Ash & Adams, 2020; Nikishin et al., 2022), to reach
new levels of replay ratio scaling and push the sample efficiency limits of deep RL. Both in continuous
control, with SAC in DeepMind Control Suite (Haarnoja et al., 2018; Tassa et al., 2018), and discrete
control, with SPR in Atari 100k (Schwarzer et al., 2021a; Kaiser et al., 2020), we break the replay
ratio barrier, unlocking a training regime in which orders of magnitude of additional agent updates
can be used to increase the performance of an algorithm for a given budget of interactions with the
environment. By doing so, we obtain better aggregated scores than strong baselines, with a general
blueprint to improve sample efficiency of potentially any off-policy deep RL algorithm.
To understand how this can be feasible, it is useful to reflect on one of the most common patterns
observed in the development of deep RL algorithms (Mnih et al., 2015b). With a few exceptions,
researchers typically ground their methods on the well-established dynamic programming mathemati-
cal machinery, combining it with optimization strategies common in deep learning. However, the RL
setting is inherently different from the one in which most deep learning architectures and optimization
methods were developed. In deep RL, neural networks have to deal with dynamic datasets, whose
composition changes over the course of training; their training actively determines the value of future
inputs, but also the value of future targets. We argue that the recently identified tendency of neural
networks to lose their ability to learn and generalize from new information during training (Chaudhry
et al., 2018; Ash & Adams, 2020; Berariu et al., 2021; Igl et al., 2021; Dohare et al., 2022; Lyle et al.,
2022a;b; Nikishin et al., 2022), against which most RL methods deploy no countermeasures, has
been the main roadblock in achieving better sample efficiency through replay ratio scaling.
After presenting and evaluating our algorithmic solution leading to better replay ratio scaling, we
discuss some of the aspects of thinking about deep RL algorithms under the lens of this paradigm.
We show some examples of algorithm design decisions important, or not important, for effective
replay ratio scaling to be possible, with particular attention to the role of online interaction. Then, we
visualize in an explicit way the data-computations tradeoff implied by this approach and, after having
shown the potential of replay ratio scaling, we discuss its inherent limits.
2 R ELATED WORK
Loss of Ability to Learn and Generalize in Neural Networks A growing body of evidence
suggests that artificial neural networks lose their ability to learn and generalize during training. The
phenomenon is not clearly visible when learning with a static dataset on a fixed task, but it starts
appearing when the data distribution changes. In the continual learning setting, an alleviation of the
problem by partially resetting the network parameters already provides a consistent improvement (Ash
& Adams, 2020). Berariu et al. (2021) provides an in-depth study of how this phenomenon happens,
including how many training updates are required for the performance of a network on future tasks to
be unrecoverably damaged. The phenomenon becomes even more prominent in deep RL, where it has
been identified in multiple settings. In the context of on-policy algorithms, it has been investigated as
a consequence of transient non-stationarity and mitigated via self-distillation (Igl et al., 2021); in
off-policy RL, it has been studied under the name of capacity loss (Lyle et al., 2022a), counteracted
by the use of auxiliary tasks; in the sparse reward setting, it has been mitigated by post-training policy
distillation (Lyle et al., 2022b). To address what they call loss of plasticity, Dohare et al. (2022)
proposes a variation of backpropagation compatible with continual learning, also applying it to the
continual RL context. In this paper, we primarily leverage a periodic hard resetting method (Zhou
et al., 2022), as investigated in Nikishin et al. (2022) to address the primacy bias phenomenon. Our
work demonstrates that addressing this phenomenon allows for increased sample efficiency by scaling
the replay ratio to much higher values than other model-free methods. We report in Appendix A a
more precise summary and glossary of the different related definitions from previous work.
2

Published as a conference paper at ICLR 2023
Scaling in Deep and Reinforcement Learning The topic of understanding and exploiting the scaling
behavior of a deep learning algorithm’s performance with respect to the amount of resources used for
training has recently gained attention. Hestness et al. (2017) pioneered the idea of empirically studying
and predicting performance when increasing a model’s size, and subsequent work investigated scaling
with respect to both model and dataset size, as well as training time (Kaplan et al., 2020; Bahri et al.,
2021; Djolonga et al., 2021). Recent work in language modeling has also highlighted the importance
of having high-quality data and the right training setup for efficient scaling to be possible (Hoffmann
et al., 2022). In RL, scaling with respect to model size has been investigated in the offline setting
for decision transformers (Lee et al., 2022) and with respect to planning-time in model-based
RL (Hamrick et al., 2021). For what concerns replay ratio scaling, moderately increasing the replay
ratio for standard baselines has been shown to be a competitive data-efficient baseline for both
discrete and continuous control when compared to model-based RL methods (Holland et al., 2018;
Van Hasselt et al., 2019; Kielak, 2019; D’Oro & Ja´skowski, 2020), despite clear limitations (Kumar
et al., 2021). Recent approaches in continuous control leveraged high replay ratios as a strategy
to improve sample efficiency through the use of ensembles of value functions (Chen et al., 2021;
Hiraoka et al., 2022; Wu et al., 2022) or normalization strategies (Smith et al., 2022); we argue that
explicitly alleviating the progressive loss of ability to learn and generalize pushes the replay ratio
scaling capabilities much further than those techniques can achieve.
3 E FFECTIVE REPLAY RATIO SCALING WITH RESETS
Most off-policy deep RL algorithms make use of a replay buffer (Lin, 1992) for storing transitions
encountered over (a window of) an agent’s lifespan. At a fixed frequency, such methods sample a
batch of transitions from the buffer, update the parameters of the agent by following the gradient
of some loss function, and let the agent interact again with the environment before adding new
experience to the buffer. The number of agent updates per environment step is usually called replay
ratio1 (Wang et al., 2016; Fedus et al., 2020), and most standard algorithms are trained with a value
around 1 (Mnih et al., 2015a; Haarnoja et al., 2018). It is natural to view increasing the replay ratio
beyond these values as a way to improve sample efficiency. For ease of discussion, we now explicitly
state and give a name to this idea, which has been an object of interest in previous studies (Van Hasselt
et al., 2019; Kumar et al., 2021).
Replay Ratio Scaling
Change in an agent’s performance caused by doing more updates for a fixed number of
environment interactions.
This definition does not have any positive connotation per se; any deep RL algorithm will have a
certain replay ratio scaling behavior, and a desirable property for an algorithm is to have particularly
favorable replay ratio scaling, so that its performance can improve by increasing the replay ratio.
In contrast to other performance scaling properties analyzed for deep learning algorithms (Kaplan
et al., 2020), replay ratio scaling is intertwined with the online RL paradigm: if the agent has a
significantly better data-collection policy due to more training, the next collected sample will be
potentially different with respect to the one collected if doing less training before the interaction; by
this virtue, also future learning will be directly impacted by the presence of different data in the replay
buffer. In other words, this type of scaling can only be understood by considering the interaction of
an agent with an environment: training more on a small dataset of interactions, without any further
collection of data, will eventually lead to challenges associated to off-policy learning (Ostrovski et al.,
2021); but training more while the data is collected can drastically change the stream of incoming
data and the overall learning dynamics.
Given its appeal, what are the limiting factors to increasing the replay ratio? We argue that the
main factor inhibiting effective replay ratio scaling in existing deep RL algorithms has been the
progressive loss of the ability to learn and generalize in neural networks(Dohare et al., 2022; Lyle
et al., 2022b; Nikishin et al., 2022). It has been shown that this property hinders a neural network’s
performance under task switches (Ash & Adams, 2020; Berariu et al., 2021) and, from the perspective
1Related quantities are also known as update-to-data (UTD) ratio (Chen et al., 2021; Smith et al., 2022).
3