Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 2/6

Cleaned-text code points [12290, 24711). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
0.0 0.2 0.4 0.6 0.8 1.0
Environment Steps (×106)
0
200
400
600
800IQM
SAC
REDQ
SR-SAC
DDPG
DMC15-500k
Method IQM Median Mean
SR-SAC 740 (642, 818) 667 (573, 742) 658 (589, 722)
REDQ 511 (440, 577) 493 (442, 544) 494 (452, 534)
SAC 391 (334, 448) 424 (376, 468) 424 (386, 461)
DDPG 392 (334, 445) 410 (364, 454) 408 (371, 442)
DMC15-1M
Method IQM Median Mean
SR-SAC 805 (726, 867) 729 (628, 790) 710 (643, 775)
REDQ 586 (514, 649) 546 (490, 596) 539 (498, 576)
SAC 535 (467, 597) 525 (471, 567) 519 (480, 557)
DDPG 514 (450, 572) 492 (440, 540) 489 (450, 526)
Figure 2 & Table 1: Performance of SR-SAC and of standard baselines on the DMC15 benchmark.
(5 seeds for SR-SAC, 20 for all other algorithms, 95% bootstrapped C.I.).
of a neural network employed by the agent, what is deep RL if not a long sequence of related but
distinct tasks (Dabney et al., 2021)?
Recent studies showed that, even under smooth task changes, the more training has been done on
a previous task, the worse the performance will eventually be in a new task (Ash & Adams, 2020;
Berariu et al., 2021). Since higher replay ratio correspond to an increased amount of training, this
gives a natural explanation to the limit in increasing it. The ability to learn and generalize can,
however, be restored. For instance, Nikishin et al. (2022) periodically reset the network’s parameters,
with a frequency that is fixed with respect to the number of environment steps. In this work, we
argue that the key to surprisingly effective replay ratio scaling is a periodic restoration of the ability
to learn and generalize of the network, via partial (Ash & Adams, 2020) or total (Nikishin et al.,
2022) resets of its parameters, with a reset frequency that only depends on the number of updates
and thus implicitly also on the replay ratio. This means the more an algorithm updates its neural
networks, the more frequent the restoration of its ability to learn and generalize will be, leading to
better performance, as we now show in practice.
4 R EPLAY RATIO SCALING DRASTICALLY IMPROVES SAMPLE EFFICIENCY
We apply two different reset strategies to two standard continuous control and discrete control algo-
rithms and study their replay ratio scaling behavior. We consider Soft Actor-Critic (SAC) (Haarnoja
et al., 2018), which optimizes an actor and a critic by maximizing policy entropy alongside the
environment’s reward, and SPR (Schwarzer et al., 2021a), a model-free DQN-based reinforcement
learning algorithm that augments a sample-efficient variant of Rainbow (Van Hasselt et al., 2019) with
a model-based latent dynamics prediction objective designed to improve representation learning in
the low-data regime. The two curves in Figure 1 show that it is possible, with the same algorithm, to
almost double the performance for the same number of environment steps, by just varying the replay
ratio. We call the modified versions of these two algorithms Scaled-by-Resetting SAC (SR-SAC) and
Scaled-by-Resetting SPR (SR-SPR). In the rest of this section, we are going to describe the precise
the details of the reset strategies that we employ for the two algorithms, as well as the benchmarks to
which they are applied, by describing our decisions first in continuous control and then in discrete
control. For evaluation and comparisons, we follow the protocol suggested by Agarwal et al. (2021).
4.1 C ONTINUOUS CONTROL
The DMC15 Benchmark To appropriately compare the performance of different algorithms, we
consider a benchmark based on 15 environments from DeepMind Control Suite (Tassa et al., 2018).
Our selection of tasks, reported in Table 6, is a set for which discussing sample efficiency is sensible
(i.e., neither immediately solvable nor unsolvable by common deep RL algorithms). For ease of
comparison, we specialize the benchmark to DMC15-500k, in which 5 × 105 interactions with the
environment are allowed, and DMC15-1M, in which 106 interactions are allowed.
4

Published as a conference paper at ICLR 2023
Reset Strategy We adapt the approach of Nikishin et al. (2022), and completely reset all the agent
parameters every 2.56 × 106 of its updates. This lets us avoid individually specifying the moments at
which resets should happen for different replay ratios. In terms of environment steps, resets will just
occur more often at higher replay ratios. For instance, for replay ratio 128 (128x higher than what
typically used by SAC), a reset occurs once every 20000 steps of interaction with the environment.
Results In Figure 2, we compare a version of SR-SAC that uses a replay ratio of 128 to standard deep
RL baselines. This also includes the recently proposed REDQ (Chen et al., 2021), which obtained
state-of-the-art sample efficiency by using a replay ratio of 20. At any budget of interactions with the
environment, SR-SAC compares favorably with REDQ, despite being a simpler algorithm. SR-SAC
establishes a new state-of-the-art result for model-free continuous control. Following (Agarwal
et al., 2021) we focus on interquartile mean (IQM) performance, defined as the 25% trimmed mean
performance over all runs on all considered tasks, and report 95% bootstrap confidence intervals.
4.2 A TARI 100 K
Reset Strategy We follow Nikishin et al. (2022) in performing one reset every 40,000 updates; at
replay ratio 16, the highest considered, this corresponds to a reset every 2,500 environment steps,
or roughly once every three minutes of interaction. However, Nikishin et al. (2022) only reset a
subset of the agent’s parameters when training on the ALE, leaving the agent’s convolutional encoder
untouched by resets. While this leaves the encoder vulnerable to plasticity loss, fully resetting the
encoder is impractical, as Nikishin et al. (2022) observe. As an intermediate solution, we apply soft
resets, using a variant of Shrink and Perturb (Ash & Adams, 2020) in which encoder parameters are
interpolated between their previous value and a random re-initialized parameter vector on each reset:
θt =αθt−1 + (1 −α)ϕ,ϕ ∼ initializer. This formulation is different from that used by (Ash
& Adams, 2020) but allows easy interpolation between completely resetting a layer and leaving it
unchanged; we useα = 0.8 by default. We examine the impact of this decision in Section 5.2.
Target Networks By default, SPR does not employ a separate target network, unlike traditional
DQNs (Mnih et al., 2015a). However, we find that this leads replay ratio scaling to stop improving
performance at relatively low replay ratios, which we hypothesize is due to fundamental variance in
optimization limiting the accuracy to which the value function may be estimated. To alleviate it, we
directly adopt the target strategy employed by SR-SAC, with an exponential moving average (EMA)
target network with coefficientτ = 0.005, which we find allows beneficial replay ratio scaling out to
at least replay ratio 16. Moreover, following (Ghavamzadeh et al., 2011), SR-SPR also uses its target
network for action selection. We elaborate on this design decision in Section 5.2.
Results Figure 3 shows performance profiles of SR-SPR at various replay ratios, demonstrating
that replay ratio scaling consistently improves performance up to at least replay ratio 16. We also
compare a version of SR-SPR that uses replay ratio 16 to standard baselines (DrQ, DER, Kostrikov
0.0 0.2 0.4 0.6 0.8 1.0
Human Normalized Score ( )
0.00
0.25
0.50
0.75
1.00Fraction of runs with score >
SR-SPR: 16
SR-SPR: 8
SR-SPR: 4
SR-SPR: 2
SR-SPR: 1
SPR
Atari 100k
Method IQM Median Mean
SR-SPR 0.632 (0.60, 0.66) 0.685 (0.60, 0.77) 1.272 (1.18, 1.37)
IRIS 0.501 (0.44, 0.56) 0.289 (0.25, 0.41) 1.046 (0.96, 1.13)
SPR 0.380 (0.36, 0.39) 0.433 (0.38, 0.48) 0.578 (0.56, 0.60)
DrQ(ϵ) 0.280 (0.27, 0.29) 0.304 (0.28, 0.33) 0.465 (0.46, 0.48)
DER 0.183 (0.18, 0.19) 0.191 (0.18, 0.21) 0.351 (0.34, 0.36)
0.40 0.48 0.56 0.64
IQM
SR-SPR
IRIS
SPR
0.30 0.45 0.60 0.75
Median
0.75 1.00 1.25
Mean
Figure 3 & Table 2: Performance profiles (left, higher is better) of SR-SPR at various replay ratios,
and 95% C.I.s of SR-SPR: 16 and of standard baselines on Atari 100k (right, 20 seeds for SR-SPR
and SPR, 5 seeds for IRIS, 100 seeds for all other algorithms as taken from Agarwal et al. (2021))
5

Published as a conference paper at ICLR 2023
105 106 107 1080
200
400
600
800IQM
SAC
105 106 107 108
SR-SAC
105 106 107 108
Tandem SR-SAC
105 106 107 108
Iterated Offline SR-SAC
8.0 × 105
6.1 × 105
4.2 × 105
2.2 × 105
2.5 × 104
Environment steps
Number of agent updates
Figure 4: Scaling behavior of SAC, SR-SAC and its tandem and iterated offline variations in the
DMC15 benchmark. Each individual line shows performance at a given number of environment
steps, denoted by color, across different numbers of agent updates. Each point in a line is obtained by
measuring performance with a different replay ratio for that number of environment steps. Each line
is computed over 5 seeds.
et al., 2022; Van Hasselt et al., 2019) and recent work (IRIS, Micheli et al., 2022) in table 2. SR-SPR
establishes a new state-of-the-art for model-free control on Atari 100k, and rivals prior work that has
aggressively pretrained on additional data (Liu & Abbeel, 2021; Schwarzer et al., 2021b). We present
full results and per-game scores for SR-SPR in table 4, and show training curves in fig. 15. We report
IQM performance, as well as plotting a performance profile (Agarwal et al., 2021), which visualizes
the full distribution of performance across all runs2 and demonstrates that increasing SR-SPR’s replay
ratio comprehensively improves performance.
5 A LGORITHM DESIGN IN LIGHT OF REPLAY RATIO SCALING
5.1 A NALYZING THE IMPORTANCE OF ONLINE INTERACTION
When training with high replay ratios and short reset intervals, the training regime an agent is subjected
to begins to resemble offline RL; the agent is primarily learning from data collected by policies
unrelated to its own, with only a small amount of online data available to correct its policy. Given
many classical analyses from offline RL (Levine et al., 2020), it is perhaps surprising that an agent
trained in a pseudo-offline setting with no explicit regularization towards conservatism (e.g., Kumar
et al., 2020) can learn successfully. What is then the role of the incoming stream of interactions?
To gain some understanding, in this section we attack the problem from different angles and study
the scaling behavior of variants of SR-SAC. We consider different data collection patterns and how
interleaving them with agent optimization changes the training dynamics. The appendix also presents
a comparisons with NFQI (Riedmiller, 2005) and with the online use of an offline RL algorithm.
5.1.1 I TERATED OFFLINE SETTING
Changing the replay ratio in a deep RL algorithm can be seen as a specific way of increasing the
proportion of offline training an agent is subject to. Specifically, the agent’s parameters are updated a
number of times exactly equal to the replay ratio before a new sample is collected. This implies a
uniform distribution of the number of offline updates across time steps. Is this an important variable
for determining the replay ratio scaling behavior of an algorithm?
To answer this question, we resort to what we call iterated offline RL (Matsushima et al., 2021;
Riedmiller et al., 2021), which alternates between purely offline updates and data collection. In this
setting, a certain value of replay ratio is not distributed uniformly during the course of the interactions
with the environment. Instead, the agent is not updated during data collection, and an amount of
updates equal to the one that would be due in that data collection time frame in virtue of the replay
ratio is applied completely offline, right after each reset.
As visible in Figure 4, the iterated offline paradigm has a different replay ratio scaling behavior.
Applying a very large number of updates with a fixed dataset, with an algorithm such as SAC, incurs
serious risk of generating a degenerate policy, not able to outperform the previous one. As exemplified
in Figure 6, in the absence of any mechanism for stopping this natural degeneration to happen, this
circle is broken only when enough data is collected. Collecting enough data in this sense is possible
2Broadly speaking, a transposed and clipped plot of the cumulative distribution function of performance.
6