Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 5/6

Cleaned-text code points [52107, 65916). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
Expression Definition Used In
Damage from
Warm-Starting
[Phenomenon for which] “a warm-started network performs
worse on test samples than a network trained on the same data
but with a new random initialization”
Ash & Adams
(2020)
Damage from
Non-Stationarity
“A memory effect where these transient non-stationarities can
permanently impact the latent representation and adversely affect
generalisation performance”
Igl et al. (2021)
Capacity Loss “Reduced ability to fit new targets in deep neural networks” Lyle et al. (2022b),
Lyle et al. (2022a)
Loss of Plasticity “Loss of the ability of the model to keep learning”
Berariu et al.
(2021), Dohare
et al. (2022)
Primacy Bias “A tendency to overfit initial experiences that damages the rest
of the learning process”
Nikishin et al.
(2022)
Table 3: Definitions of coinciding and related phenomena from previous work justifying the effective-
ness of our strategy for replay ratio scaling.
1 2 3 4 5 6 7 8
Reset Interval 1e6
400
500
600
700IQM
RR=16
RR=32
(a) DeepMind Control Suite (DMC15-500k)
20000 40000 80000 160000
Reset Interval
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65IQM
RR=16
RR=8
RR=4
RR=2
RR=1 (b) Atari 100k
Figure 9: Sensitivity of the IQM to varying reset intervals (in terms of gradient updates) of SR-SAC
on the DeepMind Control Suite (DMC15-500k) benchmark, and of SR-SPR on the Atari 100k
benchmark. (10 seeds, 95% bootstrapped C.I.).
A D EFINITIONS FROM RELATED WORKS
To further clarify our description of previous work from the related work section, we report in Table 3
definitions of the different terms used to refer to the loss of the ability to learn and generalize in neural
networks. Each definition is directly taken from one of the papers corresponding to it. Note that,
despite their overlap, they reflect slightly different perspectives on the nature of this phenomenon,
and it can be worth for future investigations to pin down which one of these is more relevant for
replay ratio scaling or reinforcement learning as a whole.
B A DDITIONAL EXPERIMENTAL RESULTS
B.1 A DDITIONAL STUDIES
Reset Interval An important hyperparameter for both SR-SAC and SR-SPR is the interval at which
resets are performed, as denominated in terms of number of agent updates. In Figure 9, we study how
performance is impacted by this choice, at different replay ratios. Overall, both SR-SAC and SR-SPR
perform well for a vast range of reset intervals, with favorable replay ratio scaling and generally
smooth performance degradation. Note that, for large intervals (e.g., the last point on the right for
SR-SAC withRR = 16, and last two points in the bottom right for SR-SPR), this is equivalent to
actually performing no resets, just running the unmodified baseline algorithms. Thus, performance
experiences non-smooth drops only in these easily avoidable cases.
14

Published as a conference paper at ICLR 2023
0 20000 40000 60000 80000 100000
Step
0.00
0.01
0.02
0.03
0.04
0.05
0.06Target Churn
0 20000 40000 60000 80000 100000
Step
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35Online Churn
0 20000 40000 60000 80000 100000
Step
0.3
0.4
0.5
0.6
0.7
0.8Online-Target Agreement
0 20000 40000 60000 80000 100000
Step
0.65
0.70
0.75
0.80
0.85
0.90Off-Policy Rate
RR 16.0 RR 8.0 RR 4.0 RR 2.0 RR 1.0 
Figure 10: Churn-related diagnostics (based on the policy churn definition from Schaul et al. (2022))
for the online and target networks. Different colors and RR denote different values of replay ratio.
Counteracting Policy Churn by Acting with the Target Network Schaul et al. (2022) defined
the policy churn as the change in the agent’s policy due to optimization. It was shown that a certain
amount of policy churn can be beneficial for exploration in the absence of external noise (e.g., coming
fromϵ-greedy exploration); however, it is intuitive that excessive churn can actually hurt performance,
for instance by breaking the the consistency of trajectories. We hypothesize that mitigating excessive
policy churn is a major reason why SR-SPR performs better when actions are selected with the
target network rather than the online network. Figure 10 shows that, if we measure churn before
each interaction with the environment, increasing the replay ratio will naturally increase it, while
acting with the target network will decrease it. When the replay ratio is too high, it is likely that the
benefits coming from additional offline computations might be nullified by the inconsistency in the
exploration data; acting with the target network reduces these inconsistencies without giving up on
the more efficient optimization, at the cost of introducing minimal delays in the improvement of the
data-collecting policy.
Using an offline RL algorithm online In Section 5.1, we conducted a set of experiments with the
goal of highlighting the importance of online interactions, concluding that a consistent stream of
online interaction data and neural networks able to learn and generalize from a dataset of experiences
are key factors behind effective replay ratio scaling. Online RL algorithms such as SAC are naturally
reliant on the online stream of interactions; but it is natural to ask whether algorithms created for
offline RL setting, where no interaction with the environment is assumed to be possible during
training, can make the most out of the computational budget granted through high replay ratios in
the online setting. To test this hypothesis, we run the Implicit Q-learning (IQL) (Kostrikov et al.,
2022) algorithm as an online RL algorithm on DMC15-500k with a replay ratio of 32. Results in
Figure 11 show that, despite being by design more robust to more aggressive training, the conservative
15

Published as a conference paper at ICLR 2023
250 500 750
IQM
SAC
REDQ
SR-SAC
DDPG
IQL + resets
IQL
200 400 600
Median
200 400 600
Mean
Figure 11: Performance on DMC15-500k of running IQL (RR=32) online, with and without resets.
1 2 4 8
Replay Ratio
0.2
0.3
0.4
0.5
0.6
0.7IQM
Pretrained SR-SPR
SR-SPR
Pretrained SPR
SPR
Figure 12: The performance of SR-SPR and SPR from scratch and when fine-tuning a pre-trained
encoder on Atari 100k (10 seeds. 95% bootstrapped C.I.
nature of offline RL algorithms makes them not amenable to effective online learning, regardless of
the presence of resets. This shows the effectiveness of online interactions as a strong supervision
mechanism, able, when supported by resets, to make online RL algorithms robust to high replay
ratios without the need of overly conservative behaviors.
B.2 F INETUNING PRETRAINED REPRESENTATIONS
Finetuning pretrained representations has become increasing common in reinforcement learning and
elsewhere (Schwarzer et al., 2021b; Liu & Abbeel, 2021; Bommasani et al., 2021). One obvious
question to ask is whether or not replay scaling as demonstrated in the tabula rasa setting here can
also be used to make this finetuning more sample efficient. In Figure 12 we answer this question
in the affirmative. We initialize SPR and SR-SPR with pretrained encoders (taken for experimental
convenience from SPR agents trained at replay ratio 1 for one million steps), and initialize all other
parameters randomly. We then train at a range of replay ratios for 100k steps. For SR-SPR, we
apply shrink and perturb towards the pretrained encoder weights rather than random parameters, but
otherwise train as normal.
We find that while both SPR and SR-SPR benefit from the pretrained representations at low replay
ratios, only SR-SPR is able to improve fine-tuning performance by replay scaling. Standard SPR
with pretrained representations rapidly degrades in performance as the replay ratio is increased, while
the performance of SR-SPR steadily increases at higher replay ratios. Although the gap between
SR-SPR with and without pretraining closes somewhat at higher replay ratios, this is to be expected,
as higher replay-ratio agents have more opportunities to improve their own representations even
without pretraining.
B.3 F INETUNING AFTER OFFLINE TRAINING
The efficiency of SR-SAC and SR-SPR makes the general approach behind their design potentially
appealing for the setting of offline RL with an additional fine tuning phase. In Figure 13, we provide
16

Published as a conference paper at ICLR 2023
0.5
 0.0 0.5 1.0
Step 1e6
40
50
60
70
80
90
100Normalized Return
antmaze-umaze-v0
0.5
 0.0 0.5 1.0
Step 1e6
0
20
40
60
80
100Normalized Return
antmaze-umaze-diverse-v0
IQL SR-IQL
Figure 13: Performance of SR-IQL and IQL in two tasks from D4RL. Negative steps denotes the
pretraining phase (10 seeds, ± std).
4 6 8
Data budget 
 (Environment steps)
1e5
0
1
2
3
Computational budget  
 (Number of agent updates)
1e7
 IQM > 600
SR-SAC
Tandem SR-SAC
SAC
Figure 14: Pareto fronts for SR-SAC and Tandem SR-SAC on DMC15 (5 seeds).
preliminary evidence that the paradigm we advocated for in this paper may indeed be particularly
beneficial in this setting. We test IQL with the same pretraining scheme presented in Kostrikov
et al. (2022), consisting in a million offline training steps followed by a million interactions with
the environment for fine tuning. We implement SR-IQL by using a replay ratio of 10 and resetting,
every two million updates, all of the parameters of its neural networks during the fine tuning phase.
We compare SR-IQL to IQL on two tasks from the D4RL benchmark (Fu et al., 2020). As shown
in Figure 13, SR-IQL is roughly on par with IQL in the antmaze-umaze-v0 task, but reaches
superior performance during fine tuning in antmaze-umaze-diverse-v0. We believe this sets
the stage to experimenting with our replay ratio scaling paradigm in this setting as a promising
research direction for future work.
B.4 P ARETO FRONTS COMPARISON
With the same approach used for studying the data/computation tradeoffs of SR-SAC, it also becomes
possible to directly compare the performance of different replay ratio-scalable algorithms. As a
simple example, we compare SR-SAC, its tandem version and SAC in Figure 14. The different lines
are Pareto curves, obtained by retaining the points that are dominating the other ones in terms of either
data or computational budget, to reach an IQM of at least 600. On this plot, SAC simply appears
as a point because, not allowing for effective replay ratio scaling, it can only reach the prescribed
performance by using more data and a relatively small amount of computational resources.
17

Published as a conference paper at ICLR 2023
B.5 C OMPARISON WITH NEURAL FITTED Q-I TERATION
The approach we demonstrated for replay ratio scaling, for its relationship with offline RL and its use
of resets, could resemble the classic NFQI algorithm (Riedmiller, 2005), which train from scratch,
after each large batch of transitions, a Q-function. Our approach propagates information across
resets mainly through the use of the replay buffer, having a fast target network updated alongside
the regular agent training; NFQI instead propagates information primarily through a target network,
which is updated once per reset. We implement an-friendly variant of NFQI on SR-SPR at replay
ratio 16, performing one target network update upon each reset. However, we find that this leads
to very poor performance (IQM 0.350), achieving barely half that of standard SR-SPR. Although
we hypothesized that 2,500 environment steps (the standard reset interval for SR-SPR at replay
ratio 16) might be too infrequent for target network updates, making this interval shorter did not
improve performance. Although we cannot rule out the possibility that NFQI might be competitive at
dramatically higher replay ratios, its inherent slowness in propagating information is likely to lead
it to lag in data-efficient settings; any reset interval that is sufficiently long to allow for accurate
estimation of the value function may lead to insufficiently rapid value propagation via target updates,
and vice versa.
C C OMPUTATIONAL CONSIDERATIONS
For DMC, the running time depends on the individual environment, due to differences in dimen-
sionality of the observation as well as physics simulation time. On an NVIDIA V100 GPU, at this
highest replay ratio of RR=128, our code takes about 10.5 hours on acrobot-swingup and about
15 hours on humanoid-run to complete 500k environment steps. For a replay ratio of RR=32,
which yields remarkable, even if not best, performance, the time goes down to just about 3 hours
and about 4 hours respectively. which is well-below the typical demands of modern model-based RL
methods. With careful seed parallelization, running SR-SAC with RR=32 for 5 seeds for all tasks
in the DMC15-500k benchmark takes less than 4 GPU/days on an NVIDIA V100. For Atari 100k,
running time depends primarily on the replay ratio chosen. At the highest replay ratio used (16) and
with five seeds running in parallel, our code takes roughly 25 hours to complete 100k steps on an
NVIDIA A100, yielding a cost of roughly 5 GPU/hours per training run.
D E XPERIMENTAL DETAILS
We report in Table 6 the full list of tasks for the DMC15 benchmark. Our implementation of
continuous control algorithms is based on the jaxrl codebase (Kostrikov, 2021). For REDQ, we
use the best hyperparameters, as recommended by Chen et al. (2021), as well as a replay ratio of
20. For discrete control, we use a version of SPR implemented in Jax (Bradbury et al., 2018) in
Dopamine (Castro et al., 2018). See Table 5 for a full list of the employed hyperparameters.
D.1 F ULL EXPERIMENTAL RESULTS
In Figure 16, we show the scaling curve for DMC15-1M.
We report in Table 4 the full per-game results for SR-SPR and in Figure 17,18,19,20,21 full experi-
mental results for SR-SAC. For completeness, we also report the performance of the modified settings
and of REDQ at the same replay ratios.
18