Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 3/6

Cleaned-text code points [24711, 37726). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
0.00 0.05 0.10 0.15 0.20
Environment Steps (×106)
0
250
500
750
1000Episode Return
hopper-stand
0.00 0.05 0.10 0.15 0.20
Environment Steps (×106)
0
100
200Episode Return
humanoid-stand
0.00 0.05 0.10 0.15 0.20
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-walk
0.00 0.05 0.10 0.15 0.20
Environment Steps (×106)
0
200
400
600
800Episode Return
walker-run
SR-SAC, RR=128SR-SAC, RR=128 SAC, RR=1SR-SAC, RR=128 SAC, RR=1 Tandem SR-SAC, RR=128SR-SAC, RR=128 SAC, RR=1 Tandem SR-SAC, RR=128 Iterated Offline SR-SAC, RR=128
Figure 6: Examples of behaviors of SR-SAC and its tandem and iterated offline variations on four
environments from DMC15. (5 runs, ± std).
for easy tasks such as hopper-stand and walker-run, with a cost in sample efficiency, or
impossible on hard tasks such as humanoid-stand and quadruped-walk. This is unfortunate:
the iterated offline RL paradigm can be quite useful in practical settings, in which the agent is allowed
to only collected batches of data without any update (perhaps for safety reasons); however, current
backbone algorithms (such as SAC) are not currently compatible with such a setting, that thus leads
to favorable replay scaling only when closer to the online setting. This explains the sudden increase
in the curve of Figure 4, when the number of agent updates, and consequently the reset frequency,
becomes large enough. An interesting question, left for future work, is whether this behavior could
change if combined with conservative algorithms created for the offline RL setting.
5.1.2 T ANDEM SETTING
With high replay ratios, an agent’s training begins to resemble offline RL: although the agent still
has the possibility to interact with the environment, it is very infrequent relative to the amount of
training. Thus, an agent after a reset has a small stream of interactions collected by the agent itself,
while the vast majority of its data was collected by potentially unrelated agents. How important,
then, is this small stream of online interaction? To answer this question, we leverage the tandem
setting, as presented in Ostrovski et al. (2021). Two copies of the same agent, identical apart from
the initialization, are created. With the same algorithm (SR-SAC in this case), they are trained on
the replay buffer collected by the active agent. The passive agent thus never directly interacts with
environment, and cannot collect data to correct its own misconceptions about the environment.
0 40000 80000
Step
0.0
0.2
0.4
0.6
0.8Training IQM
SR-SPR: 16 + Offline
SR-SPR: 16
SPR: 16
0.30 0.45 0.60
IQM
SR-SPR: 16
SR-SPR: 16 + Offline
SPR: 16
Figure 5: Learning curves
(top) and evaluation perfor-
mance (bottom) at replay ratio
16 for SPR and SR-SPR with
and without offline updates af-
ter each reset.
As shown in Figure 4, the behavior of Tandem SR-SAC offers an
alternative perspective on the importance of online interactions: de-
spite the performance of the algorithm being hurt, the overall replay
ratio scaling capabilities remain similar. We can look at the perfor-
mance of the passive agent to understand what the exact effect of on-
line interactions is on training. As evident in the environments from
Figure 6, especially in hopper-stand and quadruped-walk,
there is a qualitative difference between the behavior of an active
agent (blue curve) and a passive agent (green curve): right after a
reset, with the initial high replay ratio training, the performance of
both agents is greatly improved; after a few thousands steps, training
remains stable for the active agent but causes performance collapse
in the passive agent. This experiment thus demonstrates the power of
having online interactions as an implicit regularization mechanism.
For the design of future replay ratio-scalable algorithms, one should
keep in mind that it is indeed possible to scale an algorithm poten-
tially affected by extreme off-policyness; however, online data col-
lection slows down performance collapse when training aggressively,
as shown in both the iterated offline and the tandem experiments.
5.1.3 A LTERNATIVE COMBINATIONS OF OFFLINE AND ONLINE UPDATES
The iterated offline setting can be seen as the extreme in which all of the updates are done offline,
compared to the even distribution used in the online setting. What if we use an intermediate strategy?
7

Published as a conference paper at ICLR 2023
For SR-SPR, we find that directly mixing offline and online RL by performing half the updates
allotted to each interval immediately after each reset can actually improve performance by some
metrics, such as training return (see Figure 5 upper), by mitigating the performance drop otherwise
experienced after each reset. Although we find that this has essentially no impact on final evaluation
performance (Figure 5 lower), it may allow SR-SPR to be used when cumulative regret is important.
5.2 W HAT IS REQUIRED FOR REPLAY RATIO SCALING IN DISCRETE CONTROL ?
1 2 4 8 16
Replay Ratio
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65IQM
SR-SPR
No SP
No Target Exploration
Online Targets
No SP, Online targets
Constant interval
SP Only
Figure 7: The replay ratio scaling behavior of SR-
SPR with various components ablated.
Although replay ratio scaling is relatively
straightforward for SR-SAC, achieving robust
replay ratio scaling for SR-SPR requires more
complex design decisions due to its shorter train-
ing period and more complex function approx-
imation. As a result, unlike SR-SAC, SR-SPR
contains additional modifications from the vari-
ant of SPR used by Nikishin et al. (2022). We
study the impact of these design decisions on
scaling behavior and report results in Figure 7.
Inspired by the findings of Berariu et al. (2021)
that plasticity loss is concentrated in the final lay-
ers of the network but affects all layers, we apply
Shrink and Perturb (SP) to the encoder; this is
responsible for roughly a constant increase of
IQM 0.04 past replay ratio 4. We note however
that applying Shrink and Perturb alone to all the
parameters of the network is not sufficient to
enable beneficial scaling; it is important that at
least the network’s final layers be completely
reset. We explain this using the observations
from (Berariu et al., 2021) that the last layers
are more responsible for the loss of plasticity.
That said, the most important factor in allowing
SR-SPR to continue scaling well is its use of
a target network. This effect is primarily due
to better action selection through the target net-
work; we found that the stabilizing effect on optimization was a less important factor. This is
reminiscent of speedy Q-learning (Ghavamzadeh et al., 2011), where the use of an exponential
moving average policy was shown to improve convergence speed, and can also be understood in
relationship to the policy churn phenomenon (Schaul et al., 2022) (see Figure 10 in the appendix).
Meanwhile, removing both Shrink and Perturb and the target network is roughly equivalent to taking
the method of Nikishin et al. (2022) but setting reset intervals as in SR-SPR. As Figure 7 suggests, this
alone suffices to yield some replay ratio scaling but not as efficient compared to SR-SPR. However,
maintaining a fixed reset interval (in terms of environment steps) when varying replay ratio, as done
by Nikishin et al. (2022), leads to poor performance at replay ratios above 4.
Intriguingly, we note that these modifications are beneficial specifically for replay ratio scaling;
at replay ratios 1 or 2 they do not improve performance (although for the most part they do not
significantly harm performance either). We thus hypothesize that there may be other modifications to
complex algorithms such as SPR that could be made to further improve replay ratio scaling properties,
but that are today not in widespread use because they do not improve performance in standard low
replay ratio settings.
5.3 V ISUALIZING THE DATA/COMPUTE TRADEOFF
If an order of magnitude more of updates can be used for improving the performance of an algorithm,
additional tradeoffs start to emerge. The type of computations that replay ratio scaling implies are
fundamentally different than other concepts of scaling, (e.g., about larger models): scaling here is
inherently sequential. Thus, obtaining more hardware does not help faster execution of the algorithm.
8

Published as a conference paper at ICLR 2023
When collecting new transitions is not very expensive, the choice between collecting new samples in
the environment and spending more time updating an agent could become nontrivial.
0.2 0.4 0.6 0.8 1.0
Data budget 
 (Environment steps)
1e6
106
107
Computational budget  
 (Number of agent updates)
300
400
500
600
700
IQM
Figure 8: Performance of SR-SAC in
DMC15 as a function of the number of
interactions and of the number of agent
updates, determined by the replay ratio.
We visualize this tradeoff in Figure 8. The plot is obtained
by combining runs of SR-SAC with doubling replay ratio
from 0.25 to 128, and considering, for a fixed data budget
(in terms of environment steps), the total computational
budget (in terms of total number of agent updates at that
point), as well as the achieved performance. There exists
multiple ways to achieve the same level of performance, as
denoted by the color. This plot shows that resets provide a
knob on replay ratio scaling and allows to tradeoff data for
computation. If, for a given problem, sample efficiency
is more important than computational considerations, one
can spend about two orders of magnitude of additional
agent updates to obtain the same performance that can
be obtained by waiting for 800000 additional samples to
be collected from the environment. The peculiar feature
of the approach we advocate for in this paper is that it
allows to act on this tradeoff with an algorithm basically
as simple as the employed backbone.
6 T HE LIMITS OF REPLAY RATIO SCALING
We have seen what becomes possible when higher level of replay ratio scaling are unlocked by resets.
What are the limits of this paradigm? First of all, replay ratio scaling is always possible up to a
finite value, at which there is simply not enough information left to be extracted from the existing
dataset of experience. Current methods, including the one proposed in this paper, are not able to
automatically identify when this limit is reached, and they are therefore still subject to performance
collapse when increasing the replay ratio too much. Second, replay ratio scaling cannot go beyond
the intrinsic limitations of the given deep RL algorithm: for example, if the task is simply impossible
to solve because of hard credit assignment or exploration, then replay ratio scaling is only of limited
help. Third, the strategy we proposed for replay ratio scaling is based on keeping the entire history of
interactions with the environment in the replay buffer. While this is feasible for the kind of sample-
efficiency benchmarks that we have used in this paper, it might also require special consideration to
be applied to larger problems; for instance, it is possible to keep a large replay buffer on permanent
storage, albeit at the cost of slower batch retrieval. Lastly, replay ratio scaling can inherently become
time-consuming for a training agent, which can limit the applicability of methodologies like ours to
settings requiring high-frequency interactions with an environment.
7 C ONCLUSIONS
In this paper, we have shown that, by leveraging partial or full resets of an agent’s parameters,
it is possible to unlock new levels of favorable replay ratio scaling and, consequently, of sample-
efficiency for model-free deep RL algorithms. We demonstrated this by a careful evaluation on
the DeepMind Control Suite and Atari 100k benchmarks, where our approach (SR-SAC and SR-
SPR) demonstrated far superior performance compared to strong baselines, with minimal amounts
of additional algorithmic complexity. Then, we discussed which algorithmic design choices are
important for achieving such levels of replay ratio scaling with a deep RL algorithm, as well as the
tradeoffs implied by this paradigm. Through our empirical analysis, we showed the value of online
data collection, offering a perspective on its relationship with offline RL (Levine et al., 2020).
More generally, this paper is about how to leverage a discovery for the design of future deep RL
algorithms. We believe this work to be an example of how the development of effective deep RL
methods should be achieved not only through extending existing algorithms or creating new ones,
but also through the discovery of new phenomena related to deep RL systems, and of techniques for
exploiting them to increase performance. It is natural to wonder whether deeper understanding or
exploitation of surprising empirical properties (Ostrovski et al., 2021; Schaul et al., 2022) beyond the
one behind this work could lead to the emergence of new capabilities in deep RL algorithms.
9