Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 6/6

Cleaned-text code points [65916, 79593). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
Game Random Human IRIS SR-SPR:2 SR-SPR:4 SR-SPR:8 SR-SPR:16
Alien 227.8 7127.7 420.0 877.9 964.4 1015.5 1107.8
Amidar 5.8 1719.5 143.0 189.2 211.8 203.1 203.4
Assault 222.4 742.0 1524.4 891.9 987.3 1069.5 1088.9
Asterix 210.0 8503.3 853.6 836.7 894.2 916.5 903.1
Bank Heist 14.2 753.1 53.1 253.6 460.0 472.3 531.7
Battle Zone 2360.0 37187.5 13074.0 14493.5 17800.6 19398.4 17671.0
Boxing 0.1 12.1 70.1 36.1 42.0 46.7 45.8
Breakout 1.7 30.5 83.7 24.5 26.1 28.8 25.5
Chopper Command 811.0 7387.8 1565.0 1609.4 1933.7 2201.0 2362.1
Crazy Climber 10780.5 35829.4 59324.2 28004.7 38341.7 43122.3 45544.1
Demon Attack 152.1 1971.0 2034.4 2969.0 3016.2 2898.1 2814.4
Freeway 0.0 29.6 31.1 24.1 24.5 24.9 25.4
Frostbite 65.2 4334.7 259.1 1450.4 1809.9 1752.8 2584.8
Gopher 257.6 2412.5 2236.1 735.3 717.5 711.2 712.4
Hero 1027.0 30826.4 7037.4 6832.1 7195.7 7679.6 8524.0
Jamesbond 29.0 302.8 462.7 412.9 408.8 392.8 389.1
Kangaroo 52.0 3035.0 838.2 1651.2 2024.1 3254.9 3631.7
Krull 1598.0 2665.5 6616.4 5206.4 5364.3 5824.8 5914.4
Kung Fu Master 258.5 22736.3 21759.8 14165.6 17656.5 17095.6 18649.4
Ms Pacman 307.3 6951.6 999.1 1472.6 1544.7 1522.6 1574.1
Pong -20.7 14.6 14.6 -10.5 -5.5 -3.0 2.9
Private Eye 24.9 69571.3 100.0 98.8 95.8 95.8 97.9
Qbert 163.9 13455.0 745.7 3431.7 3699.8 3850.6 4044.1
Road Runner 11.5 7845.0 9614.6 12199.0 14287.3 13623.5 13463.4
Seaquest 68.4 42054.7 661.3 714.7 766.6 800.5 819.0
Up N Down 533.4 11693.2 3546.2 61851.2 91435.2 95501.1 112450.3
Games> Human 0 0 9 7 8 9 9
IQM (↑) 0.000 1.000 0.501 0.444 0.544 0.589 0.632
Optimality Gap (↓) 1.000 0.000 0.512 0.516 0.470 0.452 0.433
Median (↑) 0.000 1.000 0.289 0.336 0.523 0.560 0.685
Mean (↑) 0.000 1.000 1.046 0.910 1.111 1.188 1.272
Table 4: Full results for individual games in Atari 100k for SR-SPR at various replay ratios.
0 20000 40000 60000 80000 100000
Step
40
60
80
100
120Parameter Norm
0 20000 40000 60000 80000 100000
Step
0.50
0.75
1.00
1.25
1.50
1.75
2.00
2.25Gradient Norm
0 20000 40000 60000 80000 100000
Step
0.06
0.08
0.10
0.12
0.14
0.16
0.18TD Error
0 20000 40000 60000 80000 100000
Step
0.0
0.2
0.4
0.6Training IQM
SR-SPR:2
SPR:2
SR-SPR:4
SPR:4
SR-SPR:8
SPR:8
SR-SPR:16
SPR:16
Figure 15: Learning curves for SR-SPR (solid) and SPR (dashed) at various replay ratios. Note that
all SR-SPR runs converge to similar TD errors, gradient norms and parameter norms, while these
metrics greatly differ for SPR at different replay ratios. IQM training performance does not match
evaluation performance, as ongoing training episodes are often disrupted by the reset procedure.
19

Published as a conference paper at ICLR 2023
Parameter Setting
Gray-scaling True
Observation down-sampling 84x84
Frames stacked 4
Action repetitions 4
Reward clipping [-1, 1]
Terminal on loss of life True
Max frames per episode 108K
Update Distributional Q
Dueling True
Support of Q-distribution 51
Discount factor 0.99
Minibatch size 32
Optimizer Adam
Optimizer: learning rate 0.0001
Optimizer:β1 0.9
Optimizer:β2 0.999
Optimizer:ϵ 0.00015
Max gradient norm 10
Priority exponent 0.5
Priority correction 0.4 → 1
Exploration Noisy nets
Noisy nets parameter 0.5
Training steps 100K
Evaluation trajectories 100
Min replay size for sampling 2000
Replay period every 1 step
Updates per step Variable (1, 2, 4, 8, 16)
Multi-step return length 10
Q network: channels 32, 64, 64
Q network: filter size 8× 8, 4× 4, 3× 3
Q network: stride 4, 2, 1
Q network: hidden units 512
Non-linearity ReLU
Target network update period 1
λ (SPR loss coefficient) 2
K (SPR prediction depth) 5
Data Augmentation Shifts ( ±4 pixels)
Intensity(scale=0.05)
τ (EMA coefficient) 0.995
Reset Interval (gradient steps) 40,000
Layers getting hard reset Final 2
Shrink and Perturbα 0.8
Action selection Target network
Parameter Setting
Discount factor 0.99
Minibatch size 256
Optimizer (all) Adam
Optimizer (all): learning rate 0.0003
Optimizer (all):β1 0.9
Optimizer (all):β2 0.999
Optimizer (all):ϵ 0.00015
Networks (all): activation ReLU
Networks (all): n. hidden layers 2
Networks (all): hidden units 256
Initial Temperature 1
Replay Buffer Size 106
Updates per step Variable (1 to 128)
Target network update period 1
τ (EMA coefficient) 0.995
Reset Interval (gradient steps) 2560000
Layers getting hard reset All
Table 5: Hyperparameters for SR-SPR and SR-SAC. The ones introduced by this work are at the
bottom of the respective tables.
20

Published as a conference paper at ICLR 2023
Environment Tasks
walker run
quadruped run , walk
reacher hard
humanoid run , walk, stand
swimmer swimmer6
cheetah run
hopper hop , stand
acrobot swingup
pendulum swingup
finger turn hard
fish swim
Table 6: The tasks from the DMC15 benchmark. We chose commonly-employed DMC tasks for
which the optimal policy is not immediately found by SAC according to https://github.com/
denisyarats/pytorch_sac#results.
1 2 4 8 16 32 64 128
Replay Ratio
200
400
600
800IQM
SAC
SR-SAC
Figure 16: Scaling curve for SR-SAC and SAC on DMC15-1M.
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200Episode Return
acrobot-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
cheetah-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
finger-turn_hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
fish-swim
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
hopper-hop
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
hopper-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
50
100
150
200Episode Return
humanoid-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
humanoid-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
humanoid-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
pendulum-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
reacher-hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
swimmer-swimmer6
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
walker-run
SR-SAC, RR=8SR-SAC, RR=8 SAC, RR=8SR-SAC, RR=8 SAC, RR=8 Tandem SR-SAC, RR=8SR-SAC, RR=8 SAC, RR=8 Tandem SR-SAC, RR=8 Iterated Offline SR-SAC, RR=8SR-SAC, RR=8 SAC, RR=8 Tandem SR-SAC, RR=8 Iterated Offline SR-SAC, RR=8 REDQ, RR=8
Figure 17: Evaluation Returns on individual DMC15 environments for replay ratio 8.
21

Published as a conference paper at ICLR 2023
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
acrobot-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
cheetah-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
finger-turn_hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
fish-swim
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
hopper-hop
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
hopper-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200Episode Return
humanoid-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
humanoid-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
humanoid-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
pendulum-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
quadruped-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
reacher-hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
swimmer-swimmer6
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750Episode Return
walker-run
SR-SAC, RR=16SR-SAC, RR=16 SAC, RR=16SR-SAC, RR=16 SAC, RR=16 Tandem SR-SAC, RR=16SR-SAC, RR=16 SAC, RR=16 Tandem SR-SAC, RR=16 Iterated Offline SR-SAC, RR=16SR-SAC, RR=16 SAC, RR=16 Tandem SR-SAC, RR=16 Iterated Offline SR-SAC, RR=16 REDQ, RR=16
Figure 18: Evaluation Returns on individual DMC15 environments for replay ratio 16.
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200Episode Return
acrobot-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
cheetah-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
finger-turn_hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
fish-swim
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
hopper-hop
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
hopper-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200Episode Return
humanoid-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
humanoid-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
humanoid-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
pendulum-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
reacher-hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
swimmer-swimmer6
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
walker-run
SR-SAC, RR=32SR-SAC, RR=32 SAC, RR=32SR-SAC, RR=32 SAC, RR=32 Tandem SR-SAC, RR=32SR-SAC, RR=32 SAC, RR=32 Tandem SR-SAC, RR=32 Iterated Offline SR-SAC, RR=32SR-SAC, RR=32 SAC, RR=32 Tandem SR-SAC, RR=32 Iterated Offline SR-SAC, RR=32 REDQ, RR=32
Figure 19: Evaluation Returns on individual DMC15 environments for replay ratio 32.
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
acrobot-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
cheetah-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
finger-turn_hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
fish-swim
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400Episode Return
hopper-hop
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
hopper-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200Episode Return
humanoid-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750Episode Return
humanoid-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750Episode Return
humanoid-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
pendulum-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
quadruped-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
reacher-hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
swimmer-swimmer6
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
walker-run
SR-SAC, RR=64SR-SAC, RR=64 SAC, RR=64SR-SAC, RR=64 SAC, RR=64 Tandem SR-SAC, RR=64SR-SAC, RR=64 SAC, RR=64 Tandem SR-SAC, RR=64 Iterated Offline SR-SAC, RR=64SR-SAC, RR=64 SAC, RR=64 Tandem SR-SAC, RR=64 Iterated Offline SR-SAC, RR=64 REDQ, RR=64
Figure 20: Evaluation Returns on individual DMC15 environments for replay ratio 64.
22

Published as a conference paper at ICLR 2023
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
acrobot-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
cheetah-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
finger-turn_hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600
800Episode Return
fish-swim
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
hopper-hop
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
hopper-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
100
200
300Episode Return
humanoid-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
humanoid-stand
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750Episode Return
humanoid-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
pendulum-swingup
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750
1000Episode Return
quadruped-run
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
quadruped-walk
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
500
1000Episode Return
reacher-hard
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
200
400
600Episode Return
swimmer-swimmer6
0.00 0.25 0.50 0.75 1.00
Environment Steps (×106)
0
250
500
750Episode Return
walker-run
SR-SAC, RR=128SR-SAC, RR=128 SAC, RR=128SR-SAC, RR=128 SAC, RR=128 Tandem SR-SAC, RR=128SR-SAC, RR=128 SAC, RR=128 Tandem SR-SAC, RR=128 Iterated Offline SR-SAC, RR=128SR-SAC, RR=128 SAC, RR=128 Tandem SR-SAC, RR=128 Iterated Offline SR-SAC, RR=128 REDQ, RR=128
Figure 21: Evaluation Returns on individual DMC15 environments for replay ratio 128.
23