Sample-Efficient RL by Breaking the Replay Ratio Barrier (ICLR 2023, precursor of BBF) — part 4/6

Cleaned-text code points [37726, 52107). Bibliography omitted here; source.md includes it.

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
The authors thank Zhixuan Lin for adapting the REDQ baseline, Nathan U. Rahn, Rishabh Agarwal,
David Yu-Tung Hui, Jesse Farebrother for insightful discussions and useful suggestions on the early
draft, the Mila community for creating a stimulating research environment, Digital Research Alliance
of Canada and Nvidia for computational resources. This work was partially supported by CIFAR,
Samsung, Hitachi, Facebook AI Chair, Borealis, IV ADO, and Gruppo Ermenegildo Zegna.
We acknowledge the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for
developing the core set of tools that enabled this work, including JAX (Bradbury et al., 2018;
Babuschkin et al., 2020), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), numpy (Oliphant,
2006; Van Der Walt et al., 2011), pandas (McKinney, 2012), and SciPy (Jones et al., 2014).
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare.
Deep reinforcement learning at the edge of the statistical precipice. In Thirty-Fifth Conference on
Neural Information Processing Systems, 2021.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in Neural
Information Processing Systems, 33:3884–3894, 2020.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David
Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Claudio Fantacci, Jonathan Godwin, Chris Jones,
Tom Hennigan, Matteo Hessel, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King,
Lena Martens, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring,
Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan
Srinivasan, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL
http://github.com/deepmind.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural
scaling laws. arXiv preprint arXiv:2102.06701, 2021.
Tudor Berariu, Wojciech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and
Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint arXiv:2106.00042,
2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx,
Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni-
ties and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal
Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and
Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github.com/google/jax.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc G. Bellemare.
Dopamine: A research framework for deep reinforcement learning. CoRR, abs/1812.06110, 2018.
URL http://arxiv.org/abs/1812.06110.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian
walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the
European Conference on Computer Vision (ECCV), pp. 532–547, 2018.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning:
Learning fast without a model. arXiv preprint arXiv:2101.05982, 2021.
Will Dabney, Andr´e Barreto, Mark Rowland, Robert Dadashi, John Quan, Marc G Bellemare, and
David Silver. The value-improvement path: Towards better representations for reinforcement
learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7160–
7168, 2021.
10

Published as a conference paper at ICLR 2023
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander
Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D’Amour, Dan Moldovan, et al. On
robustness and transferability of convolutional neural networks. In Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 16458–16468, 2021.
Shibhansh Dohare, Richard S. Sutton, and A. Rupam Mahmood. Continual backprop: Stochastic gra-
dient descent with persistent randomness, 2022. URL https://openreview.net/forum?
id=86sEVRfeGYS.
Pierluca D’Oro and Wojciech Ja´skowski. How to learn a useful critic? model-based action-gradient-
estimator policy optimization. Advances in Neural Information Processing Systems, 33:313–324,
2020.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark
Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In International
Conference on Machine Learning, pp. 3061–3071. PMLR, 2020.
Vincent Franc ¸ois-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, et al. An
introduction to deep reinforcement learning. Foundations and Trends® in Machine Learning, 11
(3-4):219–354, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Mohammad Ghavamzadeh, Hilbert Kappen, Mohammad Azar, and R ´emi Munos. Speedy
q-learning. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger
(eds.), Advances in Neural Information Processing Systems , volume 24. Curran Asso-
ciates, Inc., 2011. URL https://proceedings.neurips.cc/paper/2011/file/
ab1a4d0dd4d48a2ba1077c4494791306-Paper.pdf.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum
entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
Jessica B Hamrick, Abram L. Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Wither-
spoon, Thomas Anthony, Lars Holger Buesing, Petar Veliˇckovi´c, and Theophane Weber. On the role
of planning in model-based deep reinforcement learning. In International Conference on Learning
Representations, 2021. URL https://openreview.net/forum?id=IrM64DGB21.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad,
Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable,
empirically. arXiv preprint arXiv:1712.00409, 2017.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsu-
ruoka. Dropout q-functions for doubly efficient reinforcement learning. In International Confer-
ence on Learning Representations, 2022. URL https://openreview.net/forum?id=
xCVJMsPv3RT.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom
Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy,
Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training
compute-optimal large language models. ArXiv, abs/2203.15556, 2022.
G Zacharias Holland, Erin J Talvitie, and Michael Bowling. The effect of planning shape on dyna-style
planning in high-dimensional state spaces. arXiv preprint arXiv:1806.01825, 2018.
John D Hunter. Matplotlib: A 2d graphics environment. IEEE Annals of the History of Computing, 9
(03):90–95, 2007.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson.
Transient non-stationarity and generalisation in deep reinforcement learning. In ICLR, 2021.
Eric Jones, Travis Oliphant, and Pearu Peterson. SciPy: Open source scientific tools for Python .
2014.
11

Published as a conference paper at ICLR 2023
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell,
K. Czechowski, D. Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi,
G. Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. ArXiv,
abs/1903.00374, 2020.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott
Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv,
abs/2001.08361, 2020.
Kacper Piotr Kielak. Do recent advancements in model-based deep reinforcement learning really
improve data efficiency? 2019.
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando P´erez, Brian E Granger, Matthias Bussonnier,
Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. Jupyter
Notebooks-a publishing format for reproducible computational workflows., volume 2016. 2016.
Ilya Kostrikov. JAXRL: Implementations of Reinforcement Learning algorithms in JAX, 10 2021.
URL https://github.com/ikostrikov/jaxrl.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit
q-learning. ArXiv, abs/2110.06169, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline
reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-parameterization
inhibits data-efficient deep reinforcement learning. ArXiv, abs/2010.14498, 2021.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, L. Y . Lee, Daniel Freeman, Winnie Xu, Sergio
Guadarrama, Ian S. Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game
decision transformers. ArXiv, abs/2205.15241, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial,
review, and perspectives on open problems, 2020. URL https://arxiv.org/abs/2005.
01643.
Long-Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University,
1992.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International
Conference on Machine Learning, pp. 6736–6747. PMLR, 2021.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in
reinforcement learning. ArXiv, abs/2204.09560, 2022a.
Clare Lyle, Mark Rowland, Will Dabney, Marta Z. Kwiatkowska, and Yarin Gal. Learning dynamics
and generalization in reinforcement learning. ArXiv, abs/2206.02126, 2022b.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-
efficient reinforcement learning via model-based offline optimization. In International Conference
on Learning Representations, 2021.
Wes McKinney. Python for data analysis: Data wrangling with Pandas, NumPy, and IPython . ”
O’Reilly Media, Inc.”, 2012.
Vincent Micheli, Eloi Alonso, and Franc ¸ois Fleuret. Transformers are sample efficient world models,
2022. URL https://arxiv.org/abs/2209.00588.
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare,
Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control
through deep reinforcement learning. nature, 518(7540):529–533, 2015a.
12

Published as a conference paper at ICLR 2023
V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle-
mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen,
Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra,
Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning.
Nature, 518:529–533, 2015b.
Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron C. Courville. The
primacy bias in deep reinforcement learning. In ICML, 2022.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Travis E Oliphant. Python for scientific computing. Computing in Science & Engineering , 9(3):
10–20, 2007.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep
reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Martin Riedmiller. Neural fitted q iteration–first experiences with a data efficient neural reinforcement
learning method. In European conference on machine learning, pp. 317–328. Springer, 2005.
Martin A. Riedmiller, Jost Tobias Springenberg, Roland Hafner, and Nicolas Manfred Otto Heess.
Collect & infer - a fresh look at data-efficient reinforcement learning. In CoRL, 2021.
Tom Schaul, Andr´e Barreto, John Quan, and Georg Ostrovski. The phenomenon of policy churn.
arXiv preprint arXiv:2206.00730, 2022.
Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip
Bachman. Data-efficient reinforcement learning with self-predictive representations. In ICLR,
2021a.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De-
von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient
reinforcement learning. Advances in Neural Information Processing Systems, 34:12686–12699,
2021b.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes
with model-free reinforcement learning. ArXiv, abs/2208.07860, 2022.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden,
Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller.
Deepmind control suite. ArXiv, abs/1801.00690, 2018.
Stefan Van Der Walt, S Chris Colbert, and Gael Varoquaux. The numpy array: a structure for efficient
numerical computation. Computing in science & engineering, 13(2):22–30, 2011.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in
reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Guido Van Rossum and Fred L Drake Jr. Python tutorial, volume 620. Centrum voor Wiskunde en
Informatica Amsterdam, 1995.
Ziyu Wang, Victor Bapst, Nicolas Heess, V olodymyr Mnih, Remi Munos, Koray Kavukcuoglu,
and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint
arXiv:1611.01224, 2016.
Yanqiu Wu, Xinyue Chen, Che Wang, Yiming Zhang, Zijian Zhou, and Keith W. Ross. Aggressive q-
learning with ensembles: Achieving both high sample efficiency and high asymptotic performance,
2022. URL https://openreview.net/forum?id=NOApNZTiTNU.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist
networks. In International Conference on Learning Representations , 2022. URL https://
openreview.net/forum?id=ei3SY1_zYsE.
13