The Primacy Bias in Deep Reinforcement Learning

arxiv:2205.07802

Original source · Paper JSON

Reinforcement learning · 2022-05-16

Cites (4) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• The Primacy Bias in Deep Reinforcement Learning
• Abstract
• Keywords:
• 1 Introduction
• 2 Preliminaries
• 3 The Primacy Bias
• 3.1 Heavy Priming Causes Unrecoverable Overfitting
• 3.2 Experiences of Primed Agents are Sufficient
• 4 Have You Tried Resetting It?
• 5 Experiments
• 5.1 Setup
• 5.2 Resets Consistently Improve Performance
• 5.3 Learning Dynamics of Agents with Resets
• 5.4 Elements Behind the Success of Resets
• Replay ratio
• n n -step targets
• TD failure modes
• What and how to reset
• 5.5 Summary
• 6 Related Work
• Overfitting in RL
• Forgetting mechanisms
• Cognitive science
• 7 Future Work and Limitations
• 8 Conclusion
• Acknowledgements
• Appendix A Experimental Details
• Appendix B Ablations
• Replay buffer
• Initialization
• Optimizer state
• Reset depth
• Which networks to reset
• Number of resets
• Other regularizers
• Appendix C Per-Environment and Additional Results
• Instructions for reporting errors