The Primacy Bias in Deep Reinforcement Learning — part 3/3

Cleaned-text code points [31391, 46704). Bibliography omitted here; source.md includes it.

The primacy bias (also known as the primacy effect ) is a well-studied cognitive bias in human learning ( Marshall & Werder, 1972 ) . Given a sequence of facts, humans often form generalizations based on the first ones and pay less attention to the later ones. Asch (1961) shows that this tendency can foster a creation of harmful prejudices by examining the difference in responses after presenting the same data but in different order. Shteingart et al. (2013) argue that outcomes of first experiences affect future decision making and have a substantial and lasting effect on subsequent behavior. Yalnizyan-Carson & Richards (2021) use RL as a framework to test a hypothesis that some degree of forgetting in natural brains is beneficial for decision making. Resets can be linked to cultural transmission between generations ( Kirby, 2001 ) , where an agent before a reset transmits its knowledge to an agent after the reset through a buffer. Lastly, studies of a critical period ( Johnson & Newport, 1989 ) show that if a child fails to develop a skill during a particular stage of its development, it might much more difficult to acquire the skill later, drawing the connection to proper learning from early experiences.

Even though humans and RL systems learn under different conditions, our paper provides evidence that artificial agents exhibit the primacy bias noted in humans.

## 7 Future Work and Limitations

This paper focuses on empirical investigation of the primacy bias phenomenon. An exciting avenue for future work is developing theoretical understanding of risks associated to overfitting to first experiences. Likewise, deriving guarantees for learning with resets similarly to the results of Li et al. (2021) in games and Besson & Kaufmann (2018) for bandits would make the technique more theoretically sound.

Our version of resets is appealing because of its simplicity. However, the reset periodicity is a hyperparameter that an RL practitioner needs to choose. A version of the technique based on the feedback from the RL system or even meta-learning the resetting strategy can potentially improve the performance even further.

Finally, we note that brief collapses in performance induced by resetting may be undesirable from a regret minimization perspective. Potential remedies include having a period of offline post-training after each reset or sampling actions from an interpolation between pre- and post-reset agents for some period of time after each reset.

## 8 Conclusion

This paper identifies the primacy bias in deep RL, a damaging tendency of artificial agents to overfit early experiences. We demonstrate dangers associated with this form of overfitting and propose a simple solution based on resetting a part of the agent. The experimental evidence across domains and algorithms suggests that resetting is an effective and generally applicable technique in RL.

The general trend in RL research for many years was to first establish a sound algorithm for the tabular case and then use a neural network for representing parts of the agent. The last step was not rarely seen as just a technical detail. The primacy bias, however, is a phenomenon specific to RL with function approximation. The findings of our paper point to the importance of studying the profound interaction of reinforcement and deep learning. Similarly to techniques like Batch Normalization ( Ioffe & Szegedy, 2015 ) that revolutionized training of supervised models, future progress might come not only from advancements in core RL but rather by approaching the problem in conjunction. It is striking that something as simple as periodic resetting improves performance so drastically, suggesting that there is still much to understand about the behavior of deep RL.

Overall, this work sheds light on the learning processes of deep RL agents, unlocks training regimes that were unavailable without resets, and opens possibilities for further studies improving both understanding and performance of deep reinforcement learning algorithms.

## Acknowledgements

The authors thank Rishabh Agarwal, David Yu-Tung Hui, Ilya Kostrikov, Ankit Vani, Zhixuan Lin, Tristan Deleu, Wes Chung, Mandana Samiei, Hattie Zhou, Marc G. Bellemare for insightful discussions and useful suggestions on the early draft, Compute Canada for computational resources, and Sara Hooker for Isaac Asimov’s quote. This work was partially supported by Facebook CIFAR AI Chair, Samsung, Hitachi, IVADO, and Gruppo Ermenegildo Zegna.

We acknowledge the Python community ( Van Rossum & Drake Jr, 1995 ; Oliphant, 2007 ) for developing the core set of tools that enabled this work, including JAX ( Bradbury et al., 2018 ; Babuschkin et al., 2020 ) , Jupyter ( Kluyver et al., 2016 ) , Matplotlib ( Hunter, 2007 ) , numpy ( Oliphant, 2006 ; Van Der Walt et al., 2011 ) , pandas ( McKinney, 2012 ) , and SciPy ( Jones et al., 2014 ) .

## Appendix A Experimental Details

We use an open-source JAX implementation ( Kostrikov, 2021 ) of the SAC and DrQ algorithms and an open-source JAX implementation 2 2 2 https://github.com/MaxASchwarzer/dopamine/tree/atari100k_spr of SPR. This SPR implementation exhibit a slightly higher aggregate performance than the scores in Schwarzer et al. (2020) based on a PyTorch implementation. All algorithms use default hyperparameters unless specified otherwise (for example, in experiments with replay ratio and n n -step targets). SAC and DrQ experiments use 10 random seeds for evaluating the performance, DrQ uses 20 random seeds.

Tables 2 and 3 report the sets of DeepMind Control Suite tasks for testing SAC and DrQ algorithms respectively. Note that SAC learns from dense states, while DrQ learns from raw pixel observations. SPR trains on a standard set of 26 tasks from the Atari 100k benchmark used by Kaiser et al. (2019) ; van Hasselt et al. (2019a) ; Schwarzer et al. (2020) . The list of all Atari environments is available in Table 6 .

## Appendix B Ablations

Section 5.4 outlines the interaction of different moving parts of RL algorithms and our proposed reset strategy. This appendix elaborates on our observations and provides supporting figures to help understanding the effect of resets. We now show, one by one, how resetting behaves under a diverse set of controlled settings.

#### Replay buffer

The reset mechanism we propose is a form of forgetting based on retaining all of the collected knowledge, stored in the replay buffer, and not retaining a part of the learned behavior, stored in the parameters of an agent’s function approximators. How critical is it to preserve the knowledge in the replay buffer? We test its importance in DrQ by periodically resetting the replay buffer in addition to the last layers. Figure 8 shows that keeping the buffer is essential; resetting it amounts to learning almost from scratch. These results suggests that knowledge retention is significantly more important than behavior retention for preventing the negative effects of the primacy bias and simultaneously being able to recover from resets.

#### Initialization

After each reset, the new parameters are sampled from a canonical initialization distribution followed by the algorithms. To understand whether the susceptibility of an agent to the primacy bias is just a consequence of an unlucky initialization of one of the layers of its neural networks, we run DrQ with resets, but set the value of the re-initialized parameters to the one they had at the beginning of training (i.e., by initializing them with the same seed). The results in Figure 8 show that the performance of this variant of our reset strategy is almost identical to the original version: the problem alleviated by resets is not one of pathological initializations, in line with findings of other works ( Bjorck et al., 2021 ) , but instead one resulting from the peculiar interactions happening when learning from a growing dataset of interactions.

#### Optimizer state

As highlighted in the main text, resetting the optimizer’s moment estimates together with the corresponding neural network parameters have almost no effect. We show results demonstrating this for DrQ (see Figure 9 ). We believe this is mostly due to the momentum-based optimizers: with Adam with default parameters, the first moment ( β 1 = 0.9 \beta_{1}=0.9 ) will vanish in about 10 updates, the second moment ( β 2 = 0.999 \beta_{2}=0.999 ) after 1000 updates. On the scale of our tasks, it is indeed a quite rapid recovery time.

#### Reset depth

One of the two hyperparameters introduced by our reset strategy on top of any backbone algorithm is the number of layers of the agent’s neural networks to be re-initialized. We investigated the impact of this choice for both SPR and DrQ, while sticking for SAC to the default choice of re-initializing all networks. Results for DrQ in Figure 10 demonstrate that resetting the last layer yields slightly inferior performance to resetting the entire Q-learning head (3 layers). For SPR, as shown in Figure 13 , we found the reverse to be true. Thus, how many layers to reset is a hyperparameter that may need tuning, and the choice can be informed by the difficulty of representation learning for each domain. We recommend starting the exploration of this hyperparameter from resetting the last 1-3 layers.

#### Which networks to reset

In our experiments, we reset a subset of the value function parameters in SPR and a subset of the parameters of all the trained neural networks in DrQ and SAC. The latter two algorithms use three groups of function approximators: an actor, a critic, and a target critic. We investigate the impact of resetting each one of these modules in DrQ. Results in Figure 11 show that a simultaneous reset of all the neural networks is generally the most robust technique to improve performance over the backbone algorithm, while resetting the critic had the most impact on the performance. We have also tried a version of resets where each weight is re-initialized with probability 0.5. Figure 10 shows that such a random subnetwork resetting was either on par or worse than the standard scheme.

#### Number of resets

Intuitively, the primacy bias affects the agent in a progressively milder way after each reset. It is natural to ask whether a limited number of resets, or even a single one, is sufficient to overcome the effects of overfitting to initial experiences. We test this hypothesis using DrQ, showing in Figure 12 that, despite the first reset contributing the most to mitigating the primacy bias, it is not always sufficient to reach the same performance of the standard continual resetting strategy. As a default choice, we recommend using the reset periodicity resulting in 3-10 resets over the course of training.

#### Other regularizers

Resets can be seen as a form of regularization because they implicitly constrain the final solutions to be not too far from the initial parameters. However, they specifically tackle the primacy bias better than other common forms of regularization. To test this conjecture, we repeat the heavy priming experiment on the quadruped-run task with standard L2 regularization of both critic and actor weights of SAC. We find that no value of regularization coefficient among the set [ 10 − 5 , 3 ⋅ 10 − 5 , 10 − 4 , 3 ⋅ 10 − 4 , 10 − 3 , 3 ⋅ 10 − 3 , 10 − 2 , 3 ⋅ 10 − 2 , 10 − 1 , 3 ⋅ 10 − 1 , 1.0 ] [10^{-5},3\cdot 10^{-5},10^{-4},3\cdot 10^{-4},10^{-3},3\cdot 10^{-3},10^{-2},3\cdot 10^{-2},10^{-1},3\cdot 10^{-1},1.0] can overcome heavy priming, obtaining results almost identical to the ones reported on Figure 1 . The heavy priming setting artificially creates the conditions for the effect of the primacy bias to be particularly highlighted. To test whether resets offer superior performance also in the context of standard training of reinforcement learning algorithms, we compare the performance of SAC and DrQ enriched with standard regularization methods to that of SAC and DrQ augmented with resets. In particular, we leverage L2 regularization of both the actor and the critic, as well as dropout ( Srivastava et al., 2014 ) . We report in Table 4 the best value over the grid [ 10 − 4 , 5 ⋅ 10 − 4 , 10 − 3 , 5 ⋅ 10 − 3 ] [10^{-4},5\cdot 10^{-4},10^{-3},5\cdot 10^{-3}] suggested by ( Liu et al., 2021 ) for L2 and the standard grid [ 0.5 , 0.1 ] [0.5,0.1] for dropout. For SAC-based approaches, we also sweep over the replay ratios [ 1 , 9 , 32 ] [1,9,32] and report the best result for other regularizers. The table shows that not only standard regularization methods are not better than resets in the context of these RL algorithms, but that they do not provide any benefit, on aggregate performance, compared to the baselines.

Method IQM Median Mean SAC 501 (389, 609) 475 (407, 563) 484 (420, 548) SAC + resets 656 (549, 753) 617 (538, 681) 607 (547, 667) SAC + dropout 219 (160, 285) 254 (204, 307) 258 (216, 300) SAC + L2 412 (299, 524) 415 (337, 495) 416 (351, 481) DrQ 569 (475, 662) 521 (470, 600) 535 (481, 589) DrQ + resets 762 (704, 815) 680 (625, 731) 677 (632, 720) DrQ + dropout 492 (414, 567) 480 (420, 541) 479 (431, 527) DrQ + L2 463 (362, 566) 473 (403, 541) 472 (415, 529)

## Appendix C Per-Environment and Additional Results

The remainder of the appendix presents results for each task and supplementary plots for training with varying replay ratios and n n -step targets.

Figure 14 demonstrates learning curves for SPR. We note that the low loss and high parameter norm for high n n and replay ratios might indicate the symptoms of overfitting. Whilst resets implicitly control the weight norm, doing so explicitly through L2 regularization proved to be less effective for mitigating heavy priming.

Table 5 presents the aggregate metrics for the combinations of n n and replay ratios in SAC. We additionally probe extreme replay ratios of 128 and 256 and observe that, even in these cases, learning with resets delivers meaningful performance, while the no-reset agent achieves near-zero returns.

Lastly, per-environment training curves for SAC as well as the results for varying replay ratios and n n -step targets are available in Figures 15 and 17 respectively. Per-environment training curves for DrQ are available in Figure 18 . Table 6 provides scores for SPR in all Atari 100k tasks.

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .