The Primacy Bias in Deep Reinforcement Learning — part 2/3

Cleaned-text code points [15806, 31391). Bibliography omitted here; source.md includes it.

After resetting, we do not perform any form of pre-training for the new parameters ( Igl et al., 2021 ) and return directly to standard training, including keeping intact the cycle between environment interactions and agent updates. To provide rigorous evaluations of all algorithms, we follow the guidelines of Agarwal et al. (2021) with the focus on the interquartile mean (IQM) of the performance across tasks.

### 5.2 Resets Consistently Improve Performance

Tables 1 and 1 report the aggregated results for the three algorithms. In both tables, we report the best results over different values of replay ratio and n n for methods with and without resets. The empirical evidence suggests that resets mitigate the primacy bias and provide significant benefits across a wide range of tasks (discrete or continuous action spaces), input types (raw images or dense features), replay buffer configurations (matching or shorter than total number of steps, prioritized replay or random sampling), and depth of the employed neural networks (deep convolutional networks or 3-layer fully-connected networks). Remarkably, the magnitude of improvement provided by resets for SPR is comparable to advancements of prior algorithms, while not requiring additional computation costs.

Method IQM Median Mean SAC + resets 656 (549, 753) 617 (538, 681) 607 (547, 667) SAC 501 (389, 609) 475 (407, 563) 484 (420, 548) DrQ + resets 762 (704, 815) 680 (625, 731) 677 (632, 720) DrQ 569 (475, 662) 521 (470, 600) 535 (481, 589)

### 5.3 Learning Dynamics of Agents with Resets

At first glance, resetting may appear as a drastic (if not wasteful) measure as the agent must learn the parameters of the randomly initialized layers from scratch each time. We show in this section how, against all odds, this strategy still leads to improved performance and fast learning in a wide range of situations.

Figure 4 gives four representative examples of the learning trajectories induced by resets. By default, SAC uses policy and value function architectures which typically contain three layers; in this context, we found that resetting the whole network is an effective strategy. For environments in which the primacy bias does not appear to be an issue, such as cheetah-run , resetting causes some spikes of reduced performance in the learning curves, but the agent is able to return to the previous state in just a few thousands of steps. Instead, when dealing with environments where the algorithm is susceptible to the primacy bias, such as hopper-hop and humanoid-run , resetting not only brings performance back to the previous level but also allows the agent to surpass it.

But why is an RL agent able to recover so fast after each reset? A decisive factor for the effectiveness of resetting resides in preserving the replay buffer across iterations. Indeed, periodically emptying the replay buffer is highly detrimental for performance, as we show in Appendix B . We conjecture that a model-based perspective can offer an explanation: since the replay buffer can be seen as a non-parametric model of the world ( Vanseijen & Sutton, 2015 ; van Hasselt et al., 2019b ) , after a reset, the agent forgets the behaviors learned in the past while preserving its model in the buffer as the core of its knowledge. On the neural network training side, Zhang et al. (2019) observe that learning mostly amounts to recovering the right representations – that is, with the preserved buffer and representations, learning an actuator might be relatively straightforward.

Resets affect learning in a generally positive way, by triggering a virtuous circle. After resetting, the agent is free from the negative priming provided by its past training iterations: it can better leverage the data collected so far, thus improving its performance and unlocking the possibility to generate higher quality data for its future updates.

If the primacy bias is a special form of overfitting, resets can be seen as a tailor-made form of regularization. Table 4 in Appendix B shows that the particular nature of resets allows the agent to overcome the primacy bias even when other forms of regularization such as L2 and dropout would not.

On a practical side, resets are an easy-to-use strategy for addressing the primacy bias. Their use requires making only two choices: the periodicity of the resets and the number of layers of the neural networks to be reinitialized. Moreover, the infrequent re-initialization of a neural network comes with no additional computational cost.

### 5.4 Elements Behind the Success of Resets

The large improvement in performance provided by resets across algorithms and environments naturally raises a question about the conditions under which they are maximally useful. To find an answer, we focus the discussion on the interaction of resets with crucial algorithmic aspects and hyperparameters impacting the risk of overfitting.

#### Replay ratio

The initial experiments in Section 3 suggest that the degree of reliance on early data is a critical determinant of the strength of the primacy bias. As a consequence, we observe that the impact of resets depends heavily on the replay ratio , the number of gradient updates per each environment step. Figure 5 shows that the higher the replay ratio is, the larger the effects from resets: they improve SPR’s performance by over 40% at four updates/step and allow SAC to achieve its highest performance at the high replay ratio of 32, where adding resets increases performance by over 100%. The same phenomenon appears with a doubled amount of data in SPR, implying that the effects of resets are not due to a limited amount of data in the 100k benchmark. Even when pushing SAC to the extreme replay ratios of 128 128 and 256 256 , where learning is barely possible in most environments, resets allow the agent to obtain reasonable performance (see Table 5 in Appendix C ). Resets thus allow less careful tuning of this parameter and improve sample efficiency by performing more updates per each data point without being severely affected by the primacy bias.

#### n n -step targets

The parameter n n in TD learning controls a bias-variance trade-off, with larger values of n n decreasing the bias in value estimates but increasing the variance (and vice versa). We hypothesize that an agent learning from higher variance targets would be more prone to overfitting to the initial data and the effects of resets would increase with increasing n n . Figure 6 confirms the intuition and demonstrates up to 40% improvement for SPR for n = 20 n=20 and opposed to no improvement for n = 3 n=3 . Likewise, SAC attains 50–60% improvement for increased values of n n compared to 40% for the default n = 1 n=1 .

The results with varying replay ratios and n n -step targets suggest that resets reshape the hyperparameter landscape creating a new optimum with higher performance.

#### TD failure modes

Temporal-difference learning, when employed jointly with function approximation and off-policy training, is known to be potentially unstable ( Sutton & Barto, 2018 ) . In sparse-reward environments, the critic network might converge to a degenerate solution because of bootstrapping mostly on it’s own outputs ( Kumar et al., 2020 ) ; having the non-zero reward data might not sufficient to escape a collapse. For example, Figure 7 (left) demonstrates the behavior of DrQ in cartpole-swingup_sparse , where a collapsed agent makes no learning progress. However, after a manual examination of a replay buffer, we found that the agent was reaching goal states in roughly 2% of trajectories. This observation provides evidence for an explanation that mitigating the primacy bias addressed the issues of optimization rather than exploration . Likewise, if divergence in temporal difference learning occurs, it is essentially unrecoverable by standard optimization, with predicted values failing to decay to normal magnitudes after hundreds of thousands of steps. Figure 7 shows that adding resets solves this problem by giving the agent a second chance to find a stable solution. Even though there exist works studying in detail the pathological behaviors of TD learning ( Bengio et al., 2020 ) and this is not the main scope of our work, resets naturally address the outlined failures.

#### What and how to reset

Our particular choice of the resetting strategy calls for a number of ablations. This paragraph provides only the conclusions while Appendix B presents supporting figures. The number of layers to reset is a domain-dependent choice. For the SAC algorithm learning from dense state features, it is possible to reset the agent entirely. Resetting in SPR attains the best performance for the last layer only (out of 5 total layers), while for DrQ resetting the last layer is slightly worse than resetting last 3 out of 7 layers. We conjecture that the difference lies in the degree of representation learning required for each domain: a significant chunk of knowledge in Atari is contained in the agent’s representations; it might be notably easier to learn features in DeepMind Control, especially when dealing with dense states. In DrQ, when resetting both actor and critic, resetting critic proved to be slightly more important than the actor; likely because the DrQ encoder learns from critic loss only, a practice proposed in ( Yarats et al., 2021 ) .

Another seemingly important choice is whether to reset the state of the optimizer. We find that resetting the optimizer has almost no impact on training because the moment estimates are updated quickly. Regarding the resets frequency, the optimal choice should depend on how fast an algorithm can recover and how much it is affected by the primacy bias; we found that sometimes even a single reset improves the performance of baseline agents. We briefly experimented with resetting a random subnetwork and observed that the performance was either comparable or worse than with resetting the last layers. Lastly, when sampling new weights after a reset, it is natural to use a new random seed; we observed that even with the same seed resets alleviate the primacy bias supporting the conclusions of Bjorck et al. (2022) that pathologies in deep RL algorithms are not due to problems with the initialization. Overall, while we see certain differences when varying the discussed design choices, resetting showed itself to be robust the choice of hyperparameters.

### 5.5 Summary

In short, the experimental results suggest that resets improve expected returns across a diverse set of environments and algorithms. They act as a form of regularization thus preventing overfitting to early data, unlock new hyperparameter configurations with possibly higher performance and sample efficiency, and address the optimization challenges arising in deep RL. While with a more thorough hyperparameter search and additional modifications it is possible to even further improve the performance upon the baselines, it is exciting to see that the proposed simple resetting scheme gives benefits comparable to previous algorithmic advancements.

## 6 Related Work

The primacy bias in deep RL is intimately related to memorization, optimization in RL, and cognitive science. Various aspects of our work existed in the literature.

#### Overfitting in RL

Generalization and overfitting have many faces in deep reinforcement learning. Generalization of values to similar states is a setting where the most classical form of overfitting can arise when using function approximation ( Sutton & Barto, 2018 ) . Kumar et al. (2020) and Lyle et al. (2022) show that an approximator for value function gradually loses its expressivity due to bootstrapping in TD learning; we conjecture that this amplifies the effect of first data points. Dabney et al. (2021) propose to treat holistically the sequence of value prediction tasks, arguing that if an agent focuses too much on a single prediction problem, it might overspecialize it’s representations to it. Song et al. (2019) study observational overfitting by examining saliency maps in visual domains and argue that agents might pick up spurious correlations for decision making. The prioneering work ( Farahmand et al., 2008 ) and a more recent one ( Liu et al., 2021 ) argue in favor of using regularization such as L2 in RL. Finally, overfitting can happen in settings with learning from offline datasets ( Fujimoto et al., 2019 ) and with multiple tasks ( Teh et al., 2017 ) . Surveys by Kirk et al. (2021) and Zhang et al. (2018) give a taxonomy of generalization aspects in RL.

Techniques similar to our resets also existed before. Anderson et al. (1993) propose to reset individual neurons of a Q-network based on a variance criterion and observes faster convergence. Forms of non-uniform sampling including re-weighting recent samples ( Wang et al., 2020 ) and prioritized experience replay (PER) ( Schaul et al., 2016 ) can be seen as a way to mitigate the primacy bias. We observe that SPR, built on top of Rainbow ( Hessel et al., 2018 ) and already using PER, still benefits from resets. Igl et al. (2021) provide demonstrations in the supervised case that learning from a pretrained network can be worse than learning from scratch for non-stationary datasets and propose a method ITER that fully resets the agent’s network in an on-policy buffer-free setting with distillation from the previous generation as a knowledge transfer mechanism. The difference contrasts the approach with our resetting scheme that uses a replay buffer as a basis for knowledge transfer after a reset.

Our work sheds light on another special form of overfitting in deep RL and proposes a simple solution for mitigating it.

#### Forgetting mechanisms

In contrast to the well-known phenomenon of catastrophic forgetting ( French, 1999 ) , several works have noted the opposite effect of catastrophic memorization ( Robins, 1993 ; Sharkey & Sharkey, 1995 ) similar to the primacy bias. Achille et al. (2018) observe the existence of critical phases in learning that have a determining effect on the resulting network. Erhan et al. (2010) notice higher sensitivity of the trained network with respect to first datapoints. While the effect of the early examples in supervised problems might be present, the consequences of overfitting to initial experiences in deep RL would be much more drastic because the agent itself collects the data to learn from. Dohare et al. (2021) adjust stochastic gradient descent for the continual learning setting; we highlight that in continual learning the agent does not have influence over the stream of data, while the primacy bias in deep RL arises because data collection is in the training loop.

The idea of resetting subnetworks recently received more attention in supervised learning. Taha et al. (2021) studies this process from an evolutionary perspective and shows increased performance in computer vision tasks. Zhou et al. (2022) show that some degree of forgetting might improve generalization and draw a connection to the emergence of compositional representations ( Ren et al., 2019 ) . Zhang et al. (2019) demonstrate that resetting different layers affect differently the performance of a network. Alabdulmohsin et al. (2021) demonstrate that resets increase margins of training examples and induce convergence to flatter minima.

These works complement the evidence about the existence of the primacy bias in deep RL and add to our analysis of the regularization effect of resets.

#### Cognitive science