The Primacy Bias in Deep Reinforcement Learning — part 1/3

Cleaned-text code points [0, 15806). Bibliography omitted here; source.md includes it.

# The Primacy Bias in Deep Reinforcement Learning

###### Abstract

This work identifies a common flaw of deep reinforcement learning (RL) algorithms: a tendency to rely on early interactions and ignore useful evidence encountered later. Because of training on progressively growing datasets, deep RL agents incur a risk of overfitting to earlier experiences, negatively affecting the rest of the learning process. Inspired by cognitive science, we refer to this effect as the primacy bias . Through a series of experiments, we dissect the algorithmic aspects of deep RL that exacerbate this bias. We then propose a simple yet generally-applicable mechanism that tackles the primacy bias by periodically resetting a part of the agent. We apply this mechanism to algorithms in both discrete (Atari 100k) and continuous action (DeepMind Control Suite) domains, consistently improving their performance.

###### Keywords:

“Your assumptions are your windows on the world. Scrub them off every once in a while, or the light won’t come in.”

–Isaac Asimov

## 1 Introduction

Bob is learning a difficult passage on a guitar to rehearse it with his band. After long nights of practice, he is able to play it but with straining finger positions and unclean sound. Alice, another fellow guitarist, shows him a less fatiguing and nicer-sounding way to play the passage which Bob is theoretically able to understand and execute well. Nonetheless, in subsequent rehearsal sessions, Bob’s unconscious mind automatically resorts to the first bad-sounding solution, discouraging him and his bandmates from trying more technically challenging pieces of music. Bob is experiencing an instance of the primacy bias , a cognitive bias demonstrated by studies of human learning ( Marshall & Werder, 1972 ; Shteingart et al., 2013 ) .

The outcomes of first experiences can have long-lasting effects on subsequent learning and behavior. Thanks to Alice, Bob has already collected new data sufficient for improving his performance on that passage and guitar playing in general; however, because of the priming provided by his long training nights with a bad technique, he is unable to leverage his new experience. The primacy bias generates a vicious circle: since Bob cannot improve his guitar skills in the face of new data, he will not be able to collect more interesting data by playing more challenging guitar passages, crippling his overall learning.

The central finding of this paper is that deep reinforcement learning (RL) algorithms are susceptible to a similar bias. We refer to the primacy bias in deep RL as a tendency to overfit early interactions with the environment preventing the agent from improving its behavior on subsequent experiences. The consequences of this phenomenon compound: an agent with poor performance will collect data of poor quality and the poor data will amplify the difficulty of recovering from an overfitted solution.

Standard components of deep RL algorithms magnify the effect of the primacy bias. For instance, experience replay ( Mnih et al., 2015 ) allows efficient data reuse but exposes the agent to its initial samples more than recent ones. In the interest of sample efficiency, deep RL algorithms often additionally use a high replay ratio ( van Hasselt et al., 2019b ; Fedus et al., 2020 ) updating the agent more times on the same data. Such design choices can improve the agent’s performance but come with a risk of exacerbating the effects of early interactions.

How can a deep RL algorithm avoid the primacy bias? Coming back to Bob, he could re-establish his learning progression by simply forgetting his bad practices and directly learning from newer experience. Similarly, deep RL agents affected by the primacy bias can forget parts of a solution which was derived by overfitting to early experiences before continuing the learning process.

As a remedy for the primacy bias, we propose a resetting mechanism allowing the agent to forget a part of its knowledge. Our strategy is simple and compatible with any deep RL algorithm equipped with a replay buffer: we periodically re-initialize the last layers of an agent’s neural networks, while maintaining the experience within the buffer.

Despite its simplicity, this resetting mechanism consistently improves performance of agents on benchmarks including the discrete-action ALE ( Bellemare et al., 2013 ) and the continuous-action DeepMind Control Suite ( Tassa et al., 2020 ) . Our strategy imposes no additional computational costs and requires only two implementation choices: which parts of the neural networks to reset and how often to reset them. We also show that resetting enables training regimes with higher replay ratio and longer n n -step targets ( Sutton & Barto, 2018 ) , where an agent without resets would be overfitting otherwise.

To summarize the contributions of this paper, we: 1. Provide demonstrations of the existence of the primacy bias in deep RL, a tendency of an agent to harm its future decision making by overfitting to early data and ignoring subsequent interactions;

2. Expose plausible causes of this phenomenon and show how algorithmic aspects in modern deep RL amplify its consequences;

3. Propose a mechanism for alleviating the primacy bias by periodically resetting a part of the agent;

4. Empirically demonstrate both qualitative and quantitative improvements in performance when applying resets to strong baseline algorithms.

## 2 Preliminaries

We adopt the standard formulation of reinforcement learning ( Sutton & Barto, 2018 ) under the Markov decision process (MDP) formalism where the agent observes a state s s from a space 𝒮 \mathcal{S} , chooses an action a a from a space 𝒜 \mathcal{A} , and receives a reward r r according to a mapping r : 𝒮 × 𝒜 → ℝ r:\mathcal{S}\times\mathcal{A}\to\mathbb{R} . The environment then transitions into a state s ′ s^{\prime} according to a distribution p : 𝒮 × 𝒜 → Δ ⁡ ( 𝒮 ) {p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})} and the interaction continues. The initial state s 0 s_{0} is sampled from a distribution ρ ∈ Δ ⁡ ( 𝒮 ) \rho\in\Delta(\mathcal{S}) . The goal of the agent is to learn a policy π : 𝒮 → Δ ⁡ ( 𝒜 ) \pi:\mathcal{S}\to\Delta(\mathcal{A}) that maximizes the expected discounted sum of rewards 𝔼 π ​ [ ∑ t = 0 ∞ γ t ​ r ​ ( s t , a t ) ] \mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})\right] with γ ∈ [ 0 , 1 ) \gamma\in[0,1) .

RL methods typically learn an action-value function Q π ( s , a ) = 𝔼 π [ ∑ t = 0 ∞ γ t r ( s t , a t ) | s 0 = s , a 0 = a ] Q_{\pi}(s,a)=\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a\right] though temporal-difference (TD) learning ( Sutton, 1988 ) by minimizing the difference between Q π ​ ( s , a ) Q_{\pi}(s,a) and 𝔼 p ⁡ ( s ′ | s , a ) , π ⁡ ( a ′ | s ′ ) ​ [ r ⁡ ( s , a ) + γ ​ Q π ​ ( s ′ , a ′ ) ] \mathbb{E}_{p(s^{\prime}|s,a),\pi(a^{\prime}|s^{\prime})}\left[r(s,a)+\gamma Q_{\pi}(s^{\prime},a^{\prime})\right] . Many RL algorithms store past experiences in a replay buffer ( Lin, 1992 ; Mnih et al., 2015 ) that increases sample efficiency by leveraging a single data point more than once. The number of resampling times of given data is controlled by the replay ratio ( van Hasselt et al., 2019b ; D’Oro & Jaśkowski, 2020 ) which plays a critical role in the algorithm’s performance. Set too low, the agent would underuse the data it has and become sample-inefficient; set too high, the agent would overfit the existing data.

The core idea behind temporal-difference methods is bootstrapping , learning from agent’s own value estimates without the need to wait until the end of the interaction. TD learning can be generalized by using n n -step targets 𝔼 π ​ [ r ⁡ ( s t , a t ) + γ ​ r ​ ( s t + 1 , a t + 1 ) + ⋯ + γ n ​ Q π ​ ( s t + n , a t + n ) ] \mathbb{E}_{\pi}\left[r(s_{t},a_{t})+\gamma r(s_{t+1},a_{t+1})+\dots+\gamma^{n}Q_{\pi}(s_{t+n},a_{t+n})\right] . Here, n n controls a trade-off between the (statistical) bias of Q π Q_{\pi} estimates and the variance of the sum of future rewards.

In the rest of the paper, we consider deep RL algorithms where Q π Q_{\pi} and π \pi (when needed) are modelled by neural network function approximators.

## 3 The Primacy Bias

The main goal of this paper is to understand how the learning process of deep reinforcement learning agents can be disproportionately impacted by initial phases of training due to an effect called the primacy bias.

The Primacy Bias in Deep RL: a tendency to overfit initial experiences that damages the rest of the learning process .

This definition is wide-ranging: the primacy bias has multiple roots and leads to multiple negative effects for the training of an RL agent, but they are all connected to improper learning from early data.

The rest of this section presents two experiments, with the goal of demonstrating the existence and the dynamics of the phenomenon in isolation. First, we show that excessive training of an agent on the very first interactions can fatally damage the rest of the learning process. The second experiment demonstrates that data collected by an agent impacted by the primacy bias is adequate for learning, although the agent cannot leverage it due to its accumulated overfitting.

### 3.1 Heavy Priming Causes Unrecoverable Overfitting

The degree of reliance of an agent on early data is a crucial factor in determining how much any primacy effect is going to affect the learning process. At the same time, in the interest of sample efficiency, it is vital to leverage initial experiences at their full potential to expedite the training. This trade-off is particularly evident for algorithms with a replay buffer, which can be used to update the agent several times before interacting further with the environment.

To uncover in an explicit way the effect of the primacy bias in RL, we probe excessive reliance to early data to its extreme: could overfitting on a single batch of early data be enough to entirely disrupt an agent’s learning process ?

To investigate this question, we train Soft Actor-Critic ( Haarnoja et al., 2018 ) on the quadruped-run environment from DeepMind Control suite (DMC) ( Tassa et al., 2020 ) . We use default hyperparameters, which imply a single update for both policy and value function per step in the environment. Then, we train an identical version of the algorithm in an experimental condition that we refer to as heavy priming : after collecting 100 100 data points, we update the agent 10 5 10^{5} times using the resulting replay buffer, before resuming standard training. Figure 1 shows that even after collecting and training on almost one million new transitions, the agent with heavy priming is unable to recover from the initial overfitting.

This experiment conveys a simple message: overfitting to early experiences might inexorably damage the rest of the learning process. Even if no practical implementation would use such a large number of training steps on such a limited dataset, Section 5 shows that even a relatively small number of updates per step can cause similar issues. The finding suggests the primacy bias has compounding effects: an overfitted agent gathers worse data that leads to less efficient learning, further damaging the ability to learn and so on.

### 3.2 Experiences of Primed Agents are Sufficient

Once the agent is heavily impacted by the primacy bias, it might struggle to reach satisfying performance. But is the data collected by an overfitted agent unusable for learning? To answer this question, we train a SAC agent with 9 9 updates per step in the MDP: due to the primacy bias, this agent performs poorly. Then, we initialize the same agent from scratch but use the data collected by the previous SAC agent as its initial replay buffer. Figure 2 demonstrates that returns collected by this agent improve rapidly approaching the optimal task performance.

This experiment articulates that the primacy bias is not a failure to collect proper data per se, but rather a failure to learn from it . The data stored in the replay buffer is in principle enough to have better performance but the overfitted agent lacks the ability to distill it into a better policy. In contrast, the randomly initialized neural networks are not affected by the primacy bias and thus capable of fully leveraging the collected experience.

## 4 Have You Tried Resetting It?

The previous section provided controlled experiments demonstrating the primacy bias phenomenon and outlined its consequences. We now present a simple technique that mitigates the effect of this bias on an agent’s training. The solution, which we call resetting in the rest of the paper, is summarized as follows:

The next section analyzes both quantitatively and qualitatively the performance improvements provided by resetting as a mean to address the overfitting to early data.

## 5 Experiments

Method IQM Median Mean SPR + resets 0.478 (0.46, 0.51) 0.512 (0.42, 0.57) 0.911 (0.84, 1.00) SPR 0.380 (0.36, 0.39) 0.433 (0.38, 0.48) 0.578 (0.56, 0.60) DrQ( ϵ \epsilon ) 0.280 (0.27, 0.29) 0.304 (0.28, 0.33) 0.465 (0.46, 0.48) DER 0.183 (0.18, 0.19) 0.191 (0.18, 0.21) 0.351 (0.34, 0.36) CURL 0.113 (0.11, 0.12) 0.102 (0.09, 0.12) 0.261 (0.25, 0.27)

[table]

The goals of our experiments are mostly twofold. First, we demonstrate across different algorithms and domains the performance gains of using resets as a remedy for the primacy bias; then, we analyze the learning dynamics induced by resetting, including its effects on TD learning and interaction with critical design choices of RL algorithms.

### 5.1 Setup

We focus on two domains: discrete control, represented by the 26-task Atari 100k benchmark ( Kaiser et al., 2019 ) , and continuous control, represented by the DeepMind Control Suite ( Tassa et al., 2020 ) . We apply resets to three baseline algorithms: SPR ( Schwarzer et al., 2020 ) for Atari, and SAC ( Haarnoja et al., 2018 ) and DrQ ( Kostrikov et al., 2021 ) for continuous control from dense states and raw pixels respectively. Appendix A provides all environment names and the number of training steps in each domain.

Since both the architectures and the number of training iterations vary across methods, the reset strategy needs slight customization. For SPR, we reset only the final linear layer of the 5-layer Q-network over the course of training spaced 2 × 10 4 2\times 10^{4} steps apart; for SAC, we reset agent’s networks entirely every 2 × 10 5 2\times 10^{5} steps; for DrQ, we reset the last 3 out of 7 layers of the policy and value networks with a periodicity of 2 × 10 5 2\times 10^{5} steps 1 1 1 In fact, the reset periodicity here is 4 × 10 5 R \frac{4\times 10^{5}}{R} , where R R is the per-environment number of action repeats (default is 2), a practice following ( Hafner et al., 2019 ) used in the codebase we build upon, but using R = 2 R=2 for all environments delivers the same results. . SAC and DrQ re-initialize target networks and both Q-networks (due to the use of double Q-learning ( Van Hasselt et al., 2016 ) ); SPR does not have these extra networks. We also reset the corresponding optimizer statistics ( Kingma & Ba, 2015 ) . Appendix B , however, shows the relative robustness to these design choices.

The replay buffer is preserved between resets; SPR and SAC store in the buffer all prior interactions, while DrQ includes only the most recent 100k transitions due to memory limitations of storing image observations. SAC and DrQ sample transitions uniformly from the buffer, while SPR uses prioritized experience replay ( Schaul et al., 2016 ) . The difference between buffer configurations suggests that effects of resets hold for varying buffer sizes and sampling schemes.