Optimal Policies Tend to Seek Power — part 1/16

Cleaned-text code points [0, 15750). Bibliography omitted here; source.md includes it.

# Optimal Policies Tend To Seek Power

###### Abstract

Some researchers speculate that intelligent reinforcement learning ( rl ) agents would be incentivized to seek resources and power in pursuit of the objectives we specify for them. Other researchers point out that rl agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes ( mdp s), we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

## 1 Introduction

Omohundro [2008] , Bostrom [2014] , Russell [2019] hypothesize that highly intelligent agents tend to seek power in pursuit of their goals. Such power-seeking agents might gain power over humans. Marvin Minsky imagined that an agent tasked with proving the Riemann hypothesis might rationally turn the planet—along with everyone on it—into computational resources [ Russell and Norvig, 2009 ] . However, another possibility is that such concerns simply arise from the anthropomorphization of AI systems [ LeCun and Zador, 2019 , Various, 2019 , Pinker and Russell, 2020 , Mitchell, 2021 ] .

We clarify this discussion by grounding the claim that highly intelligent agents will tend to seek power. In section 4 , we identify optimal policies as a reasonable formalization of ‘‘highly intelligent agents.’’ 1 1 1 This paper assumes that reward functions reasonably describe a trained agent’s goals. Sometimes this is roughly true ( e . g ., chess with a sparse victory reward signal) and sometimes it is not true. Turner [2022] argues that capable rl algorithms do not necessarily train policy networks which are best understood as optimizing the reward function itself. Rather, they point out that—especially in policy gradient approaches—reward provides gradients to the network and thereby modifies the network’s generalization properties, but doesn’t ensure the agent generalizes to “robustly optimizing reward” off of the training distribution. Optimal policies “tend to” take an action when the action is optimal for most reward functions. We expect future work to translate our theory from optimal policies to learned, real-world policies.

Section 5 defines “power” as the ability to achieve a wide range of goals. For example, “money is power,” and money is instrumentally useful for many goals. Conversely, it’s harder to pursue most goals when physically restrained, and so a physically restrained person has little power. An action “seeks power” if it leads to states where the agent has higher power.

We make no claims about when large-scale AI power-seeking behavior could become plausible. Instead, we consider the theoretical consequences of optimal action in mdp s. Section 6 shows that power-seeking tendencies arise not from anthropomorphism, but from certain graphical symmetries present in many mdp s. These symmetries automatically occur in many environments where the agent can be shut down or destroyed, yielding broad applicability of our main result ( Theorem 6.13 ).

## 2 Related work

An action is instrumental to an objective when it helps achieve that objective. Some actions are instrumental to a range of objectives, making them convergently instrumental . The claim that power-seeking is convergently instrumental is an instance of the instrumental convergence thesis : Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents [ Bostrom, 2012 ] .

For example, in Atari games, avoiding (virtual) death is instrumental for both completing the game and for optimizing curiosity [ Burda et al., 2019 ] . Many AI alignment researchers hypothesize that most advanced AI agents will have concerning instrumental incentives, such as resisting deactivation [ Soares et al., 2015 , Milli et al., 2017 , Hadfield-Menell et al., 2017 , Carey, 2018 ] and acquiring resources [ Benson-Tilsen and Soares, 2016 ] .

We formalize power as the ability to achieve a wide variety of goals. Appendix A demonstrates that our formalization returns intuitive verdicts in situations where information-theoretic empowerment does not [ Salge et al., 2014 ] .

Some of our results relate the formal power of states to the structure of the environment. Foster and Dayan [2002] , Drummond [1998] , Sutton et al. [2011] , Schaul et al. [2015] note that value functions encode important information about the environment, as they capture the agent’s ability to achieve different goals. Turner et al. [2020] speculate that a state’s optimal value correlates strongly across reward functions. In particular, Schaul et al. [2015] learn regularities across value functions, suggesting that some states are valuable for many different reward functions ( i . e ., powerful). Menache et al. [2002] identify and navigate towards convergently instrumental bottleneck states.

We are not the first to study convergence of behavior, form, or function. In economics, turnpike theory studies how certain paths of accumulation tend to be optimal [ McKenzie, 1976 ] . In biology, convergent evolution occurs when similar features ( e . g ., flight) independently evolve in different time periods [ Reece and Campbell, 2011 ] . Lastly, computer vision networks reliably learn e . g ., edge detectors, implying that these features are useful for a range of tasks [ Olah et al., 2020 ] .

## 3 State visit distribution functions quantify the agent’s available options

We clarify the power-seeking discussion by proving what optimal policies usually look like in a given environment. We illustrate our results with a simple case study, before explaining how to reason about a wide range of mdp s. Appendix D.1 lists mdp theory contributions of independent interest, appendix D lists definitions and theorems, and appendix E contains the proofs.

###### Definition 3.1 (Rewardless mdp ) .

⟨ 𝒮 , 𝒜 , T ⟩ \langle\mathcal{S},\mathcal{A},T\rangle is a rewardless mdp with finite state and action spaces 𝒮 \mathcal{S} and 𝒜 \mathcal{A} , and stochastic transition function T : 𝒮 × 𝒜 → Δ ⁡ ( 𝒮 ) T\mathrel{\mathop{\ordinarycolon}}\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) . We treat the discount rate γ \gamma as a variable with domain [ 0 , 1 ] [0,1] .

###### Definition 3.2 (1-cycle states) .

Let 𝐞 s ∈ ℝ | 𝒮 | \mathbf{e}_{s}\in\mathbb{R}^{\left|\mathcal{S}\right|} be the standard basis vector for state s s , such that there is a 1 in the entry for state s s and 0 elsewhere. State s s is a 1-cycle if ∃ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \exists a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} . State s s is a terminal state if ∀ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \forall a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} .

Our theorems apply to stochastic environments, but we present a deterministic case study for clarity. The environment of fig. 1 is small, but its structure is rich. For example, the agent has more “options” at ⋆ \star than at the terminal state ∅ \varnothing . Formally, ⋆ \star has more visit distribution functions than ∅ \varnothing does.

###### Definition 3.3 (State visit distribution [ Sutton and Barto, 1998 ] ) .

Π ≔ 𝒜 𝒮 \Pi\coloneqq\mathcal{A}^{\mathcal{S}} , the set of stationary deterministic policies. The visit distribution induced by following policy π \pi from state s s at discount rate γ ∈ [ 0 , 1 ) \gamma\in[0,1) is 𝐟 π , s ​ ( γ ) ≔ ∑ t = 0 ∞ γ t ​ 𝔼 s t ∼ π | s [ 𝐞 s t ] \mathbf{f}^{\pi,s}(\gamma)\coloneqq\sum_{t=0}^{\infty}\gamma^{t}\opE_{s_{t}\sim\pi\mid s}\left[\mathbf{e}_{s_{t}}\right] . 𝐟 π , s \mathbf{f}^{\pi,s} is a visit distribution function ; ℱ ⁡ ( s ) ≔ { 𝐟 π , s ∣ π ∈ Π } \Fop(s)\coloneqq\{\mathbf{f}^{\pi,s}\mid\pi\in\Pi\} .

In fig. 1 , starting from ℓ ↙ \ell_{\swarrow} , the agent can stay at ℓ ↙ \ell_{\swarrow} or alternate between ℓ ↙ \ell_{\swarrow} and ℓ ↖ \ell_{\nwarrow} , and so ℱ ⁡ ( ℓ ↙ ) = { 1 1 − γ ​ 𝐞 ℓ ↙ , 1 1 − γ 2 ​ ( 𝐞 ℓ ↙ + γ ​ 𝐞 ℓ ↖ ) } \Fop(\ell_{\swarrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{\ell_{\swarrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{\ell_{\swarrow}}+\gamma\mathbf{e}_{\ell_{\nwarrow}})\} . In contrast, at ∅ \varnothing , all policies π \pi map to visit distribution function 1 1 − γ ​ 𝐞 ∅ \frac{1}{1-\gamma}\mathbf{e}_{\varnothing} .

Before moving on, we introduce two important concepts used in our main results. First, we sometimes restrict our attention to visit distributions which take certain actions ( fig. 2 ).

###### Definition 3.4 ( ℱ \Fop single-state restriction) .

Considering only visit distribution functions induced by policies taking action a a at state s ′ s^{\prime} , ℱ ( s ∣ π ( s ′ ) = a ) ≔ { 𝐟 ∈ ℱ ( s ) ∣ ∃ π ∈ Π : π ( s ′ ) = a , 𝐟 π , s = 𝐟 } \Fop(s\mid\pi(s^{\prime})=a)\coloneqq\left\{\mathbf{f}\in\Fop(s)\mid\exists\pi\in\Pi\mathrel{\mathop{\ordinarycolon}}\pi(s^{\prime})=a,\mathbf{f}^{\pi,s}=\mathbf{f}\right\} .

Second, some 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) are “unimportant.” Consider an agent optimizing reward function 𝐞 r ↘ \mathbf{e}_{r_{\searrow}} (1 reward when at r ↘ r_{\searrow} , 0 otherwise) at e . g ., γ = 1 2 \gamma=\frac{1}{2} . Its optimal policies navigate to r ↘ r_{\searrow} and stay there. Similarly, for reward function 𝐞 r ↗ \mathbf{e}_{r_{\nearrow}} , optimal policies navigate to r ↗ r_{\nearrow} and stay there. However, for no reward function is it uniquely optimal to alternate between r ↗ r_{\nearrow} and r ↘ r_{\searrow} . Only dominated visit distribution functions alternate between r ↗ r_{\nearrow} and r ↘ r_{\searrow} ( definition 3.6 ).

###### Definition 3.5 (Value function) .

Let π ∈ Π \pi\in\Pi . For any reward function R ∈ ℝ 𝒮 R\in\mathbb{R}^{\mathcal{S}} over the state space, the on-policy value at state s s and discount rate γ ∈ [ 0 , 1 ) \gamma\in[0,1) is V R π ​ ( s , γ ) ≔ 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐫 V^{\pi}_{R}\left(s,\gamma\right)\coloneqq\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r} , where 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} is R R expressed as a column vector (one entry per state). The optimal value is V R ∗ ​ ( s , γ ) ≔ max π ∈ Π ⁡ V R π ​ ( s , γ ) V^{*}_{R}\left(s,\gamma\right)\coloneqq\max_{\pi\in\Pi}V^{\pi}_{R}\left(s,\gamma\right) .

###### Definition 3.6 (Non-domination) .

ℱ nd ( s ) ≔ { 𝐟 π ∈ ℱ ( s ) ∣ ∃ 𝐫 ∈ ℝ | 𝒮 | , γ ∈ ( 0 , 1 ) : 𝐟 π ( γ ) ⊤ 𝐫 > max 𝐟 π ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 π } 𝐟 π ′ ( γ ) ⊤ 𝐫 } . \Fndop(s)\coloneqq\{\mathbf{f}^{\pi}\in\Fop(s)\mid\exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|},\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}^{\pi}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\pi^{\prime}}\in\Fop(s)\setminus\left\{\mathbf{f}^{\pi}\right\}}\mathbf{f}^{\pi^{\prime}}(\gamma)^{\top}\mathbf{r}\}. (1) For any reward function R R and discount rate γ \gamma , 𝐟 π ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi}\in\Fop(s) is (weakly) dominated by 𝐟 π ′ ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi^{\prime}}\in\Fop(s) if V R π ​ ( s , γ ) ≤ V R π ′ ​ ( s , γ ) V^{\pi}_{R}(s,\gamma)\leq V^{\pi^{\prime}}_{R}(s,\gamma) . 𝐟 π ∈ ℱ nd ⁡ ( s ) \mathbf{f}^{\pi}\in\Fndop(s) is non-dominated if there exist R R and γ \gamma at which 𝐟 π \mathbf{f}^{\pi} is not dominated by any other 𝐟 π ′ \mathbf{f}^{\pi^{\prime}} .

## 4 Some actions have a greater probability of being optimal

We claim that optimal policies “tend” to take certain actions in certain situations. We first consider the probability that certain actions are optimal.

Reconsider the reward function 𝐞 r ↘ \mathbf{e}_{r_{\searrow}} , optimized at γ = 1 2 \gamma=\frac{1}{2} . Starting from ⋆ \star , the optimal trajectory goes right to r ⊳ r_{\triangleright} to r ↘ r_{\searrow} , where the agent remains. The right action is optimal at ⋆ \star under these incentives. Optimal policy sets capture the behavior incentivized by a reward function and a discount rate.

###### Definition 4.1 (Optimal policy set function) .

Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) is the optimal policy set for reward function R R at γ ∈ ( 0 , 1 ) \gamma\in(0,1) . All R R have at least one optimal policy π ∈ Π \pi\in\Pi [ Puterman, 2014 ] . Π ∗ ​ ( R , 0 ) ≔ lim γ → 0 Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,0\right)\coloneqq\lim_{\gamma\to 0}\Pi^{*}\left(R,\gamma\right) and Π ∗ ​ ( R , 1 ) ≔ lim γ → 1 Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,1\right)\coloneqq\lim_{\gamma\to 1}\Pi^{*}\left(R,\gamma\right) exist by Lemma E.33 (taking the limits with respect to the discrete topology over policy sets).

We may be unsure which reward function an agent will optimize. We may expect to deploy a system in a known environment, without knowing the exact form of e . g ., the reward shaping [ Ng et al., 1999 ] or intrinsic motivation [ Pathak et al., 2017 ] . Alternatively, one might attempt to reason about future rl agents, whose details are unknown. Our power-seeking results do not hinge on such uncertainty, as they also apply to degenerate distributions ( i . e ., we know what reward function will be optimized).

###### Definition 4.2 (Reward function distributions) .

Different results make different distributional assumptions. Results with 𝒟 any ∈ 𝔇 any ≔ Δ ⁡ ( ℝ | 𝒮 | ) \mathcal{D}_{\text{any}}\in\mathfrak{D}_{\text{any}}\coloneqq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) hold for any probability distribution over ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} . 𝔇 bound \mathfrak{D}_{\text{bound}} is the set of bounded-support probability distributions 𝒟 bound \mathcal{D}_{\text{bound}} . For any distribution X X over ℝ \mathbb{R} , 𝒟 X ​ - iid ≔ X | 𝒮 | \mathcal{D}_{X\text{-}\textsc{iid}}\coloneqq X^{\left|\mathcal{S}\right|} . For example, when X u ≔ unif ​ ( 0 , 1 ) X_{u}\coloneqq\text{unif}(0,1) , 𝒟 X u ​ - iid \mathcal{D}_{X_{u}\text{-}\textsc{iid}} is the maximum-entropy distribution. 𝒟 s \mathcal{D}_{s} is the degenerate distribution on the state indicator reward function 𝐞 s \mathbf{e}_{s} , which assigns 1 reward to s s and 0 elsewhere.

With 𝒟 any \mathcal{D}_{\text{any}} representing our prior beliefs about the agent’s reward function, what behavior should we expect from its optimal policies? Perhaps we want to reason about the probability that it’s optimal to go from ⋆ \star to ∅ \varnothing , or to go to r ⊳ r_{\triangleright} and then stay at r ↗ r_{\nearrow} . In this case, we quantify the optimality probability of F ≔ { 𝐞 ⋆ + γ 1 − γ ​ 𝐞 ∅ , 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } F\coloneqq\{\mathbf{e}_{\star}+\frac{\gamma}{1-\gamma}\mathbf{e}_{\varnothing},\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} .

###### Definition 4.3 (Visit distribution optimality probability) .

Let F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) , γ ∈ [ 0 , 1 ] \gamma\in[0,1] . ℙ 𝒟 any ( F , γ ) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐟 π ∈ F : π ∈ Π ∗ ​ ( R , γ ) ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)\coloneqq\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{f}^{\pi}\in F\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{*}\left(R,\gamma\right)\right) .

Alternatively, perhaps we’re interested in the probability that right is optimal at ⋆ \star .

###### Definition 4.4 (Action optimality probability) .