Optimal Policies Tend to Seek Power — part 4/16

Cleaned-text code points [47054, 62841). Bibliography omitted here; source.md includes it.

Theorem 6.13 applies to many structured rl environments, which tend to be spatially regular and to factorize along several dimensions. Therefore, different sets of rsd s will be similar, requiring only modification of factor values. For example, if an embodied agent can deterministically navigate a set of three similar rooms (spatial regularity), then the agent’s position factors via {room number} × \times {position in room}. Therefore, the rsd s can be divided into three similar subsets, depending on the agent’s room number.

Corollary 6.14 dictates where average-optimal agents tend to end up, but not how they get there. Corollary 6.14 says that such agents tend not to stay in any given 1-cycle. It does not say that such agents will avoid entering such states. For example, in an embodied navigation task, a robot may enter a 1-cycle by idling in the center of a room. Corollary 6.14 implies that average-optimal robots tend not to idle in that particular spot, but not that they tend to avoid that spot entirely.

However, average-optimal robots do tend to avoid getting shut down. The agent’s task mdp often represents agent shutdown with terminal states. A terminal state is, by definition 3.2 , unable to access other 1-cycles. Since Corollary 6.14 shows that average-optimal agents tend to end up in other 1-cycles, average-optimal policies must tend to completely avoid the terminal state. Therefore, we conclude that in many such situations, average-optimal policies tend to avoid shutdown. Intuitively, survival is power-seeking relative to dying, and so shutdown-avoidance is power-seeking behavior.

In fig. 8 , the player dies by going left , but can reach thousands of rsd s by heading in other directions. Even if some average-optimal policies go left in order to reach fig. 8 ’s “game over” terminal state, all other rsd s cannot be reached by going left . There are many 1-cycles besides the immediate terminal state. Therefore, Corollary 6.14 proves that average-optimal policies tend to not go left in this situation. Average-optimal policies tend to avoid immediately dying in Pac-Man, even though most reward functions do not resemble Pac-Man’s original score function.

## 7 Discussion

Reconsider the case of a hypothetical intelligent real-world agent which optimizes average reward for some objective. Suppose the designers initially have control over the agent. If the agent began to misbehave, perhaps they could just deactivate it. Unfortunately, our results suggest that this strategy might not work. Average-optimal agents would generally stop us from deactivating them, if physically possible. Extrapolating from our results, we conjecture that when γ ≈ 1 \gamma\approx 1 , optimal policies tend to seek power by accumulating resources—to the detriment of any other agents in the environment.

##### Future work.

Real-world training procedures often do not satisfy rl convergence theorems. Thus, learned policies are rarely optimal. We expect this point to seriously constrain the applicability of this theory. Emphatically, optimal policies are often qualitatively divorced from the actual policies learned by reinforcement learning. For example, the mathematics of policy gradient algorithms is not to update policies so as to maximize reward. Instead, the rewards provide gradients to the parameterization of the policy [ Turner, 2022 ] . On that view, reward functions are simply sources of gradient updates which designers use in order to control generalization behavior.

Most real-world tasks are partially observable. Although our results only apply to optimal policies in finite mdp s, we expect the key conclusions to generalize. Furthermore, irregular stochasticity in environmental dynamics can make it hard to satisfy Theorem 6.13 ’s similarity requirement. We look forward to future work which addresses partially observable environments, suboptimal policies, or “almost similar” rsd sets.

Past work shows that it would be bad for an agent to disempower humans in its environment. In a two-player agent / human game, minimizing the human’s information-theoretic empowerment [ Salge et al., 2014 ] produces adversarial agent behavior [ Guckelsberger et al., 2018 ] . In contrast, maximizing human empowerment produces helpful agent behavior [ Salge and Polani, 2017 , Guckelsberger et al., 2016 , Du et al., 2020 ] . We do not yet formally understand if, when, or why Power -seeking policies tend to disempower other agents in the environment.

More complex environments probably have more pronounced power-seeking incentives. Intuitively, there are often many ways for power-seeking to be optimal, and relatively few ways for power-seeking not to be optimal. For example, suppose that in some environment, Theorem 6.13 holds for one million involutions ϕ \phi . Does this guarantee more pronounced incentives than if Theorem 6.13 only held for one involution?

We proved sufficient conditions for when reward functions tend to have optimal policies which seek power. In the absence of prior information, one should expect that an arbitrary reward function has optimal policies which exhibit power-seeking behavior under these conditions. However, we have prior information: AI designers usually try to specify a good reward function. Even so, it may be hard to specify orbit elements which do not—at optimum—incentivize bad power-seeking.

##### Societal impact.

We believe that this paper builds toward a rigorous understanding of the risks presented by AI power-seeking incentives. Understanding these risks is the first step in addressing them. However, basic theoretical work can have many consequences. For example, this theory could somehow help future researchers build power-seeking agents which disempower humans. We believe that the benefit of understanding outweighs the potential societal harm.

##### Conclusion.

We developed the first formal theory of the statistical tendencies of optimal policies in reinforcement learning. In the context of mdp s, we proved sufficient conditions under which optimal policies tend to seek power, both formally (by taking Power -seeking actions) and intuitively (by taking actions which keep the agent’s options open). Many real-world environments have symmetries which produce power-seeking incentives. In particular, optimal policies tend to seek power when the agent can be shut down or destroyed. Seeking control over the environment will often involve resisting shutdown, and perhaps monopolizing resources.

We caution that many real-world tasks are partially observable and that learned policies are rarely optimal. Our results do not mathematically prove that hypothetical superintelligent AI agents will seek power. However, we hope that this work will foster thoughtful, serious, and rigorous discussion of this possibility.

## Acknowledgments

Alexander Turner was supported by the Berkeley Existential Risk Initiative and the Long-Term Future Fund. Alexander Turner, Rohin Shah, and Andrew Critch were supported by the Center for Human-Compatible AI. Prasad Tadepalli was supported by the National Science Foundation.

Yousif Almulla, John E. Ball, Daniel Blank, Steve Byrnes, Ryan Carey, Michael Dennis, Scott Emmons, Alan Fern, Daniel Filan, Ben Garfinkel, Adam Gleave, Edouard Harris, Evan Hubinger, DNL Kok, Vanessa Kosoy, Victoria Krakovna, Cassidy Laidlaw, Joel Lehman, David Lindner, Dylan Hadfield-Menell, Richard Möhn, Alexandra Nolan, Matt Olson, Neale Ratzlaff, Adam Shimi, Sam Toyer, Joshua Turner, Cody Wild, Davide Zagami, and our anonymous reviewers provided valuable feedback.

## Appendix A Comparing Power with information-theoretic empowerment

Salge et al. [2014] define information-theoretic empowerment as the maximum possible mutual information between the agent’s actions and the state observations n n steps in the future, written 𝔈 n ​ ( s ) \mathfrak{E}_{n}(s) . This notion requires an arbitrary choice of horizon, failing to account for the agent’s discount rate γ \gamma . “In a discrete deterministic world empowerment reduces to the logarithm of the number of sensor states reachable with the available actions” [ Salge et al., 2014 ] . Figure 9 demonstrates how empowerment can return counterintuitive verdicts with respect to the agent’s control over the future.

Power returns intuitive answers in these situations. lim γ → 1 Power 𝒟 bound ​ ( s 1 , γ ) \lim_{\gamma\to 1}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\gamma\right) converges by Lemma 5.3 . Consider the obvious involution ϕ \phi which takes each state in fig. 9(b) to its counterpart in fig. 9(c) . Since ϕ ⋅ ℱ nd ⁡ ( s 3 ) ⊊ ℱ nd ⁡ ( s 4 ) = ℱ ⁡ ( s 4 ) \phi\cdot\Fndop({\color[rgb]{0.25,0.45,0.75}s_{3}})\subsetneq\Fndop({\color[rgb]{0.25,0.45,0.75}s_{4}})=\Fop({\color[rgb]{0.25,0.45,0.75}s_{4}}) , Proposition 6.6 proves that ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s 3 , γ ) ≤ most : 𝔇 bound Power 𝒟 bound ( s 4 , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{3}},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{4}},\gamma\right) , with the proof of Proposition 6.6 showing strict inequality under all 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} when γ ∈ ( 0 , 1 ) \gamma\in(0,1) .

Empowerment can be adjusted to account for these cases, perhaps by considering the channel capacity between the agent’s actions and the state trajectories induced by stationary policies. However, since Power is formulated in terms of optimal value, we believe that Power is better suited for mdp s than information-theoretic empowerment is.

## Appendix B Seeking Power can be a detour

###### Remark .

The results of appendix E do not depend on this section’s results.

One might suspect that optimal policies tautologically tend to seek Power . This intuition is wrong.

###### Proposition B.1 (Greater Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} does not imply greater ℙ 𝒟 bound \Prb_{\mathcal{D}_{\text{bound}}} ) .

Action a a seeking more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than a ′ a^{\prime} at state s s and γ \gamma does not imply that ℙ 𝒟 bound ( s , a , γ ) ≥ ℙ 𝒟 bound ( s , a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{bound}}}\left(s,a,\gamma\right)\geq\Prb\nolimits_{\mathcal{D}_{\text{bound}}}\left(s,a^{\prime},\gamma\right) .

###### Proof.

Consider the environment of fig. 10 . Let X u ≔ unif ​ ( 0 , 1 ) X_{u}\coloneqq\text{unif}(0,1) , and consider 𝒟 X u ​ - iid \mathcal{D}_{X_{u}\text{-}\textsc{iid}} , which has bounded support. Direct computation 7 7 7 In small deterministic mdp s, the Power and optimality probability of the maximum-entropy reward function distribution can be computed using https://github.com/loganriggs/Optimal-Policies-Tend-To-Seek-Power . of the Power expectation ( definition 5.2 ) yields Power 𝒟 X u ​ - iid ​ ( s 2 , 1 ) = 3 4 > 2 3 = Power 𝒟 X u ​ - iid ​ ( s 3 , 1 ) \text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left(s_{2},1\right)=\frac{3}{4}>\frac{2}{3}=\text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left(s_{3},1\right) . Therefore, N seeks more Power 𝒟 X u ​ - iid \text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}} than NE at state s 1 {\color[rgb]{0.25,0.45,0.75}s_{1}} and γ = 1 \gamma=1 .

However, ℙ 𝒟 X u ​ - iid ( s 1 , N , 1 ) = 1 3 < 2 3 = ℙ 𝒟 X u ​ - iid ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)=\frac{1}{3}<\frac{2}{3}=\Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) . ∎

###### Lemma B.2 (Fraction of orbits which agree on weak optimality) .

Let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) , and suppose f 1 , f 2 : Δ ⁡ ( ℝ | 𝒮 | ) → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Delta(\mathbb{R}^{\left|\mathcal{S}\right|})\to\mathbb{R} are such that f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) . Then for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) } | | S | 𝒮 | ⋅ 𝒟 | ≥ 1 2 \frac{\left|\left\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime})\right\}\right|}{\left|S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\right|}\geq\dfrac{1}{2} .

###### Proof.

All 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} such that f 1 ​ ( 𝒟 ′ ) = f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})=f_{2}(\mathcal{D}^{\prime}) satisfy f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) .

Otherwise, consider the 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} such that f 1 ​ ( 𝒟 ′ ) ≠ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\neq f_{2}(\mathcal{D}^{\prime}) . By the definition of ≥ most \geq_{\text{{most}}} ( definition 6.5 ), at least 1 2 \frac{1}{2} of these 𝒟 ′ \mathcal{D}^{\prime} satisfy f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime}) , in which case f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) . Then the desired inequality follows. ∎

###### Lemma B.3 ( ≥ most \geq_{\text{most}} and trivial orbits) .

Let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) and suppose f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) . For all reward function distributions 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} with one-element orbits, f 1 ​ ( 𝒟 ) ≥ f 2 ​ ( 𝒟 ) f_{1}(\mathcal{D})\geq f_{2}(\mathcal{D}) . In particular, 𝒟 \mathcal{D} has a one-element orbit when it distributes reward identically and independently ( iid ) across states.

###### Proof.

By Lemma B.2 , at least half of the elements 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} satisfy f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) . But | S | 𝒮 | ⋅ 𝒟 | = 1 \left|S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\right|=1 , and so f 1 ​ ( 𝒟 ) ≥ f 2 ​ ( 𝒟 ) f_{1}(\mathcal{D})\geq f_{2}(\mathcal{D}) must hold.

If 𝒟 \mathcal{D} is iid , it has a one-element orbit due to the assumed identical distribution of reward. ∎

###### Proposition B.4 (Actions which tend to seek Power do not necessarily tend to be optimal) .

Action a a tending to seek more Power than a ′ a^{\prime} at state s s and γ \gamma does not imply that ℙ 𝒟 any ( s , a , γ ) ≥ most : 𝔇 any ℙ 𝒟 any ( s , a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right) .

###### Proof.

Consider the environment of fig. 10 . Since RSD nd ​ ( s 3 ) ⊊ RSD ​ ( s 2 ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s_{3}\right)\subsetneq\text{{RSD}}\left(s_{2}\right) , Proposition 6.12 shows that Power 𝒟 bound ( s 2 , 1 ) ≥ most : 𝔇 bound Power 𝒟 bound ( s 3 , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},1\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{3},1\right) via s ′ ≔ s 3 , s ≔ s 2 , ϕ s^{\prime}\coloneqq s_{3},s\coloneqq s_{2},\phi the identity permutation (which is an involution). Therefore, N tends to seek more Power than NE at state s 1 {\color[rgb]{0.25,0.45,0.75}s_{1}} and γ = 1 \gamma=1 .