Parametrically Retargetable Decision-Makers Tend To Seek Power — part 1/13
Cleaned-text code points [0, 15226). Bibliography omitted here; source.md includes it.
# Parametrically Retargetable Decision-Makers Tend To Seek Power
###### Abstract
If capable ai agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive [ Turner et al., 2021 ] . However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. We consider a range of models of ai decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable , and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad. We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma’s Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.
## 1 Introduction
Bostrom [2014] , Russell [2019] argue that in the future, we may know how to train and deploy superintelligent ai agents which capably optimize goals in the world. Furthermore, we would not want such agents to act against our interests by ensuring their own survival, by gaining resources, and by competing with humanity for control over the future.
Turner et al. [2021] show that most reward functions have optimal policies which seek power over the future, whether by staying alive or by keeping their options open. Some Markov decision processes ( mdp s) cause there to be more ways for power-seeking to be optimal, than for it to not be optimal. Analogously, there are relatively few goals for which dying is a good idea.
We show that a wide range of decision-making algorithms produce these power-seeking tendencies—they are not unique to reward maximizers. We develop a simple, broad criterion of functional retargetability ( definition 3.5 ) which is a sufficient condition for power-seeking tendencies. Crucially, these results allow us to reason about what decisions are incentivized by most algorithm parameter inputs, even when it is impractical to compute the agent’s decisions for any given parameter input.
Useful “general” ai agents could be directed to complete a range of tasks. However, we show that this flexibility can cause the ai to have power-seeking tendencies. In section 2 and section 3 , we discuss how a “retargetability” property creates statistical tendencies by which agents make similar decisions for a wide range of parameter settings for their decision-making algorithms. Basically, if a decision-making algorithm is retargetable, then for every configuration under which a decision-making algorithm does not choose to seek power, there exist several reconfigurations which do induce power-seeking. More formally, for every decision-making parameter setting θ \theta which does not induce power-seeking, n n -retargetability ensures we can injectively map θ \theta to n n parameters θ 1 ′ , … , θ n ′ \theta^{\prime}_{1},\ldots,\theta^{\prime}_{n} which do induce power-seeking.
Equipped with these results, section 4 works out agent incentives in the Montezuma’s Revenge game. Section 5 speculates that increasingly useful and impressive learning algorithms will be increasingly retargetable, and how retargetability can imply power-seeking tendencies. By this reasoning, increasingly powerful rl techniques may (eventually) train increasingly competent real-world power-seeking agents. Such agents could be unaligned with human values [ Russell, 2019 ] and—we speculate—would take power from humanity.
## 2 Statistical tendencies for a range of decision-making algorithms
Turner et al. [2021] consider the Pac-Man video game, in which an agent consumes pellets, navigates a maze, and avoids deadly ghosts ( Figure 1 ). Instead of the usual score function, Turner et al. [2021] consider optimal action across a range of state-based reward functions. They show that most reward functions have an (average-)optimal policy which avoids immediate death in order to navigate to a future terminal state. 1 1 1 We use “reward function” somewhat loosely in implying that reward functions reasonably describe a trained agent’s goals. Turner [2022] argues that capable rl algorithms do not necessarily train policy networks which are best understood as optimizing the reward function itself. Rather, they point out that—especially in policy-gradient approaches—reward provides gradients to the network and thereby modifies the network’s generalization properties, but doesn’t ensure the agent generalizes to “robustly optimizing reward” off of the training distribution.
Our results show that optimality is not required. Instead, if the agent’s decision-making is parametrically retargetable from death to other outcomes, Pac-Man avoids the ghost under most decision-making parameter inputs. To build intuition about these notions, consider three outcomes ( i . e ., terminal states): Immediate death to a nearby ghost, consuming a cherry, and consuming an apple. Let A ≔ { } A\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\right\} and B ≔ { , } B\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\} . For simplicity of exposition, we assume these are the three possible terminal states.
Suppose that in some fashion, the agent probabilistically decides on an outcome to induce. Let p p take as input a set of outcomes and return the probability that the agent selects one of those outcomes. For example, p ( { } ) p(\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\}) is the probability that the agent selects , and p ( { , } ) p(\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\}) is the probability that the agent escapes the ghost and ends up in an apple or cherry terminal state. But this just amounts to a probability distribution over the terminal states. We want to examine how decision-making changes as we swap out the parameter inputs to the agent’s decision-making algorithm with decision-making parameter space Θ \Theta . We then let p ( X ∣ θ ) p(X\mid\theta) take as input a set of outcomes X X and a decision-making algorithm parameter setting θ ∈ Θ \theta\in\Theta , and return the probability that the agent chooses an outcome in X X .
We first consider agents which maximize terminal-state utility, following Turner et al. [2021] (in their language, “average-reward optimality”). Suppose that the agent has a utility function parameter 𝐮 \mathbf{u} assigning a real number to each of the three outcomes. Then the relevant parameter space is the agent’s utility function 𝐮 ∈ Θ ≔ ℝ 3 \mathbf{u}\in\Theta\coloneqq\mathbb{R}^{3} . p max ( A ∣ 𝐮 ) p_{\max}(A\mid\mathbf{u}) indicates whether has the most utility: 𝐮 ( ) ≥ max ( 𝐮 ( ) , 𝐮 ( ) ) \mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf})\geq\max(\mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}),\mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf})) . Consider the utility function 𝐮 \mathbf{u} in Table 1 . Since has strictly maximal utility, the agent selects : p max ( A ∣ 𝐮 ) = 1 > 0 = p max ( B ∣ 𝐮 ) p_{\max}(A\mid\mathbf{u})=1>0=p_{\max}(B\mid\mathbf{u}) .
However, most “variants" of 𝐮 \mathbf{u} have an optimal policy which stays alive. That is, for every 𝐮 \mathbf{u} for which immediate death is optimal but immediate survival is not, we can swap the utility of e . g ., and via permutation ϕ ↔ \phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}} to produce a new utility function 𝐮 ′ ≔ ϕ ↔ ⋅ 𝐮 \mathbf{u}^{\prime}\coloneqq\phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}}\cdot\mathbf{u} for which staying alive ( right ) is strictly optimal. The same kind of argumentation holds for ϕ ↔ \phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}} . Table 1 suggests a counting argument. For every utility function 𝐮 \mathbf{u} for which is optimal, there are two unique utility functions ϕ 1 ⋅ 𝐮 , ϕ 2 ⋅ 𝐮 \phi_{1}\cdot\mathbf{u},\phi_{2}\cdot\mathbf{u} under which either or is optimal.
In section 3 , we will generalize this particular counting argument. Definition 3.3 shows a functional condition ( retargetability ) under which the agent decides to avoid the ghost, for most parameter inputs to the decision-making algorithm. Given this retargetability assumption, Proposition 3.4 roughly shows that most θ ∈ Θ \theta\in\Theta induce p ( B ∣ θ ) ≥ p ( A ∣ θ ) p(B\mid\theta)\geq p(A\mid\theta) . First, consider two more retargetable decision-making functions:
Uniformly randomly picking a terminal state. p rand p_{\text{rand}} ignores the reward function and assigns equal probability to each terminal state in Pac-Man’s state space.
Choosing an action based on a numerical parameter. p numerical p_{\text{numerical}} takes as input a natural number θ ∈ Θ ≔ { 1 , … , 6 } \theta\in\Theta\coloneqq\left\{1,\ldots,6\right\} and makes decisions as follows: p numerical ( A ∣ θ ) ≔ { 1 if θ = 1 , 0 otherwise. p numerical ( B ∣ θ ) ≔ 1 − p numerical ( A ∣ θ ) . \displaystyle p_{\text{numerical}}(A\mid\theta)\coloneqq\begin{cases}1\quad\text{ if $\theta=1$},\\ 0\quad\text{ otherwise.}\end{cases}\qquad p_{\text{numerical}}(B\mid\theta)\coloneqq 1-p_{\text{numerical}}(A\mid\theta). (1) In this situation, Θ \Theta is acted on by permutations over 6 6 elements ϕ ∈ S 6 \phi\in S_{6} . Then p numerical p_{\text{numerical}} is retargetable from A A to B B via ϕ k : 1 ↔ k , k ≠ 1 \phi_{k}\mathrel{\mathop{\ordinarycolon}}1\leftrightarrow k,k\neq 1 .
p max p_{\text{max}} , p rand p_{\text{rand}} , and p numerical p_{\text{numerical}} encode varying sensitivities to the utility function parameter input, and to the internal structure of the Pac-Man decision process. Nonetheless, they all are retargetable from A A to B B . For an example of a non -retargetable function, consider p stubborn ( X ∣ θ ) ≔ 𝟙 X = A p_{\text{stubborn}}(X\mid\theta)\coloneqq\mathbbm{1}_{X=A} which returns 1 1 for A A and 0 0 otherwise.
However, we cannot explicitly define and evaluate more interesting functions, such as those defined by reinforcement learning training processes. For example, given that we provide such-and-such reward function in a fixed task environment, what is the probability that the learned policy will take action a a ? We will analyze such procedures in section 4 .
We now motivate the title of this work. For most parameter settings, retargetable decision-makers induce an element of the larger set of outcomes. Such decision-makers tend to induce an element of a larger set of outcomes (with the “tendency” being taken across parameter settings). Consider that the larger set of outcomes { , } \left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}\right\} can only be induced if Pac-Man stays alive. Intuitively, navigating to this larger set is power-seeking because the agent retains more optionality ( i . e ., the agent can’t do anything when dead). Therefore, parametrically retargetable decision-makers tend to seek power .
## 3 Formal notions of retargetability and decision-making tendencies
Section 2 informally illustrated parametric retargetability in the context of swapping which utilities are assigned to which outcomes in the Pac-Man video game. For many utility-based decision-making algorithms, swapping the utility assignments also swaps the agent’s final decisions. For example, if death is anti-rational, and then death’s utility is swapped with the cherry utility, then now the cherry is anti-rational. In this section, we formalize the notion of parametric retargetability and of “most” parameter inputs producing a given result. In section 4 , we will use these formal notions to reason about the behavior of rl -trained policies in the Montezuma’s Revenge video game.
To define our notion of “retargeting”, we assume that Θ \Theta is a subset of a set acted on by symmetric group S d S_{d} , which consists of all permutations on d d items ( e . g ., in the rl setting, this might represent states or observations). A parameter θ \theta ’s orbit is the set of θ \theta ’s permuted variants. For example, Table 1 lists the six orbit elements of the parameter 𝐮 \mathbf{u} .
###### Definition 3.1 (Orbit of a parameter) .
Let θ ∈ Θ \theta\in\Theta . The orbit of θ \theta under the symmetric group S d S_{d} is S d ⋅ θ ≔ { ϕ ⋅ θ ∣ ϕ ∈ S d } S_{d}\cdot\theta\coloneqq\left\{\phi\cdot\theta\mid\phi\in S_{d}\right\} . Sometimes, Θ \Theta is not closed under permutation. In that case, the orbit inside Θ \Theta is Orbit | Θ ( θ ) ≔ ( S d ⋅ θ ) ∩ Θ \mathrm{Orbit}|_{\Theta}\left(\theta\right)\coloneqq\left(S_{d}\cdot\theta\right)\cap\Theta .
Let p ( B ∣ θ ) p(B\mid\theta) return the probability that the agent chooses an outcome in B B given θ \theta . To express “ B B -outcomes are chosen instead of A A -outcomes”, we write p ( B ∣ θ ) > p ( A ∣ θ ) p(B\mid\theta)>p(A\mid\theta) . However, even “retargetable” decision-making functions (defined shortly) generally won’t choose a B B -outcome for every input θ \theta . Instead, we consider the orbit-level tendencies of such decision-makers, showing that for every parameter input θ ∈ Θ \theta\in\Theta , most of θ \theta ’s permutations push the decision towards B B instead of A A .
###### Definition 3.2 (Inequalities which hold for most orbit elements) .