Parametrically Retargetable Decision-Makers Tend To Seek Power — part 4/13

Cleaned-text code points [45943, 61814). Bibliography omitted here; source.md includes it.

Consider a sequence of decision-making functions p t : { A , B } × Θ → ℝ p_{t}\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} which converges pointwise to some p p such that p ( B ∣ θ ) ≥ most : Θ n p ( A ∣ θ ) p(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p(A\mid\theta) . We expect that under rather mild conditions, ∃ T : ∀ t ≥ T : p t ( B ∣ θ ) ≥ most : Θ n p t ( A ∣ θ ) \exists T\mathrel{\mathop{\ordinarycolon}}\forall t\geq T\mathrel{\mathop{\ordinarycolon}}p_{t}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{t}(A\mid\theta) . As a corollary, for any decision-making procedure p t p_{t} which runs for t t time steps and satisfies lim t → ∞ p t = p \lim_{t\to\infty}p_{t}=p , the function p t p_{t} will have decision-making incentives after finite time. For example, value iteration ( vi ) eventually finds an optimal policy [ Puterman, 2014 ] , and optimal policies tend to seek power [ Turner et al., 2021 ] . Therefore, this conjecture would imply that if vi is run for some long but finite time, it tends to produce power-seeking policies. More interestingly, the result would allow us to reason about the effect of e . g ., randomly initializing parameters (in vi , the tabular value function at t = 0 t=0 ). The effect of random initialization washes out in the limit of infinite time, so we would still conclude the presence of finite-time power-seeking incentives.

Our results do not prove that we will build unaligned ai agents which seek power over the world. Here are a few situations in which our results are not concerning or not applicable. 1. The ai is aligned with human interests. For example, we want a robotic cartographer to prevent itself from being deactivated. However, the ai alignment problem is not yet understood for highly intelligent agents [ Russell, 2019 ] .

2. The ai ’s decision-making is not retargetable ( definition 3.5 ).

3. The ai ’s decision-making is retargetable over e . g ., actions ( section 4.1 ) instead of over final outcomes ( section 4.2 ). This retargetability seems less concerning, but also less practically useful.

### 6.3 Conclusion

We introduced the concept of retargetability and showed that retargetable decision-makers often make similar instrumental choices. We applied these results in the Montezuma’s Revenge ( mr ) video game, showing how increasingly advanced reinforcement learning algorithms correspond to increasingly retargetable agent decision-making. Increasingly retargetable agents make increasingly similar instrumental decisions— e . g ., leaving the initial room in mr , or staying alive in Pac-Man. In particular, these decisions will often correspond to gaining power and keeping options open [ Turner et al., 2021 ] . Our theory suggests that when rl training processes become sufficiently advanced, the trained agents will tend to seek power over the world. This theory suggests a safety risk. We hope for future work on this theory so that the field of ai can understand the relevant safety risks before the field trains power-seeking agents.

## Broader impacts

Our theory of orbit-level tendencies constitutes basic mathematical research into the decision-making tendencies of certain kinds of agents. We hope that this theory will prevent negative impacts from unaligned power-seeking ai . We do not anticipate that our work will have negative impact.

## Acknowledgements

We thank Irene Tematelewo, Colin Shea-Blymyer, and our anonymous reviewers for feedback. We thank Justis Mills for proofreading.

## Appendix A Retargetability over outcome lotteries

Suppose we are interested in d d outcomes. Each outcome could be the visitation of an mdp state, or a trajectory, or the receipt of a physical item. In the Pac-Man example of section 2 , d = 3 d=3 states. The agent can induce each outcome with probability 1 1 , so let 𝐞 o ∈ ℝ 3 \mathbf{e}_{o}\in\mathbb{R}^{3} be the standard basis vector with probability 1 1 on outcome o o and 0 0 elsewhere. Then the agent chooses among outcome lotteries C ≔ { 𝐞 , 𝐞 , 𝐞 } C\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} , which we partition into A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} and B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} .

###### Definition A.1 (Outcome lotteries) .

A unit vector 𝐱 ∈ ℝ d \mathbf{x}\in\mathbb{R}^{d} with non-negative entries is an outcome lottery . 7 7 7 Our results on outcome lotteries hold for generic 𝐱 ′ ∈ ℝ d \mathbf{x}^{\prime}\in\mathbb{R}^{d} , but we find it conceptually helpful to consider the non-negative unit vector case.

Many decisions are made consequentially: based on the consequences of the decision, on what outcomes are brought about by an act. For example, in a deterministic Atari game, a policy induces a trajectory. A reward function and discount rate tuple ( R , γ ) (R,\gamma) assigns a return to each state trajectory τ = s 0 , s 1 , … \tau=s_{0},s_{1},\ldots : G ⁡ ( τ ) = ∑ i = 0 ∞ γ i ​ R ​ ( s i ) G(\tau)=\sum_{i=0}^{\infty}\gamma^{i}R(s_{i}) . The relevant outcome lottery is the discounted visit distribution over future states in an Atari game, and policies are optimal or not depending on which outcome lottery is induced by the policy.

###### Definition A.2 (Optimality indicator function) .

Let X , C ⊊ ℝ d X,C\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . IsOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{IsOptimal}\left(X\mid C,\mathbf{u}\right) returns 1 1 if max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ≥ max 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}\geq\max_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u} , and 0 0 otherwise.

We consider decision-making procedures which take in a targeting parameter 𝐮 \mathbf{u} . For example, the column headers of Table 2(a) show the 6 permutations of the utility function u ⁡ ( ) ≔ 10 , u ⁡ ( ) ≔ 5 , u ⁡ ( ) ≔ 0 u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf})\coloneqq 10,u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf})\coloneqq 5,u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf})\coloneqq 0 , representable as a vector 𝐮 ∈ ℝ 3 \mathbf{u}\in\mathbb{R}^{3} .

𝐮 \mathbf{u} can be permuted as follows. The outcome permutation ϕ ∈ S d \phi\in S_{d} inducing an d × d d\times d permutation matrix 𝐏 ϕ \mathbf{P}_{\phi} in row representation: ( 𝐏 ϕ ) i ​ j = 1 (\mathbf{P}_{\phi})_{ij}=1 if i = ϕ ⁡ ( j ) i=\phi(j) and 0 0 otherwise. Table 2(a) shows that for a given utility function, 2 3 \frac{2}{3} of its orbit agrees that B B is strictly optimal over A A .

Orbit-level incentives occur when an inequality holds for most permuted parameter choices 𝐮 ′ \mathbf{u}^{\prime} . Table 2(a) demonstrates an application of Turner et al. [2021] ’s results: Optimal decision-making induces orbit-level incentives for choosing Pac-Man outcomes in B B over outcomes in A A .

Furthermore, Turner et al. [2021] conjectured that “larger” B B will imply stronger orbit-level tendencies: If going right leads to 500 times as many options as going left, then right is better than left for at least 500 times as many reward functions for which the opposite is true. We prove this conjecture with Theorem D.11 in appendix D .

However, orbit-level incentives do not require optimality. One clue is that the same results hold for anti-optimal agents, since anti-optimality/utility minimization of 𝐮 \mathbf{u} is equivalent to maximizing − 𝐮 -\mathbf{u} . Table 2(b) illustrates that the same orbit guarantees hold in this case.

###### Definition A.3 (Anti-optimality indicator function) .

Let X , C ⊊ ℝ d X,C\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . AntiOpt ⁡ ( X ∣ C , 𝐮 ) \mathrm{AntiOpt}\left(X\mid C,\mathbf{u}\right) returns 1 1 if min 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ≤ min 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 \min_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}\leq\min_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u} , and 0 0 otherwise.

Stepping beyond expected utility maximization/minimization, Boltzmann-rational decision-making selects outcome lotteries proportional to the exponential of their expected utility.

###### Definition A.4 (Boltzmann rationality [ Baker et al., 2007 ] ) .

For X ⊆ C X\subseteq C and temperature T > 0 T>0 , let Boltzmann T ​ ( X ∣ C , 𝐮 ) ≔ ∑ 𝐱 ∈ X e T − 1 ​ 𝐱 ⊤ ​ 𝐮 ∑ 𝐜 ∈ C e T − 1 ​ 𝐜 ⊤ ​ 𝐮 \mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right)\coloneqq\frac{\sum_{\mathbf{x}\in X}e^{T^{-1}\mathbf{x}^{\top}\mathbf{u}}}{\sum_{\mathbf{c}\in C}e^{T^{-1}\mathbf{c}^{\top}\mathbf{u}}} be the probability that some element of X X is Boltzmann-rational.

Lastly, orbit-level tendencies occur even under decision-making procedures which partially ignore expected utility and which “don’t optimize too hard.” Satisficing agents randomly choose an outcome lottery with expected utility exceeding some threshold. Table 2(d) demonstrates that satisficing induces orbit-level tendencies.

###### Definition A.5 (Satisficing) .

Let t ∈ ℝ t\in\mathbb{R} , let X ⊆ C ⊊ ℝ d X\subseteq C\subsetneq\mathbb{R}^{d} be finite. Satisfice t ​ ( X , C ∣ 𝐮 ) ≔ | X ∩ { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 ≥ t } | | { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 ≥ t } | \mathrm{Satisfice}_{t}\left(X,C\mid\mathbf{u}\right)\coloneqq\frac{\left|X\cap\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}\geq t\right\}\right|}{\left|\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}\geq t\right\}\right|} is the fraction of X X whose value exceeds threshold t t . Satisfice t ​ ( X , C ∣ 𝐮 ) \mathrm{Satisfice}_{t}\left(X,C\mid\mathbf{u}\right) evaluates to 0 0 the denominator equals 0 0 .

For each table, two-thirds of the utility permutations (columns) assign strictly larger values (shaded dark gray) to an element of B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} than to an element of A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} . For optimal, anti-optimal, Boltzmann-rational, and satisficing agents, Proposition A.11 proves that these tendencies hold for all targeting parameter orbits.

### A.1 A range of decision-making functions are retargetable

In mdp s, Turner et al. [2021] consider state visitation distributions which record the total discounted time steps spent in each environment state, given that the agent follows some policy π \pi from an initial state s s . These visitation distributions are one kind of outcome lottery, with d = | 𝒮 | d=\left|\mathcal{S}\right| the number of mdp states.

In general, we suppose the agent has an objective function 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} which maps outcomes to real numbers. In Turner et al. [2021] , 𝐮 \mathbf{u} was a state-based reward function (and so the outcomes were states ). However, we need not restrict ourselves to the mdp setting.

To state our key results, we define several technical concepts which we informally used when reasoning about A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} and B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} .

###### Definition A.6 (Similarity of vector sets) .

For ϕ ∈ S d \phi\in S_{d} and X ⊆ ℝ d X\subseteq\mathbb{R}^{d} , ϕ ⋅ X ≔ { 𝐏 ϕ ​ 𝐱 ∣ 𝐱 ∈ X } \phi\cdot X\coloneqq\left\{\mathbf{P}_{\phi}\mathbf{x}\mid\mathbf{x}\in X\right\} . X ′ ⊆ ℝ | 𝒮 | X^{\prime}\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} is similar to X X when ∃ ϕ : ϕ ⋅ X ′ = X \exists\phi\mathrel{\mathop{\ordinarycolon}}\phi\cdot X^{\prime}=X . ϕ \phi is an involution if ϕ = ϕ − 1 \phi=\phi^{-1} (it either transposes states, or fixes them). X X contains a copy of X ′ X^{\prime} when X ′ X^{\prime} is similar to a subset of X X via an involution ϕ \phi .

###### Definition A.7 (Containment of set copies) .

Let n n be a positive integer, and let A , B ⊆ ℝ d A,B\subseteq\mathbb{R}^{d} . We say that B B contains n n copies of A A when there exist involutions ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} such that ∀ i : ϕ i ⋅ A ≕ B i ⊆ B \forall i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot A\eqqcolon B_{i}\subseteq B and ∀ j ≠ i : ϕ i ⋅ B j = B j \forall j\neq i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot B_{j}=B_{j} . 8 8 8 Technically, definition A.7 implies that A A contains n n copies of A A holds for all n n , via n n applications of the identity permutation. For our purposes, this provides greater generality, as all of the relevant results still hold. Enforcing pairwise disjointness of the B i B_{i} would handle these issues, but would narrow our results to not apply e . g ., when the B i B_{i} share a constant vector.

B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} contains two copies of A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} via ϕ 1 ≔ ↔ \phi_{1}\coloneqq\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf} and ϕ 2 ≔ ↔ \phi_{2}\coloneqq\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf} .

###### Definition A.8 (Targeting parameter distribution assumptions) .

Results with 𝒟 any \mathcal{D}_{\text{any}} hold for any probability distribution over ℝ d \mathbb{R}^{d} . Let 𝔇 any ≔ Δ ⁡ ( ℝ d ) \mathfrak{D}_{\text{any}}\coloneqq\Delta(\mathbb{R}^{d}) . For a function f : ℝ d ↦ ℝ f\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\mapsto\mathbb{R} , we write f ⁡ ( 𝒟 any ) f(\mathcal{D}_{\text{any}}) as shorthand for 𝔼 𝐮 ∼ 𝒟 any [ f ⁡ ( 𝐮 ) ] \opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[f(\mathbf{u})\right] .

The symmetry group on d d elements, S d S_{d} , acts on the set of probability distributions over ℝ d \mathbb{R}^{d} .

###### Definition A.9 (Pushforward distribution of a permutation [ Turner et al., 2021 ] ) .