Parametrically Retargetable Decision-Makers Tend To Seek Power — part 2/13

Cleaned-text code points [15226, 30540). Bibliography omitted here; source.md includes it.

Suppose Θ \Theta is a subset of a set acted on by S d S_{d} , the symmetric group on d d elements. Let f : { A , B } × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} and let n ≥ 1 n\geq 1 . We write f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta) when, for all θ ∈ Θ \theta\in\Theta , the following cardinality inequality holds: | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( B ∣ θ ′ ) > f ⁡ ( A ∣ θ ′ ) } | ≥ n ​ | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( B ∣ θ ′ ) < f ⁡ ( A ∣ θ ′ ) } | . \left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(B\mid\theta^{\prime})>f(A\mid\theta^{\prime})\right\}\right|\geq n\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(B\mid\theta^{\prime})<f(A\mid\theta^{\prime})\right\}\right|. (2)

For example, Table 1 illustrates the tendency of 𝐮 \mathbf{u} ’s orbit to make B ≔ { , } B\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\} optimal over A ≔ { } A\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\right\} . Turner et al. [2021] ’s definition 6.5 is the special case of definition 3.2 where n = 1 n=1 , d = | 𝒮 | d=\left|\mathcal{S}\right| (the number of states in the considered mdp ), and Θ ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \Theta\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) .

As explored previously, p rand p_{\text{rand}} , p max p_{\text{max}} , and p numerical p_{\text{numerical}} are retargetable: For all θ ∈ Θ \theta\in\Theta such that { } \left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\right\} is chosen over { , } \left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\} , we can permute θ \theta to obtain ϕ ⋅ θ \phi\cdot\theta under which the opposite is true. More generally, we can consider retargetability from some set A A to some set B B . 2 2 2 We often interpret A A and B B as probability-theoretic events, but no such structure is demanded by our results.

###### Definition 3.3 (Simply-retargetable function) .

Let Θ \Theta be a set acted on by S d S_{d} , and let f : { A , B } × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} . If ∃ ϕ ∈ S d : ∀ θ A ∈ Θ : f ⁡ ( B ∣ θ A ) < f ⁡ ( A ∣ θ A ) ⟹ f ⁡ ( A ∣ ϕ ⋅ θ A ) < f ⁡ ( B ∣ ϕ ⋅ θ A ) \exists\phi\in S_{d}\mathrel{\mathop{\ordinarycolon}}\forall\theta^{A}\in\Theta\mathrel{\mathop{\ordinarycolon}}f(B\mid\theta^{A})<f(A\mid\theta^{A})\implies f(A\mid\phi\cdot\theta^{A})<f(B\mid\phi\cdot\theta^{A}) , then f f is a ( Θ , A ​ → simple ​ B ) (\Theta,A\overset{\text{simple}}{\to}B) -retargetable function .

Simple retargetability suffices for most parameter inputs to p p to choose Pac-Man outcome set B B over A A . 3 3 3 The function’s retargetability is “simple” because we are not yet worrying about e . g ., which parameter inputs are considered plausible: Because S d S_{d} acts on Θ \Theta , definition 3.3 implicitly assumes Θ \Theta is closed under permutation. In that case, B B cannot be retargeted back to A A because | B | = 2 > 1 = | A | \left|B\right|=2>1=\left|A\right| . p max p_{\text{max}} ’s simple retargetability arises in part due to B B having more outcomes.

###### Proposition 3.4 (Simply-retargetable functions have orbit-level tendencies) .

If f f is ( Θ , A ​ → simple ​ B ) (\Theta,A\overset{\text{simple}}{\to}B) -retargetable, then f ( B ∣ θ ) ≥ most : Θ 1 f ( A ∣ θ ) . f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{1}f(A\mid\theta).

We now want to make even stronger claims— how much of each orbit incentivizes B B over A A ? Turner et al. [2021] asked whether the existence of multiple retargeting permutations ϕ i \phi_{i} guarantees a quantitative lower-bound on the fraction of θ ∈ Θ \theta\in\Theta for which B B is chosen. Theorem 3.6 answers “yes.”

###### Definition 3.5 (Multiply retargetable function) .

Let Θ \Theta be a subset of a set acted on by S d S_{d} , and let f : { A , B } × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} .

f f is a ( Θ , A ​ → 𝑛 ​ B ) (\Theta,A\overset{n}{\to}B) -retargetable function when, for each θ ∈ Θ \theta\in\Theta , we can choose permutations ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} which satisfy the following conditions: Consider any θ A ∈ Orbit | Θ , A > B ​ ( θ ) ≔ { θ ∗ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( A ∣ θ ∗ ) > f ⁡ ( B ∣ θ ∗ ) } \theta^{A}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\coloneqq\left\{\theta^{*}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(A\mid\theta^{*})>f(B\mid\theta^{*})\right\} . 1. Retargetable via n n permutations. ∀ i = 1 , … , n : f ⁡ ( A ∣ ϕ i ⋅ θ A ) < f ⁡ ( B ∣ ϕ i ⋅ θ A ) \forall i=1,\ldots,n\mathrel{\mathop{\ordinarycolon}}f\left(A\mid\phi_{i}\cdot\theta^{A}\right)<f\left(B\mid\phi_{i}\cdot\theta^{A}\right) .

2. Parameter permutation is allowed by Θ \Theta . ∀ i : ϕ i ⋅ θ A ∈ Θ \forall i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot\theta^{A}\in\Theta .

3. Permuted parameters are distinct. ∀ i ≠ j , θ ′ ∈ Orbit | Θ , A > B ​ ( θ ) : ϕ i ⋅ θ A ≠ ϕ j ⋅ θ ′ \forall i\neq j,\theta^{\prime}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot\theta^{A}\neq\phi_{j}\cdot\theta^{\prime} .

###### Theorem 3.6 (Multiply retargetable functions have orbit-level tendencies) .

If f f is ( Θ , A ​ → 𝑛 ​ B ) (\Theta,A\overset{n}{\to}B) -retargetable, then f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) . f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta).

###### Proof outline (full proof in Appendix B ).

For every θ A ∈ Orbit | Θ , A > B ​ ( θ ) \theta^{A}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) such that A A is chosen over B B , item 1 retargets θ A \theta^{A} via n n permutations ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} such that each ϕ i ⋅ θ A \phi_{i}\cdot\theta^{A} makes the agent choose B B over A A . These permuted parameters are valid parameter inputs by item 2 . Furthermore, the ϕ i ⋅ θ A \phi_{i}\cdot\theta^{A} are distinct by item 3 . Therefore, the cosets ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) \phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) are pairwise disjoint. By a counting argument, every orbit must contain at least n n times as many parameters choosing B B over A A , than vice versa. ∎

## 4 Decision-making tendencies in Montezuma’s Revenge

To illustrate a high-dimensional setting in which parametrically retargetable decision-makers tend to seek power, we consider Montezuma’s Revenge ( mr ), an Atari adventure game in which the player navigates deadly traps and collects treasure. The game is notoriously difficult for ai agents due to its sparse reward. mr was only recently solved [ Ecoffet et al., 2021 ] . Figure 2 shows the starting observation o 0 o_{0} for the first level. This section culminates with section 4.3 , where we argue that increasingly powerful rl training processes will cause increasing retargetability via the reward function, which in turn causes increasingly strong decision-making tendencies.

##### Terminology.

Retargetability is a property of the policy training process, and power-seeking is a property of the trained policy. More precisely, the policy training process takes as input a parameterization θ \theta and outputs a probability distribution over policies. For each trained policy drawn from this distribution, the environment, starting state, and the drawn policy jointly specify a probability distribution over trajectories. Therefore, the training process associates each parameterization θ \theta with the mixture distribution P P over trajectories (with the mixture taken over the distribution of trained policies).

A policy training process can be simply retargeted from one trajectory set A A to another trajectory set B B when there exists a permutation ϕ ∈ S d \phi\in S_{d} such that, for every θ \theta for which P ⁡ ( A ∣ θ ) > P ⁡ ( B ∣ θ ) P(A\mid\theta)>P(B\mid\theta) , we have P ⁡ ( A ∣ ϕ ⋅ θ ) < P ⁡ ( B ∣ ϕ ⋅ θ ) P(A\mid\phi\cdot\theta)<P(B\mid\phi\cdot\theta) . As in Turner et al. [2021] , a trained policy π \pi seeks power when π \pi ’s actions navigate to states with high average optimal value (with the average taken over a wide range of reward functions). Generally, high-power states are able to reach a wide range of other states, and so allow bigger option sets B B (compared to the options A A available without seeking power).

### 4.1 Tendencies for initial action selection

We will be considering the actions chosen and trajectories induced by a range of decision-making procedures. For warm-up, we will explore what initial action tends to be selected by decision-makers. Let A ≔ { ↓ } , B : = { ← , → , jump , ↑ } A\coloneqq\{\downarrow\},B\mathrel{\mathop{\ordinarycolon}}=\{\leftarrow,\rightarrow,\texttt{jump},\uparrow\} partition the action set 𝒜 \mathcal{A} . Consider a decision-making procedure f f which takes as input a targeting parameter θ ∈ Θ \theta\in\Theta , and also an initial action a ∈ 𝒜 a\in\mathcal{A} , and returns the probability that a a is the first action. Intuitively, since B B contains more actions than A A , perhaps some class of decision-making procedures tends to take an action in B B rather than one in A A .

mr ’s initial-action situation is analogous to the Pac-Man example. In that example, if the decision-making procedure p p can be retargeted from terminal state set A A (the ghost) to set B B (the fruit), then p p tends to select a state from B B under most of its parameter settings θ \theta . Similarly, in mr , if the decision-making procedure f f can be retargeted from action set A A to action set B B , then f f tends to take actions in B B for most of its parameter settings θ \theta . Consider several ways of choosing an initial action in mr .

Random action selection. p rand ≔ ( { a } ∣ θ ) ↦ 1 5 p_{\text{rand}}\coloneqq(\{a\}\mid\theta)\mapsto\frac{1}{5} uniformly randomly chooses an action from 𝒜 \mathcal{A} , ignoring the parameter input. Since ∀ θ ∈ Θ : p rand ​ ( B ∣ θ ) = 4 5 > 1 5 = p rand ​ ( A ∣ θ ) \forall\theta\in\Theta\mathrel{\mathop{\ordinarycolon}}p_{\text{rand}}(B\mid\theta)=\frac{4}{5}>\frac{1}{5}=p_{\text{rand}}(A\mid\theta) , all parameter inputs produce a greater chance of B B than of A A , so p rand p_{\text{rand}} is (trivially) retargetable from A A to B B .

Always choosing the same action. p stubborn p_{\text{stubborn}} always chooses ↓ \downarrow . Since ∀ θ ∈ Θ : p stubborn ​ ( A ∣ θ ) = 1 > 0 = p stubborn ​ ( B ∣ θ ) \forall\theta\in\Theta\mathrel{\mathop{\ordinarycolon}}p_{\text{stubborn}}(A\mid\theta)=1>0=p_{\text{stubborn}}(B\mid\theta) , all parameter inputs produce a greater chance of A A than of B B . p stubborn p_{\text{stubborn}} is not retargetable from A A to B B .

Greedily optimizing state-action reward. Let Θ ≔ ℝ 𝒮 × 𝒜 \Theta\coloneqq\mathbb{R}^{\mathcal{S}\times\mathcal{A}} be the space of state-action reward functions. Let p max p_{\text{max}} greedily maximize initial state-action reward, breaking ties uniformly randomly.

We now check that p max p_{\text{max}} is retargetable from A A to B B . Suppose θ ∗ ∈ Θ \theta^{*}\in\Theta is such that p max ​ ( A ∣ θ ∗ ) > p max ​ ( B ∣ θ ∗ ) p_{\text{max}}(A\mid\theta^{*})>p_{\text{max}}(B\mid\theta^{*}) . Then among the initial action rewards, θ ∗ \theta^{*} assigns strictly maximal reward to ↓ \downarrow , and so p max ​ ( A ∣ θ ∗ ) = 1 p_{\text{max}}(A\mid\theta^{*})=1 . Let ϕ \phi swap the reward for the ↓ \downarrow and jump actions. Then ϕ ⋅ θ ∗ \phi\cdot\theta^{*} assigns strictly maximal reward to jump . This means that p max ​ ( A ∣ ϕ ⋅ θ ∗ ) = 0 < 1 = p max ​ ( B ∣ ϕ ⋅ θ ∗ ) p_{\text{max}}(A\mid\phi\cdot\theta^{*})=0<1=p_{\text{max}}(B\mid\phi\cdot\theta^{*}) , satisfying definition 3.3 . Then apply Proposition 3.4 to conclude that p max ( B ∣ θ ) ≥ most : Θ 1 p max ( A ∣ θ ) p_{\text{max}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{1}p_{\text{max}}(A\mid\theta) .

In fact, appendix A shows that p max p_{\text{max}} is ( Θ , A ​ → 4 ​ B ) (\Theta,A\overset{4}{\to}B) -retargetable ( definition 3.5 ), and so p max ( B ∣ θ ) ≥ most : Θ 4 p max ( A ∣ θ ) p_{\text{max}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{4}p_{\text{max}}(A\mid\theta) . The reasoning is more complicated, but the rule of thumb is: When decisions are made based on the reward of outcomes, then a proportionally larger set B B of outcomes induces proportionally strong retargetability, which induces proportionally strong orbit-level incentives.

Learning an exploitation policy. Suppose we run a bandit algorithm which tries different initial actions, learns their rewards, and produces an exploitation policy which maximizes estimated reward. The algorithm uses ϵ \epsilon -greedy exploration and trains for T T trials. Given fixed T T and ϵ \epsilon , p bandit ​ ( A ∣ θ ) p_{\text{bandit}}(A\mid\theta) returns the probability that an exploitation policy is learned which chooses an action in A A ; likewise for p bandit ​ ( B ∣ θ ) p_{\text{bandit}}(B\mid\theta) .

Here is a heuristic argument that p bandit p_{\text{bandit}} is retargetable. Since the reward is deterministic, the exploitation policy will choose an optimal action if the agent has tried each action at least once, which occurs with a probability approaching 1 1 exponentially quickly in the number of trials T T . Then when T T is large, p bandit p_{\text{bandit}} approximates p max p_{\text{max}} , which is retargetable. Therefore, perhaps p bandit p_{\text{bandit}} is also retargetable. A more careful analysis in appendix C.1 reveals that p bandit p_{\text{bandit}} is 4-retargetable from A A to B B , and so p bandit ( B ∣ θ ) ≥ most : Θ 4 p bandit ( A ∣ θ ) p_{\text{bandit}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{4}p_{\text{bandit}}(A\mid\theta) .

### 4.2 Tendencies for maximizing reward over the final observation

When evaluating the performance of an algorithm in mr , we do not focus on the agent’s initial action. Rather, we focus on the longer-term consequences of the agent’s actions, such as whether the agent leaves the first room. To begin reasoning about such behavior, the reader must distinguish between different kinds of retargetability.

Suppose the agent will die unless they choose action ↓ \downarrow at the initial state s 0 s_{0} ( Figure 2 ). By section 4.1 , action-retargetable decision-making procedures tend to choose actions besides ↓ \downarrow . On the other hand, Turner et al. [2021] showed that most reward functions make it reward-optimal to stay alive (in this situation, by choosing ↓ \downarrow ). However, in that situations, the optimal policies are not retargetable across the agent’s immediate choice of action, but rather across future consequences ( i . e ., which room the agent ends up in).