Parametrically Retargetable Decision-Makers Tend To Seek Power — part 10/13
Cleaned-text code points [139390, 154428). Bibliography omitted here; source.md includes it.
For eq. 86 , p decide ( O leave ∣ θ ) \displaystyle p_{\mathrm{decide}}(O_{\text{leave}}\mid\theta) (95) ≔ ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( o T ∈ O leave ) \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}}\right) (96) = ∑ o ∈ O leave ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( o at step T ) \displaystyle=\sum_{o\in O_{\text{leave}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right) (97) = ∑ o ∈ O leave ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( o at step T ) ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{leave}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right) (98) = ∑ o ∈ 𝒪 ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( o at step T ) ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) \displaystyle=\sum_{o\in\mathcal{O}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right) (99) = ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( τ has left the first room by step T ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has left the first room by step $T$}\right) (100) ≕ p decide ( leave ∣ θ ) . \displaystyle\eqqcolon p_{\mathrm{decide}}(\text{leave}\mid\theta). (101) Equation 98 follows because, since o ∈ O leave o\in O_{\text{leave}} are only realizable by leaving the first room, this implies ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) = 1 \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right)=1 . Equation 99 follows because O leave ⊆ 𝒪 O_{\text{leave}}\subseteq\mathcal{O} , and probabilities are non-negative. Then we have shown eq. 86 . ∎
###### Corollary C.6 (Final reward maximizers tend to leave the first room in mr ) .
p max ( leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( stay ∣ R ) . p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\max}(\text{stay}\mid R). (102)
###### Proof.
Using Lemma C.5 and Proposition C.4 , apply Lemma B.1 with f 0 ( R ) ≔ p max ( leave ∣ R ) , f 1 ( R ) ≔ p max ( O leave ∣ R ) , f 2 ( R ) ≔ p max ( O stay ∣ R ) , f 3 ( R ) ≔ p max ( stay ∣ R ) f_{0}(R)\coloneqq p_{\max}(\text{leave}\mid R),f_{1}(R)\coloneqq p_{\text{max}}(O_{\text{leave}}\mid R),f_{2}(R)\coloneqq p_{\text{max}}(O_{\text{stay}}\mid R),f_{3}(R)\coloneqq p_{\max}(\text{stay}\mid R) to conclude that p max ( leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( stay ∣ R ) . p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\max}(\text{stay}\mid R). ∎
### C.3 Featurized reward maximization
Θ ≔ ℝ 𝒪 \Theta\coloneqq\mathbb{R}^{\mathcal{O}} assumes we will specify complicated reward functions over observations, with | 𝒪 | \left|\mathcal{O}\right| degrees of freedom in their specification. Any observation can get any number. However, reward functions are often specified more compactly. For example, in section 4.3 , the (additively) featurized reward function R feat ( o T ) ≔ feat ( o T ) ⊤ α R_{\textrm{feat}}(o_{T})\coloneqq\textrm{feat}(o_{T})^{\top}\alpha has four degrees of freedom. Compared to typical reward functions (which would look like “random noise” to a human), R feat R_{\textrm{feat}} more easily trains competent policies because of the regularities between the reward and the state features.
In this setup, p max p_{\text{max}} chooses a policy which induces a step- T T observation with maximal reward. Reward depends only on the feature vector of the final observation—more specifically, on the agent’s item counts. There are more possible item counts available by first leaving the room, than by staying.
We will now conduct a more detailed analysis and conclude that p max ( O leave ∣ α ) ≥ most : ℝ 4 3 p max ( O stay ∣ α ) p_{\text{max}}(O_{\text{leave}}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{3}p_{\text{max}}(O_{\text{stay}}\mid\alpha) . Informally, we can retarget which items the agent prioritizes, and thereby retarget from O stay O_{\text{stay}} to O leave O_{\text{leave}} .
Consider the featurization function which takes as input an observation o ∈ 𝒪 o\in\mathcal{O} : feat ( o ) ≔ ( # of keys in inventory shown by o # of swords in inventory shown by o # of torches in inventory shown by o # of amulets in inventory shown by o ) . \textrm{feat}(o)\coloneqq\begin{pmatrix}\text{\# of keys in inventory shown by $o$}\\ \text{\# of swords in inventory shown by $o$}\\ \text{\# of torches in inventory shown by $o$}\\ \text{\# of amulets in inventory shown by $o$}\end{pmatrix}. (103) Consider A feat ≔ { feat ( o ) ∣ o ∈ O stay } , B feat ≔ { feat ( o ) ∣ o ∈ O leave } A_{\text{feat}}\coloneqq\left\{\textrm{feat}(o)\mid o\in O_{\text{stay}}\right\},B_{\text{feat}}\coloneqq\left\{\textrm{feat}(o)\mid o\in O_{\text{leave}}\right\} .
Let 𝐞 i ∈ ℝ 4 \mathbf{e}_{i}\in\mathbb{R}^{4} be the standard basis vector with a 1 1 in entry i i and 0 0 elsewhere. When restricted to the room shown in fig. 2 , the agent can either acquire the key in the first room and retain it until step T T ( 𝐞 1 \mathbf{e}_{1} ), or reach time step T T empty-handed ( 𝟎 \mathbf{0} ). We conclude that A feat = { 𝐞 1 , 𝟎 } A_{\text{feat}}=\left\{\mathbf{e}_{1},\mathbf{0}\right\} .
For B feat B_{\text{feat}} , recall that in section 4.2 we assumed the rollout length T T to be reasonably large. Then by leaving the room, some realizable trajectory induces o T o_{T} displaying an inventory containing only a sword ( 𝐞 2 \mathbf{e}_{2} ), or only a torch ( 𝐞 3 \mathbf{e}_{3} ), or only an amulet ( 𝐞 4 \mathbf{e}_{4} ), or nothing at all ( 𝟎 \mathbf{0} ). Therefore, { 𝐞 2 , 𝐞 3 , 𝐞 4 , 𝟎 } ⊆ B feat \left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4},\mathbf{0}\right\}\subseteq B_{\text{feat}} . B feat B_{\text{feat}} contains 3 3 copies of A feat A_{\text{feat}} ( definition A.7 ) via involutions ϕ i : 1 ↔ i \phi_{i}\mathrel{\mathop{\ordinarycolon}}1\leftrightarrow i , i ≠ 1 i\neq 1 . Suppose all feature coefficient vectors α ∈ ℝ 4 \alpha\in\mathbb{R}^{4} are plausible. Then Θ ≔ ℝ 4 \Theta\coloneqq\mathbb{R}^{4} .
Let us be more specific about what is entailed by featurized reward maximization. The decide max ( α ) \mathrm{decide}_{\max}(\alpha) procedure takes α \alpha as input and then considers the reward function o ↦ feat ( o ) ⊤ α o\mapsto\textrm{feat}(o)^{\top}\alpha . Then, decide max \mathrm{decide}_{\max} uniformly randomly chooses an observation o T ∈ O T -reach o_{T}\in O_{\text{$T$-reach}} which maximizes this featurized reward, and then uniformly randomly chooses a policy which implements o T o_{T} .
###### Lemma C.7 ( FracOptimal \mathrm{FracOptimal} inequalities) .
Let X ⊆ Y ′ ⊆ Y ⊊ ℝ d X\subseteq Y^{\prime}\subseteq Y\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . Then FracOptimal ( X ∣ Y , 𝐮 ) ≤ FracOptimal ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \mathrm{FracOptimal}\left(X\mid Y,\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (104)
###### Proof.
For finite X 1 ⊊ ℝ d X_{1}\subsetneq\mathbb{R}^{d} , let Best ( X 1 ∣ 𝐮 ) ≔ arg max 𝐱 1 ∈ X 1 𝐱 1 ⊤ 𝐮 \mathrm{Best}\left(X_{1}\mid\mathbf{u}\right)\coloneqq\argmax_{\mathbf{x}_{1}\in X_{1}}\mathbf{x}_{1}^{\top}\mathbf{u} . Suppose 𝐲 ′ ∈ Best ( Y ′ ∣ 𝐮 ) \mathbf{y}^{\prime}\in\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) , but 𝐲 ′ ∉ Best ( Y ∣ 𝐮 ) \mathbf{y}^{\prime}\not\in\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then for all 𝐚 ∈ Best ( Y ′ ∣ 𝐮 ) \mathbf{a}\in\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) , 𝐚 ⊤ 𝐮 = 𝐲 ′ ⊤ 𝐮 < max 𝐲 ∈ Y 𝐲 ⊤ 𝐮 . \mathbf{a}^{\top}\mathbf{u}=\mathbf{y}^{\prime\top}\mathbf{u}<\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{u}. (105) So 𝐚 ∉ Best ( Y ∣ 𝐮 ) \mathbf{a}\not\in\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then either Best ( Y ′ ∣ 𝐮 ) ⊆ Best ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , or the two sets are disjoint. FracOptimal ( X ∣ Y , 𝐮 ) \displaystyle\mathrm{FracOptimal}\left(X\mid Y,\mathbf{u}\right) ≔ | Best ( Y ∣ 𝐮 ) ∩ X | | Best ( Y ∣ 𝐮 ) | \displaystyle\coloneqq\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (106) ≤ | Best ( Y ′ ∣ 𝐮 ) ∩ X | | Best ( Y ′ ∣ 𝐮 ) | ≕ FracOptimal ( X ∣ Y ′ , 𝐮 ) \displaystyle\leq\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|}\eqqcolon\mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right) (107) If Best ( Y ′ ∣ 𝐮 ) ⊆ Best ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , then since X ⊆ Y ′ X\subseteq Y^{\prime} , we have X ∩ Best ( Y ′ ∣ 𝐮 ) = X ∩ Best ( Y ∣ 𝐮 ) X\cap\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)=X\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then in this case, eq. 106 has equal numerator and larger denominator than eq. 107 . On the other hand, if Best ( Y ′ ∣ 𝐮 ) ∩ Best ( Y ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing , then since X ⊆ Y ′ X\subseteq Y^{\prime} , X ∩ Best ( Y ∣ 𝐮 ) = ∅ X\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing . Then eq. 106 equals 0 0 , and eq. 107 is non-negative. Either way, eq. 107 ’s inequality holds. To show the second inequality, we handle the two cases separately.
##### Subset case.
Suppose that Best ( Y ′ ∣ 𝐮 ) ⊆ Best ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) . | Best ( Y ′ ∣ 𝐮 ) ∩ X | | Best ( Y ′ ∣ 𝐮 ) | \displaystyle\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|} ≤ | Best ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ( Y ∖ Y ′ ∣ 𝐮 ) | | Best ( Y ′ ∣ 𝐮 ) | + | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle\leq\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (108) = | Best ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ( Y ′ ∣ 𝐮 ) | + | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (109) = | Best ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ( Y ′ ∣ 𝐮 ) | + | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (110) = | Best ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (111) = | Best ( Y ∣ 𝐮 ) ∩ X | + | Best ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (112) = | Best ( Y ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (113) ≕ FracOptimal ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \displaystyle\eqqcolon\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (114) Equation 108 follows because when n ≤ d , k ≥ 0 n\leq d,k\geq 0 , we have n d ≤ n + k d + k \frac{n}{d}\leq\frac{n+k}{d+k} . For eq. 110 , since Best ( Y ′ ∣ 𝐮 ) ⊆ Best ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , we must have Best ( Y ∣ 𝐮 ) = Best ( Y ∖ Y ′ ∣ 𝐮 ) ∪ Best ( Y ′ ∣ 𝐮 ) . \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cup\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right). But then Best ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) \displaystyle\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime}) = ( Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) ) ∪ ( Best ( Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) ) \displaystyle=\left(\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right)\cup\left(\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right) (115) = Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) . \displaystyle=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime}). (116) Then eq. 110 follows. Equation 111 follows since Best ( Y ∣ 𝐮 ) = Best ( Y ∖ Y ′ ∣ 𝐮 ) ∪ Best ( Y ′ ∣ 𝐮 ) . \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cup\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right). Equation 112 follows since X ⊆ Y ′ X\subseteq Y^{\prime} , and so Best ( Y ′ ∣ 𝐮 ) ∩ X = Best ( Y ∣ 𝐮 ) ∩ X . \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X=\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X. Equation 113 follows because X ⊆ Y ′ X\subseteq Y^{\prime} is disjoint of Y ∖ Y ′ Y\setminus Y^{\prime} . We have shown that FracOptimal ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) \mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right) in this case.
##### Disjoint case.