Parametrically Retargetable Decision-Makers Tend To Seek Power — part 11/13
Cleaned-text code points [154428, 170300). Bibliography omitted here; source.md includes it.
Suppose that Best ( Y ′ ∣ 𝐮 ) ∩ Best ( Y ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing . | Best ( Y ′ ∣ 𝐮 ) ∩ X | | Best ( Y ′ ∣ 𝐮 ) | \displaystyle\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|} ≤ 1 \displaystyle\leq 1 (117) = | Best ( Y ∖ Y ′ ∣ 𝐮 ) | | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (118) = | Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (119) = | Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (120) = | Best ( Y ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (121) ≕ FracOptimal ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \displaystyle\eqqcolon\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (122) Equation 117 follows because Best ( Y ′ ∣ 𝐮 ) ∩ X ⊆ Best ( Y ′ ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\subseteq\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) . For eq. 120 , note that we trivially have Best ( Y ′ ∣ 𝐮 ) ∩ Best ( Y ∖ Y ′ ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)=\varnothing , and also that X ⊆ Y ′ X\subseteq Y^{\prime} . Therefore, Best ( Y ∖ Y ′ ∣ 𝐮 ) ∩ X = ∅ \mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap X=\varnothing , and eq. 120 follows. Finally, the disjointness assumption implies that max 𝐲 ′ ∈ Y ′ 𝐲 ′ ⊤ 𝐮 < max 𝐲 ∈ Y 𝐲 ⊤ 𝐮 . \max_{\mathbf{y}^{\prime}\in Y^{\prime}}\mathbf{y}^{\prime\top}\mathbf{u}<\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{u}. Therefore, the optimal elements of Y Y must come exclusively from Y ∖ Y ′ Y\setminus Y^{\prime} ; i . e ., Best ( Y ∣ 𝐮 ) = Best ( Y ∖ Y ′ ∣ 𝐮 ) \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right) . Then eq. 121 follows, and we have shown that FracOptimal ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) \mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right) in this case. ∎
###### Conjecture C.8 (Generalizing Lemma C.7 ) .
Lemma C.7 and Turner et al. [2021] ’s Lemma E.26 have extremely similar functional forms. How can they be unified?
###### Proposition C.9 (Featurized reward maximizers tend to leave the first room in mr ) .
p max ( leave ∣ α ) ≥ most : ℝ 4 3 p max ( stay ∣ α ) . p_{\max}(\text{leave}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{3}p_{\max}(\text{stay}\mid\alpha). (123)
###### Proof.
We want to show that p max ( O leave ∣ α ) ≥ most : ℝ 4 n p max ( O stay ∣ α ) p_{\text{max}}(O_{\text{leave}}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{n}p_{\text{max}}(O_{\text{stay}}\mid\alpha) . Recall that A feat = { 𝐞 1 , 𝟎 } , B feat ′ ≔ { 𝐞 2 , 𝐞 3 , 𝐞 4 } ⊆ B feat A_{\text{feat}}=\{\mathbf{e}_{1},\mathbf{0}\},B^{\prime}_{\text{feat}}\coloneqq\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\}\subseteq B_{\text{feat}} . p max ( stay ∣ α ) \displaystyle p_{\max}(\text{stay}\mid\alpha) (124) ≤ p max ( O stay ∣ α ) \displaystyle\leq p_{\text{max}}(O_{\text{stay}}\mid\alpha) (125) ≔ ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay ) \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}}\right) (126) = ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ( o T ) ≠ 𝟎 ) + ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ( o T ) = 𝟎 ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})\neq\mathbf{0}\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (127) ≤ FracOptimal ( { 𝐞 1 } ∣ C feat , α ) + ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ( o T ) = 𝟎 ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid C_{\text{feat}},\alpha\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (128) ≤ FracOptimal ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) + ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ( o T ) = 𝟎 ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (129) ≤ most : ℝ 4 > 0 3 FracOptimal ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) \displaystyle\leq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right) + ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O leave , feat ( o T ) = 𝟎 ) \displaystyle\phantom{\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)}+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}},\textrm{feat}(o_{T})=\mathbf{0}\right) (130) ≤ FracOptimal ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∪ ( C feat ∖ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } ) ∣ C feat , α ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\cup(C_{\text{feat}}\setminus\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\})\mid C_{\text{feat}},\alpha\right) (131) = FracOptimal ( C feat ∖ { 𝐞 1 } ∣ C feat , α ) \displaystyle=\mathrm{FracOptimal}\left(C_{\text{feat}}\setminus\left\{\mathbf{e}_{1}\right\}\mid C_{\text{feat}},\alpha\right) (132) ≤ ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O leave ) \displaystyle\leq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}}\right) (133) ≕ p max ( O leave ∣ α ) \displaystyle\eqqcolon p_{\text{max}}(O_{\text{leave}}\mid\alpha) (134) ≤ p max ( leave ∣ α ) . \displaystyle\leq p_{\max}(\text{leave}\mid\alpha). (135) Equation 124 and eq. 135 hold by Lemma C.5 . If o T ∈ O stay o_{T}\in O_{\text{stay}} is realized by p max p_{\text{max}} and feat ( o T ) ≠ 𝟎 \textrm{feat}(o_{T})\neq\mathbf{0} , then we must have feat ( o T ) = { 𝐞 1 } \textrm{feat}(o_{T})=\{\mathbf{e}_{1}\} be optimal and so the 𝐞 1 \mathbf{e}_{1} inventory configuration is realized. Therefore, eq. 128 follows. Equation 129 follows by applying the first inequality of Lemma C.7 with X ≔ { 𝐞 1 } , Y ′ ≔ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , Y ≔ C feat X\coloneqq\{\mathbf{e}_{1}\},Y^{\prime}\coloneqq\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\},Y\coloneqq C_{\text{feat}} .
By applying Proposition A.11 ’s item 2 with A ≔ A feat = { 𝐞 1 } A\coloneqq A_{\text{feat}}=\{\mathbf{e}_{1}\} , B ′ ≔ B feat ′ = { 𝐞 2 , 𝐞 3 , 𝐞 4 } B^{\prime}\coloneqq B^{\prime}_{\text{feat}}=\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , C ≔ A ∪ B ′ C\coloneqq A\cup B^{\prime} , we have FracOptimal ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) ≤ most : ℝ 4 > 0 3 FracOptimal ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) . \mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)\leq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}\\ \mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right). (136) Furthermore, observe that ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ( o T ) = 𝟎 ) ≤ ℙ π ∼ decide max ( α ) , τ ∼ π | s 0 ( o T ∈ O leave , feat ( o T ) = 𝟎 ) \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right)\leq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}},\textrm{feat}(o_{T})=\mathbf{0}\right) (137) because either 𝟎 \mathbf{0} is not optimal (in which case both sides equal 0), or else 𝟎 \mathbf{0} is optimal, in which case the right side is strictly greater. This can be seen by considering how decide max ( α ) \mathrm{decide}_{\max}(\alpha) uniformly randomly chooses an observation in which the agent ends up with an empty inventory. As argued previously, the vast majority of such observations can only be induced by leaving the first room.
Combining eq. 136 and eq. 137 , eq. 130 follows. Equation 131 follows by applying the second inequality of Lemma C.7 with X ≔ { 𝐞 2 , 𝐞 3 , 𝐞 4 } X\coloneqq\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , Y ′ ≔ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } Y^{\prime}\coloneqq\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , Y ≔ C feat Y\coloneqq C_{\text{feat}} . If feat ( o T ) ∈ B feat \textrm{feat}(o_{T})\in B_{\text{feat}} is realized by p max p_{\text{max}} , then by the definition of B feat B_{\text{feat}} , o T ∈ O leave o_{T}\in O_{\text{leave}} is realized, and so eq. 133 follows.
Then by applying Lemma B.1 with f 0 ( α ) \displaystyle f_{0}(\alpha) ≔ p max ( leave ∣ α ) , \displaystyle\coloneqq p_{\max}(\text{leave}\mid\alpha), (138) f 1 ( α ) \displaystyle f_{1}(\alpha) ≔ FracOptimal ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) , \displaystyle\coloneqq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right), (139) f 2 ( α ) \displaystyle f_{2}(\alpha) ≔ FracOptimal ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) , \displaystyle\coloneqq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right), (140) f 3 ( α ) \displaystyle f_{3}(\alpha) ≔ p max ( stay ∣ α ) , \displaystyle\coloneqq p_{\max}(\text{stay}\mid\alpha), (141) we conclude that p max ( leave ∣ α ) ≥ most : ℝ 4 > 0 3 p max ( stay ∣ α ) . p_{\max}(\text{leave}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}p_{\max}(\text{stay}\mid\alpha). ∎
Lastly, note that if 𝟎 ∈ Θ \mathbf{0}\in\Theta and f ( A ∣ 𝟎 ) > f ( B ∣ 𝟎 ) f(A\mid\mathbf{0})>f(B\mid\mathbf{0}) , f f cannot be even be simply retargetable for the Θ \Theta parameter set. This is because ∀ ϕ ∈ S d \forall\phi\in S_{d} , ϕ ⋅ 𝟎 = 𝟎 \phi\cdot\mathbf{0}=\mathbf{0} . For example, inductive bias ensures that, absent a reward signal, learned policies tend to stay in the initial room in mr . This is one reason why section 4.3 ’s analysis of the policy tendencies of reinforcement learning excludes the all-zero reward function.
### C.4 Reasoning for why dqn can’t explore well
In section 4.3 , we wrote: Mnih et al. [2015] ’s dqn isn’t good enough to train policies which leave the first room of mr , and so dqn (trivially) cannot be retargetable away from the first room via the reward function. There isn’t a single featurized reward function for which dqn visits other rooms, and so we can’t have α \alpha such that ϕ ⋅ α \phi\cdot\alpha retargets the agent to O leave O_{\text{leave}} . dqn isn’t good enough at exploring.
We infer this is true from Nair et al. [2015] , which shows that vanilla dqn gets zero score in mr . Thus, dqn never even gets the first key. Thus, dqn only experiences state-action-state transitions which didn’t involve acquiring an item, since (as shown in fig. 3 ) the other items are outside of the first room, which requires a key to exit. In our analysis, we considered a reward function which is featurized over item acquisition.
Therefore, for all pre-key-acquisition state-action-state transitions, the featurized reward function returns exactly the same reward signals as those returned in training during the published experiments (namely, zero, because dqn can never even get to the key in order to receive a reward signal). That is, since dqn only experiences state-action-state transitions which didn’t involve acquiring an item, and the featurized reward functions only reward acquiring an item, it doesn’t matter what reward values are provided upon item acquisition— dqn ’s trained behavior will be the same. Thus, a dqn agent trained on any featurized reward function will not explore outside of the first room.
## Appendix D Lower bounds on mdp power-seeking incentives for optimal policies
Turner et al. [2021] prove conditions under which at least half of the orbit of every reward function incentivizes power-seeking behavior. For example, in fig. 4 , they prove that avoiding ∅ \varnothing maximizes average per-timestep reward for at least half of reward functions. Roughly, there are more self-loop states ( ∅ \varnothing , ℓ ↙ \ell_{\swarrow} , r ↘ r_{\searrow} , r ↗ r_{\nearrow} ) available if the agent goes left or right instead of up towards ∅ \varnothing . We strengthen this claim, with Corollary D.12 showing that for at least three-quarters of the orbit of every reward function, it is average-optimal to avoid ∅ \varnothing .
Therefore, we answer Turner et al. [2021] ’s open question of whether increased number of environmental symmetries quantitatively strengthens the degree to which power-seeking is incentivized. The answer is yes . In particular, it may be the case that only one in a million state-based reward functions makes it average-optimal for Pac-Man to die immediately.
We will briefly restate several definitions needed for our key results, Theorem D.11 and Corollary D.12 . For explanation, see Turner et al. [2021] .
###### Definition D.1 (Non-dominated linear functionals) .
Let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. ND ( X ) ≔ { 𝐱 ∈ X ∣ ∃ 𝐫 ∈ ℝ | 𝒮 | : 𝐱 ⊤ 𝐫 > max 𝐱 ′ ∈ X ∖ { 𝐱 } 𝐱 ′ ⊤ 𝐫 } \text{{ND}}\left(X\right)\coloneqq\left\{\mathbf{x}\in X\mid\exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{r}>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}\mathbf{x}^{\prime\top}\mathbf{r}\right\} .
###### Definition D.2 (Bounded reward function distribution) .
𝔇 bound \mathfrak{D}_{\text{bound}} is the set of bounded-support probability distributions 𝒟 bound \mathcal{D}_{\text{bound}} .
###### Remark .
When n = 1 n=1 , Lemma D.3 reduces to the first part of Turner et al. [2021] ’s lemma E.24, and Lemma D.5 reduces to the first part of Turner et al. [2021] ’s lemma E.28.
###### Lemma D.3 (Quantitative expectation superiority lemma) .