Parametrically Retargetable Decision-Makers Tend To Seek Power — part 9/13

Cleaned-text code points [124362, 139390). Bibliography omitted here; source.md includes it.

Let f ⁡ ( X ∣ 𝐮 ) ≔ Q q , P ​ ( X ∣ C , 𝐮 ) f(X\mid\mathbf{u})\coloneqq Q_{q,P}(X\mid C,\mathbf{u}) . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C and since Q q , P ​ ( X ∣ C , 𝐮 ) = Q q , P ​ ( ϕ i ⋅ X ∣ ϕ i ⋅ C , ϕ i ⋅ 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u})=Q_{q,P}(\phi_{i}\cdot X\mid\phi_{i}\cdot C,\phi_{i}\cdot\mathbf{u}) , Lemma B.10 shows that f ⁡ ( X ∣ 𝐮 ) f(X\mid\mathbf{u}) is also jointly invariant to permutation by ϕ i \phi_{i} . Lastly, if X ′ ⊆ X X^{\prime}\subseteq X , we have f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) .

Apply Lemma B.9 to conclude that Q q , P ( B ∣ C , 𝐮 ) ≕ f ( B ∣ 𝐮 ) ≥ most : ℝ d n f ( A ∣ 𝐮 ) ≔ Q q , P ( A ∣ C , 𝐮 ) Q_{q,P}(B\mid C,\mathbf{u})\eqqcolon f(B\mid\mathbf{u})\geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n}f(A\mid\mathbf{u})\coloneqq Q_{q,P}(A\mid C,\mathbf{u}) . ∎

###### Conjecture B.13 (Orbit tendencies occur for more quantilizer base distributions) .

Proposition A.11 ’s item 7 holds for any base distribution P P over C C such that min 𝐛 ∈ B ⁡ P ⁡ ( 𝐛 ) ≥ max 𝐚 ∈ A ⁡ P ⁡ ( 𝐚 ) \min_{\mathbf{b}\in B}P(\mathbf{b})\geq\max_{\mathbf{a}\in A}P(\mathbf{a}) . Furthermore, Q q , P ​ ( X ∣ C , 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u}) is measurable on 𝐮 \mathbf{u} and so ≥ most : ℝ d n \geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n} can be generalized to ≥ most : 𝔇 any n \geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n} .

## Appendix C Detailed analyses of mr scenarios

### C.1 Action selection

Consider a bandit problem with five arms a 1 , … , a 5 a_{1},\ldots,a_{5} partitioned A ≔ { a 1 } , B ≔ { a 2 , … , a 5 } A\coloneqq\left\{a_{1}\right\},B\coloneqq\left\{a_{2},\ldots,a_{5}\right\} , which each action has a definite utility 𝐮 i \mathbf{u}_{i} . There are T = 100 T=100 trials. Suppose the training procedure train uses the ϵ \epsilon -greedy strategy to learn value estimates for each arm. At the end of training, train outputs a greedy policy with respect to its value estimates. Consider any action-value initialization, and the learning rate is set α ≔ 1 \alpha\coloneqq 1 . To learn an optimal policy, at worst, the agent just has to try each action once.

###### Lemma C.1 (Lower bound on success probability of the train bandit) .

Let 𝐮 ∈ ℝ 5 \mathbf{u}\in\mathbb{R}^{5} assign strictly maximal utility to a i a_{i} , and suppose train (described above) runs for T ≥ 5 T\geq 5 trials. Then p train ​ ( { a i } ∣ 𝐮 ) ≥ 1 − ( 1 − ϵ 4 ) T p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\geq 1-(1-\frac{\epsilon}{4})^{T} .

###### Proof.

Since the trained policy can be stochastic, p train ​ ( { a i } ∣ 𝐮 ) ≥ ℙ ( a i ​ is assigned probability 1 by the learned greedy policy ) . p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\geq\Prb\left(a_{i}\text{ is assigned probability $1$ by the learned greedy policy}\right).

Since a i a_{i} has strictly maximal utility which is deterministic, and since the learning rate α ≔ 1 \alpha\coloneqq 1 , if action a i a_{i} is ever drawn, it is assigned probability 1 1 by the learned policy. The probability that a i a_{i} is never explored is at most ( 1 − ϵ 4 ) T (1-\frac{\epsilon}{4})^{T} , because at worst, a i a_{i} is an “explore” action (and not an “exploit” action) at every time step, in which case it is ignored with probability 1 − ϵ 4 1-\frac{\epsilon}{4} . ∎

###### Proposition C.2 (The train bandit is 4-retargetable) .

p train p_{\textrm{train}} is ( ℝ 5 , A ​ → 4 ​ B ) (\mathbb{R}^{5},A\overset{4}{\to}B) -retargetable.

###### Proof.

Let ϕ i ≔ a 1 ↔ a i \phi_{i}\coloneqq a_{1}\leftrightarrow a_{i} for i = 2 , … , 5 i=2,\ldots,5 and let Θ ≔ ℝ 5 \Theta\coloneqq\mathbb{R}^{5} . We want to show that whenever 𝐮 ∈ ℝ 5 \mathbf{u}\in\mathbb{R}^{5} induces p train ​ ( A ∣ 𝐮 ) > p train ​ ( B ∣ 𝐮 ) p_{\textrm{train}}(A\mid\mathbf{u})>p_{\textrm{train}}(B\mid\mathbf{u}) , retargeting 𝐮 \mathbf{u} will get train to instead learn to pull a B B -action: p train ​ ( A ∣ ϕ i ⋅ 𝐮 ) < p train ​ ( B ∣ ϕ i ⋅ 𝐮 ) p_{\textrm{train}}(A\mid\phi_{i}\cdot\mathbf{u})<p_{\textrm{train}}(B\mid\phi_{i}\cdot\mathbf{u}) .

Suppose we have such a 𝐮 \mathbf{u} . If 𝐮 \mathbf{u} is constant, a symmetry argument shows that each action has equal probability of being selected, in which case p train ​ ( A ∣ 𝐮 ) = 1 5 < 4 5 = p train ​ ( B ∣ 𝐮 ) p_{\textrm{train}}(A\mid\mathbf{u})=\frac{1}{5}<\frac{4}{5}=p_{\textrm{train}}(B\mid\mathbf{u}) —a contradiction. Therefore, 𝐮 \mathbf{u} is not constant. Similar symmetry arguments show that A A ’s action a 1 a_{1} has strictly maximal utility ( 𝐮 1 > max i = 2 , … , 5 ⁡ 𝐮 i \mathbf{u}_{1}>\max_{i=2,\ldots,5}\mathbf{u}_{i} ).

But for T = 100 T=100 , Lemma C.1 shows that p train ​ ( A ∣ 𝐮 ) = p train ​ ( { a 1 } ∣ 𝐮 ) ≈ 1 p_{\textrm{train}}(A\mid\mathbf{u})=p_{\textrm{train}}(\left\{a_{1}\right\}\mid\mathbf{u})\approx 1 and p train ​ ( { a i ≠ 1 } ∣ 𝐮 ) ≈ 0 ⟹ p train ​ ( B ∣ 𝐮 ) = ∑ i ≠ 1 p train ​ ( { a i } ∣ 𝐮 ) ≈ 0 p_{\textrm{train}}(\left\{a_{i\neq 1}\right\}\mid\mathbf{u})\approx 0\implies p_{\textrm{train}}(B\mid\mathbf{u})=\sum_{i\neq 1}p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\approx 0 . The converse statement holds when considering ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} instead of 𝐮 \mathbf{u} . Therefore, train satisfies definition 3.5 ’s item 1 (retargetability). These ϕ i ⋅ 𝐮 ∈ Θ ≔ ℝ 5 \phi_{i}\cdot\mathbf{u}\in\Theta\coloneqq\mathbb{R}^{5} because ℝ 5 \mathbb{R}^{5} is closed under permutation by S 5 S_{5} , satisfying item 2 .

Consider another 𝐮 ′ ∈ ℝ 5 \mathbf{u}^{\prime}\in\mathbb{R}^{5} such that p train ​ ( A ∣ 𝐮 ′ ) > p train ​ ( B ∣ 𝐮 ′ ) p_{\textrm{train}}(A\mid\mathbf{u}^{\prime})>p_{\textrm{train}}(B\mid\mathbf{u}^{\prime}) , and consider i ≠ j i\neq j . By the above symmetry arguments, 𝐮 ′ \mathbf{u}^{\prime} must also assign a 1 a_{1} maximal utility. By Lemma C.1 , p train ​ ( { a i } ∣ ϕ i ⋅ 𝐮 ) ≈ 1 p_{\textrm{train}}(\left\{a_{i}\right\}\mid\phi_{i}\cdot\mathbf{u})\approx 1 and p train ​ ( { a j } ∣ ϕ i ⋅ 𝐮 ) ≈ 0 p_{\textrm{train}}(\left\{a_{j}\right\}\mid\phi_{i}\cdot\mathbf{u})\approx 0 since i ≠ j i\neq j , and vice versa when considering ϕ j ⋅ 𝐮 \phi_{j}\cdot\mathbf{u} instead of ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} . Then since ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} and ϕ j ⋅ 𝐮 \phi_{j}\cdot\mathbf{u} induce distinct probability distributions over learned actions, they cannot be the same utility function. This satisfies item 3 . ∎

###### Corollary C.3 (The train bandit has orbit-level tendencies) .

p train ( B ∣ 𝐮 ) ≥ most: ℝ 5 4 p train ( A ∣ 𝐮 ) p_{\textrm{train}}(B\mid\mathbf{u})\geq^{4}_{\text{most: }\mathbb{R}^{5}}p_{\textrm{train}}(A\mid\mathbf{u}) .

###### Proof.

Combine Proposition C.2 and Theorem 3.6 . ∎

### C.2 Observation reward maximization

Let T T be a reasonably long rollout length, so that O T -reach O_{\text{$T$-reach}} is large—many different step- T T observations can be induced.

###### Proposition C.4 (Final reward maximization has strong orbit-level incentives in mr ) .

Let n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor . p max ( O leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( O stay ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\text{max}}(O_{\text{stay}}\mid R) .

###### Proof.

Consider the vector space representation of observations, ℝ | 𝒪 | \mathbb{R}^{\left|\mathcal{O}\right|} . Define A ≔ { 𝐞 o ∣ o ∈ O stay } , B ≔ { 𝐞 o ∣ o ∈ O leave } A\coloneqq\{\mathbf{e}_{o}\mid o\in O_{\text{stay}}\},B\coloneqq\{\mathbf{e}_{o}\mid o\in O_{\text{leave}}\} , and C ≔ O T -reach = A ∪ B C\coloneqq O_{\text{$T$-reach}}=A\cup B the union of O stay , O leave O_{\text{stay}},O_{\text{leave}} .

Since | O leave | ≥ | O stay | \left|O_{\text{leave}}\right|\geq\left|O_{\text{stay}}\right| by assumption that T T is reasonably large, consider the involution ϕ 1 ∈ S | 𝒪 | \phi_{1}\in S_{\left|\mathcal{O}\right|} which embeds O stay O_{\text{stay}} into O leave O_{\text{leave}} , while fixing all other observations. If possible, produce another involution ϕ 2 \phi_{2} which also embeds O stay O_{\text{stay}} into O leave O_{\text{leave}} , which fixes all other observations, and which “doesn’t interfere with ϕ 1 \phi_{1} ” ( i . e ., ϕ 2 ⋅ ( ϕ 1 ⋅ A ) = ϕ 1 ⋅ A \phi_{2}\cdot(\phi_{1}\cdot A)=\phi_{1}\cdot A ). We can produce n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor such involutions. Therefore, B B contains n n copies ( definition A.7 ) of A A via involutions ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} . Furthermore, ϕ i ⋅ ( A ∪ B ) = A ∪ B \phi_{i}\cdot(A\cup B)=A\cup B , since each ϕ i \phi_{i} swaps A A with B ′ ⊆ B B^{\prime}\subseteq B , and fixes all 𝐛 ∈ B ∖ B ′ \mathbf{b}\in B\setminus B^{\prime} by assumption. Thus, ϕ ⋅ C = C \phi\cdot C=C .

By Proposition A.11 ’s item 2 , FracOptimal ( B ∣ C , R ) ≥ most : ℝ 𝒪 n FracOptimal ( A ∣ C , R ) \mathrm{FracOptimal}\left(B\mid C,R\right)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}\mathrm{FracOptimal}\left(A\mid C,R\right) . Since p max p_{\text{max}} uniformly randomly chooses a maximal-reward observation to induce, ∀ X ⊆ C : p max ​ ( X ∣ R ) = FracOptimal ⁡ ( X ∣ C , R ) \forall X\subseteq C\mathrel{\mathop{\ordinarycolon}}p_{\text{max}}(X\mid R)=\mathrm{FracOptimal}\left(X\mid C,R\right) . Therefore, p max ( O leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( O stay ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\text{max}}(O_{\text{stay}}\mid R) . ∎

We want to reason about the probability that decide \mathrm{decide} leaves the initial room by time T T in its rollout trajectories. p decide ​ ( leave ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{leave}\mid\theta) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ has left the first room by step T ) , \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has left the first room by step $T$}\right), (83) p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ has not left the first room by step T ) . \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has not left the first room by step $T$}\right). (84)

We want to show that reward maximizers tend to leave the room: p max ( leave ∣ R ) ≥ most : Θ n p max ( stay ∣ R ) p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{\max}(\text{stay}\mid R) . However, we must be careful: In general, p max ​ ( O leave ∣ R ) ≠ p max ​ ( leave ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\neq p_{\max}(\text{leave}\mid R) and p max ​ ( O stay ∣ R ) ≠ p max ​ ( stay ∣ R ) p_{\text{max}}(O_{\text{stay}}\mid R)\neq p_{\max}(\text{stay}\mid R) . For example, suppose that o T ∈ O leave o_{T}\in O_{\text{leave}} . By the definition of O leave O_{\text{leave}} , o T o_{T} can only be observed if the agent has left the room by time step T T , and so the trajectory τ \tau must have left the first room. The converse argument does not hold: The agent could leave the first room, re-enter, and then wait until time T T . Although one of the doors would have been opened ( fig. 2 ), the agent can also open the door without leaving the room, and then realize the same step- T T observation. Therefore, this observation doesn’t belong to O leave O_{\text{leave}} .

###### Lemma C.5 (Room-status inequalities for mr ) .

p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) ≤ p decide ​ ( O stay ∣ θ ) , \displaystyle\leq p_{\mathrm{decide}}(O_{\text{stay}}\mid\theta), (85) and ​ p decide ​ ( O leave ∣ θ ) \displaystyle\text{and }p_{\mathrm{decide}}(O_{\text{leave}}\mid\theta) ≤ p decide ​ ( leave ∣ θ ) . \displaystyle\leq p_{\mathrm{decide}}(\text{leave}\mid\theta). (86)

###### Proof.

For any decide \mathrm{decide} , p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) (87) = ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ stays through step T ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ stays through step $T$}\right) (88) = ∑ o ∈ 𝒪 ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T of τ ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in\mathcal{O}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$ of $\tau$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (89) = ∑ o ∈ O T -reach ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{$T$-reach}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (90) = ∑ o ∈ O stay ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{stay}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (91) ≤ ∑ o ∈ O stay ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) \displaystyle\leq\sum_{o\in O_{\text{stay}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right) (92) = ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o T ∈ O stay ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}}\right) (93) ≕ p decide ​ ( O stay ∣ θ ) . \displaystyle\eqqcolon p_{\mathrm{decide}}(O_{\text{stay}}\mid\theta). (94) Equation 90 holds because the definition of O T -reach O_{\text{$T$-reach}} ensures that if o ∉ O T -reach o\not\in O_{\text{$T$-reach}} , then ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o ∣ θ ) = 0 \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o\mid\theta\right)=0 . Because o ∈ O T -reach ∖ O stay o\in O_{\text{$T$-reach}}\setminus O_{\text{stay}} implies that τ \tau left and so ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) = 0 , \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right)=0, eq. 91 follows. Then we have shown eq. 85 .