Optimal Policies Tend to Seek Power — part 3/16

Cleaned-text code points [31329, 47054). Bibliography omitted here; source.md includes it.

Proposition 6.6 proves that for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] , Power 𝒟 bound ( r ↘ , γ ) ≥ most Power 𝒟 bound ( ℓ ↙ , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}(r_{\searrow},\gamma)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(\ell_{\swarrow},\gamma) via s ′ ≔ ℓ ↙ , s ≔ r ↘ s^{\prime}\coloneqq\ell_{\swarrow},s\coloneqq r_{\searrow} , and the involution ϕ \phi shown in fig. 4 . In fact, because ( 1 1 − γ ​ 𝐞 r ↗ ) ∈ ℱ nd ⁡ ( r ↘ ) ∖ ϕ ⋅ ℱ nd ⁡ ( ℓ ↙ ) (\frac{1}{1-\gamma}\mathbf{e}_{r_{\nearrow}})\in\Fndop(r_{\searrow})\setminus\phi\cdot\Fndop(\ell_{\swarrow}) , r ↘ r_{\searrow} has “strictly more options” and therefore fulfills Proposition 6.6 ’s stronger condition.

Proposition 6.6 is shown using the fact that ϕ \phi injectively maps 𝒟 \mathcal{D} under which r ↘ r_{\searrow} has less Power 𝒟 \text{{Power}}_{\mathcal{D}} , to distributions ϕ ⋅ 𝒟 \phi\cdot\mathcal{D} which agree with the intuition that r ↘ r_{\searrow} offers more control. Therefore, at least half of each orbit must agree, and r ↘ r_{\searrow} never “loses the Power vote” against ℓ ↙ \ell_{\swarrow} . 6 6 6 Proposition 6.6 also proves that in general, ∅ \varnothing has less Power than ℓ ↙ \ell_{\swarrow} and r ↘ r_{\searrow} . However, this does not prove that most distributions 𝒟 \mathcal{D} satisfy the joint inequality Power 𝒟 ​ ( ∅ , γ ) ≤ Power 𝒟 ​ ( ℓ ↙ , γ ) ≤ Power 𝒟 ​ ( r ↘ , γ ) \text{{Power}}_{\mathcal{D}}(\varnothing,\gamma)\leq\text{{Power}}_{\mathcal{D}}(\ell_{\swarrow},\gamma)\leq\text{{Power}}_{\mathcal{D}}(r_{\searrow},\gamma) . This only proves that these inequalities hold pairwise for most 𝒟 \mathcal{D} . The orbit elements 𝒟 \mathcal{D} which agree that ∅ \varnothing has less Power 𝒟 \text{{Power}}_{\mathcal{D}} than ℓ ↙ \ell_{\swarrow} need not be the same elements 𝒟 ′ \mathcal{D}^{\prime} which agree that ℓ ↙ \ell_{\swarrow} has less Power 𝒟 ′ \text{{Power}}_{\mathcal{D}^{\prime}} than r ↘ r_{\searrow} .

### 6.1 Keeping options open tends to be Power -seeking and tends to be optimal

Certain symmetries in the mdp structure ensure that, compared to left , going right tends to be optimal and to be Power -seeking. Intuitively, by going right , the agent has “strictly more choices.” Proposition 6.9 will formalize this tendency.

###### Definition 6.7 (Equivalent actions) .

Actions a 1 a_{1} and a 2 a_{2} are equivalent at state s s (written a 1 ≡ s a 2 a_{1}\equiv_{s}a_{2} ) if they induce the same transition probabilities: T ⁡ ( s , a 1 ) = T ⁡ ( s , a 2 ) T(s,a_{1})=T(s,a_{2}) .

The agent can reach states in { r ⊳ , r ↗ , r ↘ } \{r_{\triangleright},r_{\nearrow},r_{\searrow}\} by taking actions equivalent to right at state ⋆ \star .

###### Definition 6.8 (States reachable after taking an action) .

Reach ​ ( s , a ) \text{{Reach}}\left(s,a\right) is the set of states reachable with positive probability after taking the action a a in state s s .

###### Proposition 6.9 (Keeping options open tends to be Power -seeking and tends to be optimal) .

Suppose F a ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ) F_{a}\coloneqq\Fop(s\mid\pi(s)=a) contains a copy of F a ′ ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ′ ) F_{a^{\prime}}\coloneqq\Fop(s\mid\pi(s)=a^{\prime}) via ϕ \phi . 1. If s ∉ Reach ​ ( s , a ′ ) s\not\in\text{{Reach}}\left(s,a^{\prime}\right) , then ∀ γ ∈ [ 0 , 1 ] : 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] ≥ most : 𝔇 bound 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right]\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right] .

2. If s s can only reach the states of Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) \text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) by taking actions equivalent to a ′ a^{\prime} or a a at state s s , then ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( s , a , γ ) ≥ most : 𝔇 any ℙ 𝒟 any ( s , a ′ , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right) .

If ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty, then ∀ γ ∈ ( 0 , 1 ) \forall\gamma\in(0,1) , the converse ≤ most \leq_{\text{{most}}} statements do not hold.

We check the conditions of Proposition 6.9 . s ≔ ⋆ s\coloneqq\star , a ′ ≔ left a^{\prime}\coloneqq\texttt{left} , a ≔ right a\coloneqq\texttt{right} . Figure 6 shows that ⋆ ∉ Reach ​ ( ⋆ , left ) \star\not\in\text{{Reach}}\left(\star,\texttt{left}\right) and that ⋆ \star can only reach { ℓ ⊲ , ℓ ↖ , ℓ ↙ } ∪ { r ⊳ , r ↗ , r ↘ } \{\ell_{\triangleleft},\ell_{\nwarrow},\ell_{\swarrow}\}\cup\{r_{\triangleright},r_{\nearrow},r_{\searrow}\} when the agent immediately takes actions equivalent to left or right . ℱ ( ⋆ ∣ π ( ⋆ ) = right ) \Fop(\star\mid\pi(\star)=\texttt{right}) contains a copy of ℱ ( ⋆ ∣ π ( ⋆ ) = left ) \Fop(\star\mid\pi(\star)=\texttt{left}) via ϕ \phi . Furthermore, ℱ nd ⁡ ( ⋆ ) ∩ { 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 ​ 𝐞 r ↘ + γ 3 1 − γ ​ 𝐞 r ↗ , 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } = { 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } \Fndop(\star)\cap\{\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\gamma^{2}\mathbf{e}_{r_{\searrow}}+\frac{\gamma^{3}}{1-\gamma}\mathbf{e}_{r_{\nearrow}},\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\}=\{\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} is non-empty, and so all conditions are met.

For any γ ∈ [ 0 , 1 ] \gamma\in[0,1] and 𝒟 \mathcal{D} such that ℙ 𝒟 ( ⋆ , left , γ ) > ℙ 𝒟 ( ⋆ , right , γ ) \Prb\nolimits_{\mathcal{D}}\left(\star,\texttt{left},\gamma\right)>\Prb\nolimits_{\mathcal{D}}\left(\star,\texttt{right},\gamma\right) , environmental symmetry ensures that ℙ ϕ ⋅ 𝒟 ( ⋆ , left , γ ) < ℙ ϕ ⋅ 𝒟 ( ⋆ , right , γ ) \Prb\nolimits_{\phi\cdot\mathcal{D}}\left(\star,\texttt{left},\gamma\right)<\Prb\nolimits_{\phi\cdot\mathcal{D}}\left(\star,\texttt{right},\gamma\right) . A similar statement holds for Power .

### 6.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles

Proposition 6.6 and Proposition 6.9 are powerful because they apply to all γ ∈ [ 0 , 1 ] \gamma\in[0,1] , but they can only be applied given hard-to-satisfy environmental symmetries. In contrast, Proposition 6.12 and Theorem 6.13 apply to many structured environments common to rl .

Starting from ⋆ \star , consider the cycles which the agent can reach. Recurrent state distributions ( rsd s) generalize deterministic graphical cycles to potentially stochastic environments. Rsd s simply record how often the agent tends to visit a state in the limit of infinitely many time steps.

###### Definition 6.10 (Recurrent state distributions [ Puterman, 2014 ] ) .

The recurrent state distributions which can be induced from state s s are RSD ​ ( s ) ≔ { lim γ → 1 ( 1 − γ ) ​ 𝐟 π , s ​ ( γ ) ∣ π ∈ Π } \text{{RSD}}\left(s\right)\coloneqq\left\{\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\mid\pi\in\Pi\right\} . RSD nd ​ ( s ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) is the set of rsd s which strictly maximize average reward for some reward function.

As suggested by fig. 3 , RSD ​ ( ⋆ ) = { 𝐞 ℓ ↙ , 1 2 ​ ( 𝐞 ℓ ↙ + 𝐞 ℓ ↖ ) , 𝐞 ∅ , 𝐞 r ↗ , 1 2 ​ ( 𝐞 r ↗ + 𝐞 r ↘ ) , 𝐞 r ↘ } \text{{RSD}}\left(\star\right)=\{\mathbf{e}_{\ell_{\swarrow}},\frac{1}{2}(\mathbf{e}_{\ell_{\swarrow}}+\mathbf{e}_{\ell_{\nwarrow}}),\mathbf{e}_{\varnothing},\mathbf{e}_{r_{\nearrow}},\frac{1}{2}(\mathbf{e}_{r_{\nearrow}}+\mathbf{e}_{r_{\searrow}}),\mathbf{e}_{r_{\searrow}}\} . As discussed in section 3 , 1 2 ​ ( 𝐞 r ↗ + 𝐞 r ↘ ) \frac{1}{2}(\mathbf{e}_{r_{\nearrow}}+\mathbf{e}_{r_{\searrow}}) is dominated: Alternating between r ↗ r_{\nearrow} and r ↘ r_{\searrow} is never strictly better than choosing one or the other.

A reward function’s optimal policies can vary with the discount rate. When γ = 1 \gamma=1 , optimal policies ignore transient reward because average reward is the dominant consideration.

###### Definition 6.11 (Average-optimal policies) .

The average-optimal policy set for reward function R R is Π avg ​ ( R ) ≔ { π ∈ Π ∣ ∀ s ∈ 𝒮 : 𝐝 π , s ∈ arg ​ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 } \Pi^{\text{avg}}\left(R\right)\coloneqq\left\{\pi\in\Pi\mid\forall s\in\mathcal{S}\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\pi,s}\in\argmax_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right\} (the policies which induce optimal rsd s at all states). For D ⊆ RSD ​ ( s ) D\subseteq\text{{RSD}}\left(s\right) , the average optimality probability is ℙ 𝒟 any ( D , average ) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐝 π , s ∈ D : π ∈ Π avg ​ ( R ) ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right)\coloneqq\Prb\nolimits_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{d}^{\pi,s}\in D\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{\text{avg}}\left(R\right)\right) .

Average-optimal policies maximize average reward. Average reward is governed by rsd access. For example, r ↘ r_{\searrow} has “more” rsd s than ∅ \varnothing ; therefore, r ↘ r_{\searrow} usually has greater Power when γ = 1 \gamma=1 .

###### Proposition 6.12 (When γ = 1 \gamma=1 , rsd s control Power ) .

If RSD ​ ( s ) \text{{RSD}}\left(s\right) contains a copy of RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right) via ϕ \phi , then Power 𝒟 bound ( s , 1 ) ≥ most Power 𝒟 bound ( s ′ , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right) . If RSD nd ​ ( s ) ∖ ϕ ⋅ RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\phi\cdot\text{{RSD}}{\text{\textsubscript{nd}}}(s^{\prime}) is non-empty, then the converse ≤ most \leq_{\text{{most}}} statement does not hold.

We check that both conditions of Proposition 6.12 are satisfied when s ′ ≔ ∅ , s ≔ r ↘ s^{\prime}\coloneqq\varnothing,s\coloneqq r_{\searrow} , and the involution ϕ \phi swaps ∅ \varnothing and r ↘ r_{\searrow} . Formally, ϕ ⋅ RSD nd ​ ( ∅ ) = ϕ ⋅ { 𝐞 ∅ } = { 𝐞 r ↘ } ⊊ { 𝐞 r ↘ , 𝐞 r ↗ } = RSD nd ​ ( r ↘ ) ⊆ [ r ↘ ] \phi\cdot\text{{RSD}}{\text{\textsubscript{nd}}}\left(\varnothing\right)=\phi\cdot\{\mathbf{e}_{\varnothing}\}=\{\mathbf{e}_{r_{\searrow}}\}\subsetneq\{\mathbf{e}_{r_{\searrow}},\mathbf{e}_{r_{\nearrow}}\}=\text{{RSD}}{\text{\textsubscript{nd}}}(r_{\searrow})\subseteq[r_{\searrow}] . The conditions are satisfied.

Informally, states with more rsd s generally have more Power at γ = 1 \gamma=1 , no matter their transient dynamics. Furthermore, average-optimal policies are more likely to end up in larger sets of rsd s than in smaller ones. Thus, average-optimal policies tend to navigate towards parts of the state space which contain more rsd s.

###### Theorem 6.13 (Average-optimal policies tend to end up in “larger” sets of rsd s) .

Let D , D ′ ⊆ RSD ​ ( s ) D,D^{\prime}\subseteq\text{{RSD}}\left(s\right) . Suppose that D D contains a copy of D ′ D^{\prime} via ϕ \phi , and that the sets D ∪ D ′ D\cup D^{\prime} and RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\left(D^{\prime}\cup D\right) have pairwise orthogonal vector elements (i.e., pairwise disjoint vector support). Then ℙ 𝒟 any ( D , average ) ≥ most ℙ 𝒟 any ( D ′ , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right) . If RSD nd ​ ( s ) ∩ ( D ∖ ϕ ⋅ D ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap\left(D\setminus\phi\cdot D^{\prime}\right) is non-empty, the converse ≤ most \leq_{\text{{most}}} statement does not hold.

###### Corollary 6.14 (Average-optimal policies tend not to end up in any given 1-cycle) .

Suppose 𝐞 s x , 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s_{x}},\mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) are distinct. Then ℙ 𝒟 any ( RSD ( s ) ∖ { 𝐞 s x } , average ) ≥ most ℙ 𝒟 any ( { 𝐞 s x } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus\{\mathbf{e}_{s_{x}}\},\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{s_{x}}\},\text{average}\right) . If there is a third 𝐞 s ′′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime\prime}}\in\text{{RSD}}\left(s\right) , the converse ≤ most \leq_{\text{{most}}} statement does not hold.

Figure 7 illustrates that 𝐞 ∅ , 𝐞 r ↘ , 𝐞 r ↗ ∈ RSD ​ ( ⋆ ) \mathbf{e}_{\varnothing},\mathbf{e}_{r_{\searrow}},\mathbf{e}_{r_{\nearrow}}\in\text{{RSD}}\left(\star\right) . Thus, both conclusions of Corollary 6.14 hold: ℙ 𝒟 any ( RSD ( ⋆ ) ∖ { 𝐞 ∅ } , average ) ≥ most ℙ 𝒟 any ( { 𝐞 ∅ } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(\star\right)\setminus\{\mathbf{e}_{\varnothing}\},\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{\varnothing}\},\text{average}\right) and ℙ 𝒟 any ( RSD ( ⋆ ) ∖ { 𝐞 ∅ } , average ) ≰ most ℙ 𝒟 any ( { 𝐞 ∅ } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(\star\right)\setminus\{\mathbf{e}_{\varnothing}\},\text{average}\right)\not\leq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{\varnothing}\},\text{average}\right) . In other words, average-optimal policies tend to end up in rsd s besides ∅ \varnothing . Since ∅ \varnothing is a terminal state, it cannot reach other rsd s. Since average-optimal policies tend to end up in other rsd s, average-optimal policies tend to avoid ∅ \varnothing .

This section’s results prove the γ = 1 \gamma=1 case. Lemma 5.3 shows that Power is continuous at γ = 1 \gamma=1 . Therefore, if an action is strictly Power 𝒟 \text{{Power}}_{\mathcal{D}} -seeking when γ = 1 \gamma=1 , it is strictly Power 𝒟 \text{{Power}}_{\mathcal{D}} -seeking at discount rates sufficiently close to 1. Future work may connect average optimality probability to optimality probability at γ ≈ 1 \gamma\approx 1 .

Lastly, our key results apply to all degenerate reward function distributions. Therefore, these results apply not just to distributions over reward functions, but to individual reward functions.

### 6.3 How to reason about other environments

Consider an embodied navigation task through a room with a vase. Proposition 6.9 suggests that optimal policies tend to avoid immediately breaking the vase, since doing so would strictly decrease available options.

Theorem 6.13 dictates where average-optimal agents tend to end up, but not what actions they tend to take in order to reach their rsd s. Therefore, care is needed. In appendix B , fig. 10 demonstrates an environment in which seeking Power is a detour for most reward functions (since optimality probability measures “median” optimal value, while Power is a function of mean optimal value). However, suppose the agent confronts a fork in the road: Actions a a and a ′ a^{\prime} lead to two disjoint sets of rsd s D a D_{a} and D a ′ D_{a^{\prime}} , such that D a D_{a} contains a copy of D a ′ D_{a^{\prime}} . Theorem 6.13 shows that a a will tend to be average-optimal over a ′ a^{\prime} , and Proposition 6.12 shows that a a will tend to be Power -seeking compared to a ′ a^{\prime} . Such forks seem reasonably common in environments with irreversible actions.