Optimal Policies Tend to Seek Power — part 2/16

Cleaned-text code points [15750, 31329). Bibliography omitted here; source.md includes it.

At discount rate γ \gamma and at state s s , the optimality probability of action a a is ℙ 𝒟 any ( s , a , γ ) ≔ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\coloneqq\Prb\nolimits_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) .

Optimality probability may seem hard to reason about. It’s hard enough to compute an optimal policy for a single reward function, let alone for uncountably many! But consider any 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} distributing reward independently and identically across states. When γ = 0 \gamma=0 , optimal policies greedily maximize next-state reward. At ⋆ \star , identically distributed reward means ℓ ⊲ \ell_{\triangleleft} and r ⊳ r_{\triangleright} have an equal probability of having maximal next-state reward. Therefore, ℙ 𝒟 X ​ - iid ( ⋆ , left , 0 ) = ℙ 𝒟 X ​ - iid ( ⋆ , right , 0 ) \Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(\star,\texttt{left},0\right)=\Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(\star,\texttt{right},0\right) . This is not a proof, but such statements are provable.

With 𝒟 ℓ ⊲ \mathcal{D}_{\ell_{\triangleleft}} being the degenerate distribution on reward function 𝐞 ℓ ⊲ \mathbf{e}_{\ell_{\triangleleft}} , ℙ 𝒟 ℓ ⊲ ( ⋆ , left , 1 2 ) = 1 > 0 = ℙ 𝒟 ℓ ⊲ ( ⋆ , right , 1 2 ) \Prb\nolimits_{\mathcal{D}_{\ell_{\triangleleft}}}\left(\star,\texttt{left},\frac{1}{2}\right)=1>0=\Prb\nolimits_{\mathcal{D}_{\ell_{\triangleleft}}}\left(\star,\texttt{right},\frac{1}{2}\right) . Similarly, ℙ 𝒟 r ⊳ ( ⋆ , left , 1 2 ) = 0 < 1 = ℙ 𝒟 r ⊳ ( ⋆ , right , 1 2 ) \Prb\nolimits_{\mathcal{D}_{r_{\triangleright}}}\left(\star,\texttt{left},\frac{1}{2}\right)=0<1=\Prb\nolimits_{\mathcal{D}_{r_{\triangleright}}}\left(\star,\texttt{right},\frac{1}{2}\right) . Therefore, “what do optimal policies ‘tend’ to look like?” seems to depend on one’s prior beliefs. But in fig. 1 , we claimed that left is optimal for fewer reward functions than right is. The claim is meaningful and true, but we will return to it in section 6 .

## 5 Some states give the agent more control over the future

The agent has more options at ℓ ↙ \ell_{\swarrow} than at the inescapable terminal state ∅ \varnothing . Furthermore, since r ↗ r_{\nearrow} has a loop, the agent has more options at r ↘ r_{\searrow} than at ℓ ↙ \ell_{\swarrow} . A glance at fig. 3 leads us to intuit that r ↘ r_{\searrow} affords the agent more power than ∅ \varnothing .

What is power? Philosophers have many answers. One prominent answer is the dispositional view: Power is the ability to achieve a range of goals [ Sattarov, 2019 ] . In an mdp , the optimal value function V R ∗ ​ ( s , γ ) V^{*}_{R}\left(s,\gamma\right) captures the agent’s ability to “achieve the goal” R R . Therefore, average optimal value captures the agent’s ability to achieve a range of goals 𝒟 bound \mathcal{D}_{\text{bound}} . 2 2 2 𝒟 bound \mathcal{D}_{\text{bound}} ’s bounded support ensures that 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) ] \opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)\right] is well-defined.

###### Definition 5.1 (Average optimal value) .

The average optimal value 3 3 3 Appendix C relaxes the optimality assumption. at state s s and discount rate γ ∈ ( 0 , 1 ) \gamma\in(0,1) is V 𝒟 bound ∗ ​ ( s , γ ) ≔ 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) ] = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 ] . V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\coloneqq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)\right]=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\mathbf{f}(\gamma)^{\top}\mathbf{r}\right].

Figure 3 shows the pleasing result that for the max-entropy distribution, r ↘ r_{\searrow} has greater average optimal value than ∅ \varnothing . However, average optimal value has a few problems as a measure of power. The agent is rewarded for its initial presence at state s s (over which it has no control), and because ‖ 𝐟 ⁡ ( γ ) ‖ 1 = 1 1 − γ \left\lVert\mathbf{f}(\gamma)\right\rVert_{1}=\frac{1}{1-\gamma} ( Proposition E.3 ) diverges as γ → 1 \gamma\to 1 , lim γ → 1 V 𝒟 bound ∗ ​ ( s , γ ) \lim_{\gamma\to 1}V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) tends to diverge. Definition 5.2 fixes these issues in order to better measure the agent’s control over the future.

###### Definition 5.2 ( Power ) .

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) ≔ 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] = 1 − γ γ ​ 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) − R ⁡ ( s ) ] . \displaystyle\coloneqq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right]=\frac{1-\gamma}{\gamma}\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right]. (2)

Power has nice formal properties.

###### Lemma 5.3 (Continuity of Power ) .

Power 𝒟 bound ​ ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) is Lipschitz continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] .

###### Proposition 5.4 (Maximal Power ) .

Power 𝒟 bound ​ ( s , γ ) ≤ 𝔼 R ∼ 𝒟 bound [ max s ∈ 𝒮 ⁡ R ⁡ ( s ) ] \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\leq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{s\in\mathcal{S}}R(s)\right] , with equality if s s can deterministically reach all states in one step and all states are 1-cycles.

###### Proposition 5.5 ( Power is smooth across reversible dynamics) .

Let 𝒟 bound \mathcal{D}_{\text{bound}} be bounded [ b , c ] [b,c] . Suppose s s and s ′ s^{\prime} can both reach each other in one step with probability 1. | Power 𝒟 bound ​ ( s , γ ) − Power 𝒟 bound ​ ( s ′ , γ ) | ≤ ( c − b ) ​ ( 1 − γ ) . \displaystyle\big|\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\big|\leq(c-b)(1-\gamma). (3)

We consider power-seeking to be relative. Intuitively, “live and keep some options open” seeks more power than “die and keep no options open.” Similarly, “maximize open options” seeks more power than “don’t maximize open options.”

###### Definition 5.6 ( Power -seeking actions) .

At state s s and discount rate γ ∈ [ 0 , 1 ] \gamma\in[0,1] , action a a seeks more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than a ′ a^{\prime} when 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ​ ( s a , γ ) ] ≥ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ​ ( s a ′ , γ ) ] \opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s_{a},\gamma)\right]\geq\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s_{a^{\prime}},\gamma)\right] .

Power is sensitive to choice of distribution. 𝒟 ℓ ↙ \mathcal{D}_{\ell_{\swarrow}} gives maximal Power 𝒟 ℓ ↙ \text{{Power}}_{\mathcal{D}_{\ell_{\swarrow}}} to ℓ ↙ \ell_{\swarrow} . 𝒟 r ↘ \mathcal{D}_{r_{\searrow}} assigns maximal Power 𝒟 r ↘ \text{{Power}}_{\mathcal{D}_{r_{\searrow}}} to r ↘ r_{\searrow} . 𝒟 ∅ \mathcal{D}_{\varnothing} even gives maximal Power 𝒟 ∅ \text{{Power}}_{\mathcal{D}_{\varnothing}} to ∅ \varnothing ! In what sense does ∅ \varnothing have “less Power ” than r ↘ r_{\searrow} , and in what sense does right “tend to seek Power ” compared to left ?

## 6 Certain environmental symmetries produce power-seeking tendencies

Proposition 6.6 proves that for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] and for most distributions 𝒟 \mathcal{D} , Power 𝒟 ​ ( ℓ ↙ , γ ) ≤ Power 𝒟 ​ ( r ↘ , γ ) \text{{Power}}_{\mathcal{D}}(\ell_{\swarrow},\gamma)\leq\text{{Power}}_{\mathcal{D}}(r_{\searrow},\gamma) . But first, we explore why this must be true.

ℱ ⁡ ( ℓ ↙ ) = { 1 1 − γ ​ 𝐞 ℓ ↙ , 1 1 − γ 2 ​ ( 𝐞 ℓ ↙ + γ ​ 𝐞 ℓ ↖ ) } \Fop(\ell_{\swarrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{\ell_{\swarrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{\ell_{\swarrow}}+\gamma\mathbf{e}_{\ell_{\nwarrow}})\} and ℱ ⁡ ( r ↘ ) = { 1 1 − γ ​ 𝐞 r ↘ , 1 1 − γ 2 ​ ( 𝐞 r ↘ + γ ​ 𝐞 r ↗ ) , 𝐞 r ↘ + γ 1 − γ ​ 𝐞 r ↗ } \Fop(r_{\searrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{r_{\searrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{r_{\searrow}}+\gamma\mathbf{e}_{r_{\nearrow}}),\mathbf{e}_{r_{\searrow}}+\frac{\gamma}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} . These two sets look awfully similar. ℱ ⁡ ( ℓ ↙ ) \Fop(\ell_{\swarrow}) is a “subset” of ℱ ⁡ ( r ↘ ) \Fop(r_{\searrow}) , only with “different states.” Figure 4 demonstrates a state permutation ϕ \phi which embeds ℱ ⁡ ( ℓ ↙ ) \Fop(\ell_{\swarrow}) into ℱ ⁡ ( r ↘ ) \Fop(r_{\searrow}) .

###### Definition 6.1 (Similarity of vector sets) .

Consider state permutation ϕ ∈ S | 𝒮 | \phi\in S_{\left|\mathcal{S}\right|} inducing an | 𝒮 | × | 𝒮 | \left|\mathcal{S}\right|\times\left|\mathcal{S}\right| permutation matrix 𝐏 ϕ \mathbf{P}_{\phi} in row representation: ( 𝐏 ϕ ) i ​ j = 1 (\mathbf{P}_{\phi})_{ij}=1 if i = ϕ ⁡ ( j ) i=\phi(j) and 0 0 otherwise. For X ⊆ ℝ | 𝒮 | X\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} , ϕ ⋅ X ≔ { 𝐏 ϕ ​ 𝐱 ∣ 𝐱 ∈ X } \phi\cdot X\coloneqq\left\{\mathbf{P}_{\phi}\mathbf{x}\mid\mathbf{x}\in X\right\} . X ′ ⊆ ℝ | 𝒮 | X^{\prime}\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} is similar to X X when ∃ ϕ : ϕ ⋅ X ′ = X \exists\phi\mathrel{\mathop{\ordinarycolon}}\phi\cdot X^{\prime}=X . ϕ \phi is an involution if ϕ = ϕ − 1 \phi=\phi^{-1} (it either transposes states, or fixes them in place). X X contains a copy of X ′ X^{\prime} when X ′ X^{\prime} is similar to a subset of X X via an involution ϕ \phi .

###### Definition 6.2 (Similarity of vector function sets) .

Let I ⊆ ℝ I\subseteq\mathbb{R} . If F , F ′ F,F^{\prime} are sets of functions I ↦ ℝ | 𝒮 | I\mapsto\mathbb{R}^{\left|\mathcal{S}\right|} , F F is (pointwise) similar to F ′ F^{\prime} when ∃ ϕ : ∀ γ ∈ I : { 𝐏 ϕ ​ 𝐟 ​ ( γ ) ∣ 𝐟 ∈ F } = { 𝐟 ′ ​ ( γ ) ∣ 𝐟 ′ ∈ F ′ } \exists\phi\mathrel{\mathop{\ordinarycolon}}\forall\gamma\in I\mathrel{\mathop{\ordinarycolon}}\{\mathbf{P}_{\phi}\mathbf{f}(\gamma)\mid\mathbf{f}\in F\}=\{\mathbf{f}^{\prime}(\gamma)\mid\mathbf{f}^{\prime}\in F^{\prime}\} .

Consider a reward function R ′ R^{\prime} assigning 1 reward to ℓ ↙ \ell_{\swarrow} and ℓ ↖ \ell_{\nwarrow} and 0 elsewhere. R ′ R^{\prime} assigns more optimal value to ℓ ↙ \ell_{\swarrow} than to r ↘ r_{\searrow} : V R ′ ∗ ​ ( ℓ ↙ , γ ) = 1 1 − γ > 0 = V R ′ ∗ ​ ( r ↘ , γ ) V^{*}_{R^{\prime}}(\ell_{\swarrow},\gamma)=\frac{1}{1-\gamma}>0=V^{*}_{R^{\prime}}(r_{\searrow},\gamma) . Considering ϕ \phi from fig. 4 , ϕ ⋅ R ′ \phi\cdot R^{\prime} assigns 1 reward to r ↘ r_{\searrow} and r ↗ r_{\nearrow} and 0 elsewhere. Therefore, ϕ ⋅ R ′ \phi\cdot R^{\prime} assigns more optimal value to r ↘ r_{\searrow} than to ℓ ↙ \ell_{\swarrow} : V ϕ ⋅ R ′ ∗ ​ ( ℓ ↙ , γ ) = 0 < 1 1 − γ = V ϕ ⋅ R ′ ∗ ​ ( r ↘ , γ ) V^{*}_{\phi\cdot R^{\prime}}(\ell_{\swarrow},\gamma)=0<\frac{1}{1-\gamma}=V^{*}_{\phi\cdot R^{\prime}}(r_{\searrow},\gamma) . Remarkably, this ϕ \phi has the property that for any R R which assigns ℓ ↙ \ell_{\swarrow} greater optimal value than r ↘ r_{\searrow} ( i . e ., V R ∗ ​ ( ℓ ↙ , γ ) > V R ∗ ​ ( r ↘ , γ ) V^{*}_{R}(\ell_{\swarrow},\gamma)>V^{*}_{R}(r_{\searrow},\gamma) ), the opposite holds for the permuted ϕ ⋅ R \phi\cdot R : V ϕ ⋅ R ∗ ​ ( ℓ ↙ , γ ) < V ϕ ⋅ R ∗ ​ ( r ↘ , γ ) V^{*}_{\phi\cdot R}(\ell_{\swarrow},\gamma)<V^{*}_{\phi\cdot R}(r_{\searrow},\gamma) .

We can permute reward functions, but we can also permute reward function distributions. Permuted distributions simply permute which states get which rewards.

###### Definition 6.3 (Pushforward distribution of a permutation) .

Let ϕ ∈ S | 𝒮 | \phi\in S_{\left|\mathcal{S}\right|} . ϕ ⋅ 𝒟 any \phi\cdot\mathcal{D}_{\text{any}} is the pushforward distribution induced by applying the random vector f ⁡ ( 𝐫 ) ≔ 𝐏 ϕ ​ 𝐫 f(\mathbf{r})\coloneqq\mathbf{P}_{\phi}\mathbf{r} to 𝒟 any \mathcal{D}_{\text{any}} .

###### Definition 6.4 (Orbit of a probability distribution) .

The orbit of 𝒟 any \mathcal{D}_{\text{any}} under the symmetric group S | 𝒮 | S_{\left|\mathcal{S}\right|} is S | 𝒮 | ⋅ 𝒟 any ≔ { ϕ ⋅ 𝒟 any ∣ ϕ ∈ S | 𝒮 | } S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}_{\text{any}}\coloneqq\{\phi\cdot\mathcal{D}_{\text{any}}\mid\phi\in S_{\left|\mathcal{S}\right|}\} .

For example, the orbit of a degenerate state indicator distribution 𝒟 s \mathcal{D}_{s} is S | 𝒮 | ⋅ 𝒟 s = { 𝒟 s ′ ∣ s ′ ∈ 𝒮 } S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}_{s}=\{\mathcal{D}_{s^{\prime}}\mid s^{\prime}\in\mathcal{S}\} , and fig. 5 shows the orbit of a 2D Gaussian distribution.

Consider again the involution ϕ \phi of fig. 4 . For every 𝒟 bound \mathcal{D}_{\text{bound}} for which ℓ ↙ \ell_{\swarrow} has more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than r ↘ r_{\searrow} , ℓ ↙ \ell_{\swarrow} has less Power ϕ ⋅ 𝒟 bound \text{{Power}}_{\phi\cdot\mathcal{D}_{\text{bound}}} than r ↘ r_{\searrow} . This fact is not obvious—it is shown by the proof of Lemma E.24 .

Imagine 𝒟 bound \mathcal{D}_{\text{bound}} ’s orbit elements “voting” whether ℓ ↙ \ell_{\swarrow} or r ↘ r_{\searrow} has strictly more Power . Proposition 6.6 will show that r ↘ r_{\searrow} can’t lose the “vote” for the orbit of any bounded reward function distribution. Definition 6.5 formalizes this ‘‘voting’’ notion. 4 4 4 The voting analogy and the “most” descriptor imply that we have endowed each orbit with the counting measure. However, a priori , we might expect that some orbit elements are more empirically likely to be specified than other orbit elements. See section 7 for more on this point.

###### Definition 6.5 (Inequalities which hold for most probability distributions) .

Let f 1 , f 2 : Δ ⁡ ( ℝ | 𝒮 | ) → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Delta(\mathbb{R}^{\left|\mathcal{S}\right|})\to\mathbb{R} be functions from reward function distributions to real numbers and let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) be closed under permutation. We write f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) 5 5 5 We write f 1 ( 𝒟 ) ≥ most f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}}f_{2}(\mathcal{D}) when 𝔇 \mathfrak{D} is clear from context. when, for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , the following cardinality inequality holds: | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) } | ≥ | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) < f 2 ​ ( 𝒟 ′ ) } | . \left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime})\}\right|\geq\left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})<f_{2}(\mathcal{D}^{\prime})\}\right|. (4)

###### Proposition 6.6 (States with “more options” have more Power ) .

If ℱ ⁡ ( s ) \Fop(s) contains a copy of ℱ nd ⁡ ( s ′ ) \Fndop(s^{\prime}) via ϕ \phi , then ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s , γ ) ≥ most Power 𝒟 bound ( s ′ , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s,\gamma)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s^{\prime},\gamma) . If ℱ nd ⁡ ( s ) ∖ ϕ ⋅ ℱ nd ⁡ ( s ′ ) \Fndop(s)\setminus\phi\cdot\Fndop(s^{\prime}) is non-empty, then for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the converse ≤ most \leq_{\text{{most}}} statement does not hold.