Optimal Policies Tend to Seek Power — part 14/16

Cleaned-text code points [201642, 217561). Bibliography omitted here; source.md includes it.

Suppose γ = 0 \gamma=0 or γ = 1 \gamma=1 . Consider any sequence ( γ n ) n = 1 ∞ \left(\gamma_{n}\right)_{n=1}^{\infty} converging to γ \gamma , and let 𝒟 any \mathcal{D}_{\text{any}} induce probability measure F F . ℙ 𝒟 any ( F a , γ ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) ≔ lim γ ∗ → γ ℙ 𝒟 any ( F a , γ ∗ ) \displaystyle\coloneqq\lim_{\gamma^{*}\to\gamma}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma^{*}\right) (173) = lim γ ∗ → γ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ ∗ ) : π ∗ ​ ( s ) = a ) \displaystyle=\lim_{\gamma^{*}\to\gamma}\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma^{*}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) (174) = lim n → ∞ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a ) \displaystyle=\lim_{n\to\infty}\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) (175) = lim n → ∞ ∫ ℝ 𝒮 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\lim_{n\to\infty}\int_{\mathbb{R}^{\mathcal{S}}}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (176) = ∫ ℝ 𝒮 lim n → ∞ 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\int_{\mathbb{R}^{\mathcal{S}}}\lim_{n\to\infty}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (177) = ∫ ℝ 𝒮 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\int_{\mathbb{R}^{\mathcal{S}}}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (178) ≕ ℙ 𝒟 any ( s , a , γ ) . \displaystyle\eqqcolon\Prb_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right). (179)

Equation 174 follows by eq. 172 . for γ ∗ ∈ [ 0 , 1 ] \gamma^{*}\in[0,1] , let f γ ∗ ( R ) ≔ 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ ∗ ) : π ∗ ​ ( s ) = a f_{\gamma^{*}}(R)\coloneqq\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma^{*}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a} . For each R ∈ ℝ 𝒮 R\in\mathbb{R}^{\mathcal{S}} , Lemma E.33 exists γ x ≈ γ \gamma_{x}\approx\gamma such that for all intermediate γ x ′ \gamma_{x}^{\prime} between γ x \gamma_{x} and γ \gamma , Π ∗ ​ ( R , γ x ′ ) = Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma_{x}^{\prime}\right)=\Pi^{*}\left(R,\gamma\right) . Since γ n → γ \gamma_{n}\to\gamma , this means that ( f γ n ) n = 1 ∞ \left(f_{\gamma_{n}}\right)_{n=1}^{\infty} converges pointwise to f γ f_{\gamma} . Furthermore, ∀ n ∈ ℕ , R ∈ ℝ 𝒮 : | f γ n ​ ( R ) | ≤ 1 \forall n\in\mathbb{N},R\in\mathbb{R}^{\mathcal{S}}\mathrel{\mathop{\ordinarycolon}}\left|f_{\gamma_{n}}(R)\right|\leq 1 by definition. Therefore, eq. 177 follows by Lebesgue’s dominated convergence theorem. ∎

See 6.9

###### Proof.

Note that by definition 3.3 , F a ′ ​ ( 0 ) = { 𝐞 s } = F a ​ ( 0 ) F_{a^{\prime}}(0)=\left\{\mathbf{e}_{s}\right\}=F_{a}(0) . Since ϕ ⋅ F a ′ ⊆ F a \phi\cdot F_{a^{\prime}}\subseteq F_{a} , in particular we have ϕ ⋅ F a ′ ​ ( 0 ) = { 𝐏 ϕ ​ 𝐞 s } ⊆ { 𝐞 s } = F a ​ ( 0 ) \phi\cdot F_{a^{\prime}}(0)=\left\{\mathbf{P}_{\phi}\mathbf{e}_{s}\right\}\subseteq\left\{\mathbf{e}_{s}\right\}=F_{a}(0) , and so ϕ ⁡ ( s ) = s \phi(s)=s .

Item 1 . For state probability distribution Δ s ∈ Δ ⁡ ( 𝒮 ) \Delta_{s}\in\Delta(\mathcal{S}) , let F Δ s ∗ ≔ { 𝔼 s ′ ∼ Δ s [ 𝐟 π , s ′ ] ∣ π ∈ Π } F^{*}_{\Delta_{s}}\coloneqq\left\{\opE_{s^{\prime}\sim\Delta_{s}}\left[\mathbf{f}^{\pi,s^{\prime}}\right]\mid\pi\in\Pi\right\} . Unless otherwise stated, we treat γ \gamma as a variable in this item; we apply element-wise vector addition, constant multiplication, and variable multiplication via the conventions outlined in definition E.14 . F a ′ \displaystyle F_{a^{\prime}} = { 𝐞 s + γ ​ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ∣ π ∈ Π : π ⁡ ( s ) = a ′ } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi\mathrel{\mathop{\ordinarycolon}}\pi(s)=a^{\prime}\right\} (180) = { 𝐞 s + γ ​ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ∣ π ∈ Π } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi\right\} (181) = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ . \displaystyle=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})}. (182)

Equation 180 follows by definition 3.3 , since each 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) has an initial term of 𝐞 s \mathbf{e}_{s} . Equation 181 follows because s ∉ Reach ​ ( s , a ′ ) s\not\in\text{{Reach}}\left(s,a^{\prime}\right) , and so for all s a ′ ∈ supp ⁡ ( T ⁡ ( s , a ′ ) ) s_{a^{\prime}}\in\optSupp(T(s,a^{\prime})) , 𝐟 π , s a ′ \mathbf{f}^{\pi,s_{a^{\prime}}} is unaffected by the choice of action π ⁡ ( s ) \pi(s) . Note that similar reasoning implies that F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} (because eq. 181 is a containment relation in general).

Since F a ′ = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ F_{a^{\prime}}=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})} , if F a F_{a} contains a copy of F a ′ F_{a^{\prime}} via ϕ \phi , then F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} contains a copy of F T ⁡ ( s , a ′ ) ∗ F^{*}_{T(s,a^{\prime})} via ϕ \phi . Then ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) ⊆ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ ⊆ F T ⁡ ( s , a ) ∗ \phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\subseteq\phi\cdot F^{*}_{T(s,a^{\prime})}\subseteq F^{*}_{T(s,a)} , and so F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} contains a copy of ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) . Then apply Lemma E.44 with Δ 1 ≔ T ⁡ ( s , a ′ ) \Delta_{1}\coloneqq T(s,a^{\prime}) and Δ 2 ≔ T ⁡ ( s , a ) \Delta_{2}\coloneqq T(s,a) to conclude that ∀ γ ∈ [ 0 , 1 ] : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] ≤ most : 𝔇 bound 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right]\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right] .

Suppose ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty. To apply the second condition of Lemma E.44 , we want to demonstrate that ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is also non-empty.

First consider 𝐟 ∈ ℱ nd ⁡ ( s ) ∩ F a \mathbf{f}\in\Fndop(s)\cap F_{a} . Because F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} , we have that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ F T ⁡ ( s , a ) ∗ \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in F^{*}_{T(s,a)} . Because 𝐟 ∈ ℱ nd ⁡ ( s ) \mathbf{f}\in\Fndop(s) , by definition 3.6 , ∃ 𝐫 ∈ ℝ | 𝒮 | , γ x ∈ ( 0 , 1 ) \exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|},\gamma_{x}\in(0,1) such that 𝐟 ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle\mathbf{f}(\gamma_{x})^{\top}\mathbf{r} > max 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 . \displaystyle>\max_{\mathbf{f}^{\prime}\in\Fop(s)\setminus\left\{\mathbf{f}\right\}}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r}. (183) Then since γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , γ x − 1 ​ ( 𝐟 ⁡ ( γ x ) − 𝐞 s ) ⊤ ​ 𝐫 \displaystyle\gamma_{x}^{-1}(\mathbf{f}(\gamma_{x})-\mathbf{e}_{s})^{\top}\mathbf{r} > max 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } ⁡ γ x − 1 ​ ( 𝐟 ′ ​ ( γ x ) − 𝐞 s ) ⊤ ​ 𝐫 \displaystyle>\max_{\mathbf{f}^{\prime}\in\Fop(s)\setminus\left\{\mathbf{f}\right\}}\gamma_{x}^{-1}(\mathbf{f}^{\prime}(\gamma_{x})-\mathbf{e}_{s})^{\top}\mathbf{r} (184) = max 𝐟 ′ ∈ γ x − 1 ​ ( ( ℱ ⁡ ( s ) ∖ { 𝐟 } ) − 𝐞 s ) ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle=\max_{\mathbf{f}^{\prime}\in\gamma_{x}^{-1}\left((\Fop(s)\setminus\left\{\mathbf{f}\right\})-\mathbf{e}_{s}\right)}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r} (185) ≥ max 𝐟 ′ ∈ γ x − 1 ​ ( ( F a ∖ { 𝐟 } ) − 𝐞 s ) ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle\geq\max_{\mathbf{f}^{\prime}\in\gamma_{x}^{-1}\left((F_{a}\setminus\left\{\mathbf{f}\right\})-\mathbf{e}_{s}\right)}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r} (186) = max 𝐟 ′ ∈ F T ⁡ ( s , a ) ∗ ∖ { γ x − 1 ​ ( 𝐟 − 𝐞 s ) } ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 . \displaystyle=\max_{\mathbf{f}^{\prime}\in F^{*}_{T(s,a)}\setminus\left\{\gamma_{x}^{-1}(\mathbf{f}-\mathbf{e}_{s})\right\}}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r}. (187)

Equation 186 holds because F a ⊆ ℱ ⁡ ( s ) F_{a}\subseteq\Fop(s) . By assumption, action a a is optimal for 𝐫 \mathbf{r} at state s s and at discount rate γ x \gamma_{x} . Equation 181 shows that F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} potentially allows the agent a non-stationary policy choice at s s , but non-stationary policies cannot increase optimal value [ Puterman, 2014 ] . Therefore, eq. 187 holds.

We assumed that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in\gamma^{-1}(\Fndop(s)-\mathbf{e}_{s}) . Furthermore, since we just showed that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ F T ⁡ ( s , a ) ∗ \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in F^{*}_{T(s,a)} is strictly optimal over the other elements of F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} for reward function 𝐫 \mathbf{r} at discount rate γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , we conclude that it is an element of ND ​ ( F T ⁡ ( s , a ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right) by definition E.13 . Then we conclude that γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ⊆ ND ​ ( F T ⁡ ( s , a ) ∗ ) \gamma^{-1}(\Fndop(s)-\mathbf{e}_{s})\cap F^{*}_{T(s,a)}\subseteq\text{{ND}}\left(F^{*}_{T(s,a)}\right) .

We now show that ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is non-empty. 0 \displaystyle 0 < | ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) | \displaystyle<\left|\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right)\right| (188) = | γ − 1 ​ ( ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) − 𝐞 s ) | \displaystyle=\left|\gamma^{-1}\left(\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right)-\mathbf{e}_{s}\right)\right| (189) ≤ | γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ ( F T ⁡ ( s , a ) ∗ ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ ) | \displaystyle\leq\left|\gamma^{-1}\left(\Fndop(s)-\mathbf{e}_{s}\right)\cap\left(F^{*}_{T(s,a)}\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right)\right| (190) = | ( γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ | \displaystyle=\left|\left(\gamma^{-1}\left(\Fndop(s)-\mathbf{e}_{s}\right)\cap F^{*}_{T(s,a)}\right)\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right| (191) ≤ | ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ | \displaystyle\leq\left|\text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right| (192) ≤ | ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) | . \displaystyle\leq\left|\text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\right|. (193) Equation 188 follows by the assumption that ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty. Let 𝐟 , 𝐟 ′ ∈ ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \mathbf{f},\mathbf{f}^{\prime}\in\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) be distinct. Then we must have that for some γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , 𝐟 ⁡ ( γ x ) ≠ 𝐟 ′ ​ ( γ x ) \mathbf{f}(\gamma_{x})\neq\mathbf{f}^{\prime}(\gamma_{x}) . This holds iff γ x − 1 ​ ( 𝐟 ⁡ ( γ x ) − 𝐞 s ) ≠ γ x − 1 ​ ( 𝐟 ′ ​ ( γ x ) − 𝐞 s ) \gamma_{x}^{-1}(\mathbf{f}(\gamma_{x})-\mathbf{e}_{s})\neq\gamma_{x}^{-1}(\mathbf{f}^{\prime}(\gamma_{x})-\mathbf{e}_{s}) , and so eq. 189 holds.

Equation 190 holds because F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} and F a ′ = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ F_{a}^{\prime}=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})} by eq. 182 . Equation 192 holds because we showed above that γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ⊆ ND ​ ( F T ⁡ ( s , a ) ∗ ) \gamma^{-1}(\Fndop(s)-\mathbf{e}_{s})\cap F^{*}_{T(s,a)}\subseteq\text{{ND}}\left(F^{*}_{T(s,a)}\right) . Equation 193 holds because ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) ⊆ F T ⁡ ( s , a ′ ) ∗ \text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\subseteq F^{*}_{T(s,a^{\prime})} by definition E.13 .

Therefore, ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is non-empty, and so apply the second condition of Lemma E.44 to conclude that for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , ∀ γ ∈ ( 0 , 1 ) : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 X ​ - iid ​ ( s a ′ , γ ) ] < 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 X ​ - iid ​ ( s a , γ ) ] \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(s_{a^{\prime}},\gamma\right)\right]<\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(s_{a},\gamma\right)\right] , and that ∀ γ ∈ ( 0 , 1 ) : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] ≱ most : 𝔇 bound 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right] .

Item 2 . Let ϕ ′ ​ ( s x ) ≔ ϕ ⁡ ( s x ) \phi^{\prime}(s_{x})\coloneqq\phi(s_{x}) when s x ∈ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s_{x}\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) , and equal s x s_{x} otherwise. Since ϕ \phi is an involution, so is ϕ ′ \phi^{\prime} . ϕ ′ ⋅ F a ′ \displaystyle\phi^{\prime}\cdot F_{a^{\prime}} ≔ { 𝐏 ϕ ′ ( 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ) ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle\coloneqq\left\{\mathbf{P}_{\phi^{\prime}}\left(\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\right)\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (194) = { 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐏 ϕ ′ 𝐟 π , s a ′ ] ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{P}_{\phi^{\prime}}\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (195) = { 𝐏 ϕ 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐏 ϕ 𝐟 π , s a ′ ] ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle=\left\{\mathbf{P}_{\phi}\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{P}_{\phi}\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (196) ≕ ϕ ⋅ F a ′ \displaystyle\eqqcolon\phi\cdot F_{a^{\prime}} (197) ⊆ F a . \displaystyle\subseteq F_{a}. (198)

Equation 195 follows because if s ∈ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) , then we already showed that ϕ \phi fixes s s . Otherwise, ϕ ′ ​ ( s ) = s \phi^{\prime}(s)=s by definition. Equation 196 follows by the definition of ϕ ′ \phi^{\prime} on Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) \text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) and because 𝐞 s = 𝐏 ϕ ​ 𝐞 s \mathbf{e}_{s}=\mathbf{P}_{\phi}\mathbf{e}_{s} . Next, we assumed that ϕ ⋅ F a ′ ⊆ F a \phi\cdot F_{a^{\prime}}\subseteq F_{a} , and so eq. 198 holds.