Optimal Policies Tend to Seek Power — part 15/16

Cleaned-text code points [217561, 232856). Bibliography omitted here; source.md includes it.

Therefore, F a F_{a} contains a copy of F a ′ F_{a^{\prime}} via ϕ ′ \phi^{\prime} fixing all s x ∉ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s_{x}\not\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) . Therefore, F a F_{a} contains a copy of F nd , a ′ ≔ ℱ nd ⁡ ( s ) ∩ F a ′ F_{\text{nd},a^{\prime}}\coloneqq\Fndop(s)\cap F_{a^{\prime}} via the same ϕ ′ \phi^{\prime} . Then apply Lemma E.47 with s ′ ≔ s s^{\prime}\coloneqq s to conclude that ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( F a ′ , γ ) ≤ most : 𝔇 any ℙ 𝒟 any ( F a , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a^{\prime}},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) . By Lemma E.48 , ℙ 𝒟 any ( s , a ′ , γ ) = ℙ 𝒟 any ( F a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a^{\prime}},\gamma\right) and ℙ 𝒟 any ( s , a , γ ) = ℙ 𝒟 any ( F a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) . Therefore, ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( s , a ′ , γ ) ≤ most : 𝔇 any ℙ 𝒟 any ( s , a , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right) .

If ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty, then apply the second condition of Lemma E.47 to conclude that for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and ℙ 𝒟 any ( s , a ′ , γ ) ≱ most : 𝔇 any ℙ 𝒟 any ( s , a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right) . ∎

#### E.4.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles

###### Lemma E.49 ( Power identity when γ = 1 \gamma=1 ) .

Power 𝒟 bound ​ ( s , 1 ) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] . \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right)=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]. (199)

###### Proof.

Power 𝒟 bound ​ ( s , 1 ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ → 1 𝐟 π , s ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 π , s ​ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}^{\pi,s}\in\Fop(s)}\lim_{\gamma\to 1}\frac{1-\gamma}{\gamma}\left(\mathbf{f}^{\pi,s}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (200) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right] (201) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] . \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]. (202) Equation 200 follows by Lemma E.43 . Equation 201 follows by the definition of RSD ​ ( s ) \text{{RSD}}\left(s\right) ( definition 6.10 ). Equation 202 follows because for all 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , Corollary E.11 shows that max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 = max 𝐝 ∈ ND ​ ( RSD ​ ( s ) ) ⁡ 𝐝 ⊤ ​ 𝐫 ≕ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 \max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}=\max_{\mathbf{d}\in\text{{ND}}\left(\text{{RSD}}\left(s\right)\right)}\mathbf{d}^{\top}\mathbf{r}\eqqcolon\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r} . ∎

See 6.12

###### Proof.

Suppose RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right) is similar to D ⊆ RSD ​ ( s ) D\subseteq\text{{RSD}}\left(s\right) via involution ϕ \phi . Power 𝒟 bound ​ ( s ′ , 1 ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ′ ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right)}\mathbf{d}^{\top}\mathbf{r}\right] (203) ≤ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) 𝐝 ⊤ 𝐫 ] \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right] (204) = Power 𝒟 bound ​ ( s , 1 ) \displaystyle=\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) (205) Equation 203 and eq. 205 follow from Lemma E.49 . By applying Lemma E.24 with A ≔ RSD ​ ( s ′ ) , B ′ ≔ D , B ≔ RSD ​ ( s ) A\coloneqq\text{{RSD}}\left(s^{\prime}\right),B^{\prime}\coloneqq D,B\coloneqq\text{{RSD}}\left(s\right) and g g the identity function, eq. 204 follows.

Suppose RSD nd ​ ( s ) ∖ D \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus D is non-empty. By the same result, eq. 204 is a strict inequality for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and we conclude that Power 𝒟 bound ( s ′ , 1 ) ≱ most : 𝔇 bound Power 𝒟 bound ( s , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) . ∎

See 6.13

###### Proof.

Let D sub ≔ ϕ ⋅ D ′ D_{\text{sub}}\coloneqq\phi\cdot D^{\prime} , where D sub ⊆ D D_{\text{sub}}\subseteq D by assumption. Let X ≔ { s i ∈ 𝒮 ∣ max 𝐝 ∈ D ′ ∪ D ⁡ 𝐝 ⊤ ​ 𝐞 s i > 0 } X\coloneqq\left\{s_{i}\in\mathcal{S}\mid\max_{\mathbf{d}\in D^{\prime}\cup D}\mathbf{d}^{\top}\mathbf{e}_{s_{i}}>0\right\} . Define ϕ ′ ​ ( s i ) ≔ { ϕ ⁡ ( s i ) if ​ s i ∈ X s i else . \phi^{\prime}(s_{i})\coloneqq\begin{cases}\phi(s_{i})&\text{ if }s_{i}\in X\\ s_{i}&\text{ else}.\end{cases} (206)

Since ϕ \phi is an involution, ϕ ′ \phi^{\prime} is also an involution. Furthermore, by the definition of X X , ϕ ′ ⋅ D ′ = D sub \phi^{\prime}\cdot D^{\prime}=D_{\text{sub}} and ϕ ′ ⋅ D sub = D ′ \phi^{\prime}\cdot D_{\text{sub}}=D^{\prime} (because we assumed that both equalities hold for ϕ \phi ).

Let D ∗ ≔ D ′ ∪ D sub ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) D^{*}\coloneqq D^{\prime}\cup D_{\text{sub}}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) . ϕ ′ ⋅ D ∗ \displaystyle\phi^{\prime}\cdot D^{*} ≔ ϕ ′ ⋅ ( D ′ ∪ D sub ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) ) \displaystyle\coloneqq\phi^{\prime}\cdot\left(D^{\prime}\cup D_{\text{sub}}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)\right) (207) = ( ϕ ′ ⋅ D ′ ) ∪ ( ϕ ′ ⋅ D sub ) ∪ ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) \displaystyle=\left(\phi^{\prime}\cdot D^{\prime}\right)\cup\left(\phi^{\prime}\cdot D_{\text{sub}}\right)\cup\phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) (208) = D sub ∪ D ′ ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) \displaystyle=D_{\text{sub}}\cup D^{\prime}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) (209) ≕ D ∗ . \displaystyle\eqqcolon D^{*}. (210)

In eq. 209 , we know that ϕ ′ ⋅ D ′ = D sub \phi^{\prime}\cdot D^{\prime}=D_{\text{sub}} and ϕ ′ ⋅ D sub = D ′ \phi^{\prime}\cdot D_{\text{sub}}=D^{\prime} . We just need to show that ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D) .

Suppose ∃ s i ∈ X , 𝐝 ′ ∈ RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) : 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i > 0 \exists s_{i}\in X,\mathbf{d}^{\prime}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}}>0 . By the definition of X X , ∃ 𝐝 ∈ D ′ ∪ D : 𝐝 ⊤ ​ 𝐞 s i > 0 \exists\mathbf{d}\in D^{\prime}\cup D\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\top}\mathbf{e}_{s_{i}}>0 . Then 𝐝 ⊤ ​ 𝐝 ′ \displaystyle\mathbf{d}^{\top}\mathbf{d}^{\prime} = ∑ j = 1 | 𝒮 | 𝐝 ⊤ ​ ( 𝐝 ′ ⊙ 𝐞 s j ) \displaystyle=\sum_{j=1}^{\left|\mathcal{S}\right|}\mathbf{d}^{\top}(\mathbf{d}^{\prime}\odot\mathbf{e}_{s_{j}}) (211) ≥ 𝐝 ⊤ ​ ( 𝐝 ′ ⊙ 𝐞 s i ) \displaystyle\geq\mathbf{d}^{\top}(\mathbf{d}^{\prime}\odot\mathbf{e}_{s_{i}}) (212) = 𝐝 ⊤ ​ ( ( 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i ) ​ 𝐞 s i ) \displaystyle=\mathbf{d}^{\top}\left((\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}})\mathbf{e}_{s_{i}}\right) (213) = ( 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i ) ⋅ ( 𝐝 ⊤ ​ 𝐞 s i ) \displaystyle=(\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}})\cdot(\mathbf{d}^{\top}\mathbf{e}_{s_{i}}) (214) > 0 . \displaystyle>0. (215)

Equation 211 follows from the definitions of the dot and Hadamard products. Equation 212 follows because 𝐝 \mathbf{d} and 𝐝 ′ \mathbf{d}^{\prime} have non-negative entries. Equation 215 follows because 𝐝 ⊤ ​ 𝐞 s i \mathbf{d}^{\top}\mathbf{e}_{s_{i}} and 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i \mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}} are both positive. But eq. 215 shows that 𝐝 ⊤ ​ 𝐝 ′ > 0 \mathbf{d}^{\top}\mathbf{d}^{\prime}>0 , contradicting our assumption that 𝐝 \mathbf{d} and 𝐝 ′ \mathbf{d}^{\prime} are orthogonal.

Therefore, such an s i s_{i} cannot exist, and X ′ ≔ { s i ′ ∈ 𝒮 ∣ max 𝐝 ′ ∈ RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ⁡ 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i > 0 } ⊆ ( 𝒮 ∖ X ) X^{\prime}\coloneqq\left\{s_{i}^{\prime}\in\mathcal{S}\mid\max_{\mathbf{d}^{\prime}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)}\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}}>0\right\}\subseteq(\mathcal{S}\setminus X) . By eq. 206 , ∀ s i ′ ∈ X ′ : ϕ ′ ​ ( s i ′ ) = s i ′ \forall s_{i}^{\prime}\in X^{\prime}\mathrel{\mathop{\ordinarycolon}}\phi^{\prime}(s_{i}^{\prime})=s_{i}^{\prime} . Thus, ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D) , and eq. 209 follows. We conclude that ϕ ′ ⋅ D ∗ = D ∗ \phi^{\prime}\cdot D^{*}=D^{*} .

Consider Z ≔ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) ∪ D ∪ D ′ Z\coloneqq\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)\cup D\cup D^{\prime} . First, Z ⊆ RSD ​ ( s ) Z\subseteq\text{{RSD}}\left(s\right) by definition. Second, RSD nd ​ ( s ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ∪ ( RSD nd ​ ( s ) ∩ D ′ ) ∪ ( RSD nd ​ ( s ) ∩ D ) ⊆ Z \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\cup(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D^{\prime})\cup(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D)\subseteq Z . Note that D ∗ = Z ∖ ( D ∖ D sub ) D^{*}=Z\setminus(D\setminus D_{\text{sub}}) . ℙ 𝒟 any ( D ′ , average ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right) = p 𝒟 any ​ ( D ′ ≥ RSD ​ ( s ) ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(D^{\prime}\geq\text{{RSD}}\left(s\right)\right) (216) ≤ most : 𝔇 any p 𝒟 any ( D ≥ RSD ( s ) ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(D\geq\text{{RSD}}\left(s\right)\right) (217) = ℙ 𝒟 any ( D , average ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right). (218)

Since ϕ ⋅ D ′ ⊆ D \phi\cdot D^{\prime}\subseteq D and ND ​ ( D ′ ) ⊆ D ′ \text{{ND}}\left(D^{\prime}\right)\subseteq D^{\prime} , ϕ ⋅ ND ​ ( D ′ ) ⊆ D \phi\cdot\text{{ND}}\left(D^{\prime}\right)\subseteq D . Then eq. 217 holds by applying Lemma E.28 with A ≔ D ′ , B ′ ≔ D sub , B ≔ D , C ≔ RSD ​ ( s ) A\coloneqq D^{\prime},B^{\prime}\coloneqq D_{\text{sub}},B\coloneqq D,C\coloneqq\text{{RSD}}\left(s\right) , and the previously defined Z Z which we showed satisfies ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C . Furthermore, involution ϕ ′ \phi^{\prime} satisfies ϕ ′ ⋅ B ∗ = ϕ ′ ⋅ ( Z ∖ ( B ∖ B ′ ) ) = Z ∖ ( B ∖ B ′ ) = B ∗ \phi^{\prime}\cdot B^{*}=\phi^{\prime}\cdot\left(Z\setminus(B\setminus B^{\prime})\right)=Z\setminus(B\setminus B^{\prime})=B^{*} by eq. 210 .

When RSD nd ​ ( s ) ∩ ( D ∖ D sub ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap\left(D\setminus D_{\text{sub}}\right) is non-empty, since B ′ ⊆ C B^{\prime}\subseteq C by assumption, Lemma E.28 also shows that eq. 217 is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and that ℙ 𝒟 any ( D ′ , average ) ≱ most : 𝔇 any ℙ 𝒟 any ( D , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right) . ∎

###### Proposition E.50 ( Rsd properties) .

Let 𝐝 ∈ RSD ​ ( s ) \mathbf{d}\in\text{{RSD}}\left(s\right) . 𝐝 \mathbf{d} is element-wise non-negative and ‖ 𝐝 ‖ 1 = 1 \left\lVert\mathbf{d}\right\rVert_{1}=1 .

###### Proof.

𝐝 \mathbf{d} has non-negative elements because it equals the limit of lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) \lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma) , whose elements are non-negative by Proposition E.3 item 1 . ‖ 𝐝 ‖ 1 \displaystyle\left\lVert\mathbf{d}\right\rVert_{1} = ‖ lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) ‖ 1 \displaystyle=\left\lVert\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma)\right\rVert_{1} (219) = lim γ → 1 ( 1 − γ ) ​ ‖ 𝐟 ⁡ ( γ ) ‖ 1 \displaystyle=\lim_{\gamma\to 1}(1-\gamma)\left\lVert\mathbf{f}(\gamma)\right\rVert_{1} (220) = 1 . \displaystyle=1. (221) Equation 219 follows because the definition of rsd s ( definition 6.10 ) ensures that ∃ 𝐟 ∈ ℱ ⁡ ( s ) : lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) = 𝐝 \exists\mathbf{f}\in\Fop(s)\mathrel{\mathop{\ordinarycolon}}\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma)=\mathbf{d} . Equation 220 follows because ‖ ⋅ ‖ 1 \left\lVert\cdot\right\rVert_{1} is a continuous function. Equation 221 follows because ‖ 𝐟 ⁡ ( γ ) ‖ 1 = 1 1 − γ \left\lVert\mathbf{f}(\gamma)\right\rVert_{1}=\frac{1}{1-\gamma} by Proposition E.3 item 2 . ∎

###### Lemma E.51 (When reachable with probability 1, 1-cycles induce non-dominated rsd s) .

If 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) , then 𝐞 s ′ ∈ RSD nd ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) .

###### Proof.