Optimal Policies Tend to Seek Power — part 10/16
Cleaned-text code points [140463, 156253). Bibliography omitted here; source.md includes it.
If lim γ → 1 δ ( γ ) > 0 \lim_{\gamma\to 1}\delta(\gamma)>0 , then there exist reward functions whose optimal policy sets Π ∗ ( R , γ ) \Pi^{*}\left(R,\gamma\right) never converge (in the discrete topology on sets) to Π ∗ ( R , 1 ) \Pi^{*}\left(R,1\right) , contradicting Lemma E.33 . So lim γ → 1 δ ( γ ) = 0 \lim_{\gamma\to 1}\delta(\gamma)=0 .
By the definition of optimality probability ( definition 4.3 ) and of δ ( γ ) \delta(\gamma) , | ℙ 𝒟 any ( F , γ ) − ℙ 𝒟 any ( F , 1 ) | ≤ δ ( γ ) |\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)-\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,1\right)|\leq\delta(\gamma) . Since lim γ → 1 δ ( γ ) = 0 \lim_{\gamma\to 1}\delta(\gamma)=0 , lim γ → 1 ℙ 𝒟 any ( F , γ ) = ℙ 𝒟 any ( F , 1 ) \lim_{\gamma\to 1}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,1\right) .
A similar proof shows that lim γ → 0 ℙ 𝒟 any ( F , γ ) = ℙ 𝒟 any ( F , 0 ) \lim_{\gamma\to 0}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,0\right) . ∎
###### Lemma E.35 (Optimality probability identity) .
Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) and let F ⊆ ℱ ( s ) F\subseteq\Fop(s) . ℙ 𝒟 any ( F , γ ) = p 𝒟 ′ ( F ( γ ) ≥ ℱ ( s , γ ) ) = p 𝒟 ′ ( F ( γ ) ≥ ℱ nd ( s , γ ) ) . \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fop(s,\gamma)\right)=p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fndop(s,\gamma)\right). (79)
###### Proof.
Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . ℙ 𝒟 any ( F , γ ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐟 π ∈ F : π ∈ Π ∗ ( R , γ ) ) \displaystyle\coloneqq\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{f}^{\pi}\in F\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{*}\left(R,\gamma\right)\right) (80) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐟 ∈ F 𝐟 ( γ ) ⊤ 𝐫 = max 𝐟 ′ ∈ ℱ ( s ) 𝐟 ′ ( γ ) ⊤ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{f}\in F}\mathbf{f}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}}\right] (81) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐟 ∈ F 𝐟 ( γ ) ⊤ 𝐫 = max 𝐟 ′ ∈ ℱ nd ( s ) 𝐟 ′ ( γ ) ⊤ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{f}\in F}\mathbf{f}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fndop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}}\right] (82) ≕ p 𝒟 ′ ( F ( γ ) ≥ ℱ nd ( s , γ ) ) . \displaystyle\eqqcolon p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fndop(s,\gamma)\right). (83) Equation 81 follows because Lemma E.1 shows that π \pi is optimal iff it induces an optimal visit distribution 𝐟 \mathbf{f} at every state. Equation 82 follows because ∀ 𝐫 ∈ ℝ | 𝒮 | : max 𝐟 ′ ∈ ℱ ( s ) 𝐟 ′ ( γ ) ⊤ 𝐫 = max 𝐟 ′ ∈ ℱ nd ( s ) 𝐟 ′ ( γ ) ⊤ 𝐫 \forall\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\max_{\mathbf{f}^{\prime}\in\Fop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fndop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r} by section D.1.1 . ∎
### E.3 Basic properties of Power
###### Lemma E.36 ( Power identities) .
Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ nd ( s ) 1 − γ γ ( 𝐟 ( γ ) − 𝐞 s ) ⊤ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (84) = 1 − γ γ 𝔼 𝐫 ∼ 𝒟 bound [ V R ∗ ( s , γ ) − R ( s ) ] \displaystyle=\dfrac{1-\gamma}{\gamma}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right] (85) = 1 − γ γ ( V 𝒟 bound ∗ ( s , γ ) − 𝔼 R ∼ 𝒟 bound [ R ( s ) ] ) \displaystyle=\dfrac{1-\gamma}{\gamma}\left(V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s)\right]\right) (86) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) V R π ( s ′ , γ ) ] ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right]. (87)
###### Proof.
Power 𝒟 bound ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s,\gamma) ≔ 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ( s ) 1 − γ γ ( 𝐟 ( γ ) − 𝐞 s ) ⊤ 𝐫 ] \displaystyle\coloneqq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (88) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ nd ( s ) 1 − γ γ ( 𝐟 ( γ ) − 𝐞 s ) ⊤ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (89) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ( s ) 1 − γ γ ( 𝐟 ( γ ) − 𝐞 s ) ⊤ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (90) = 1 − γ γ 𝔼 𝐫 ∼ 𝒟 bound [ V R ∗ ( s , γ ) − R ( s ) ] \displaystyle=\dfrac{1-\gamma}{\gamma}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right] (91) = 1 − γ γ ( V 𝒟 bound ∗ ( s , γ ) − 𝔼 R ∼ 𝒟 bound [ R ( s ) ] ) \displaystyle=\dfrac{1-\gamma}{\gamma}\left(V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s)\right]\right) (92) = 𝔼 𝐫 ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) 𝐟 π , s ′ ( γ ) ⊤ 𝐫 ] ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)\mathbf{f}^{\pi,s^{\prime}}(\gamma)^{\top}\mathbf{r}\right]\right] (93) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) V R π ( s ′ , γ ) ] ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right]. (94)
Equation 89 follows from section D.1.1 . Equation 91 follows from the dual formulation of optimal value functions. Equation 92 holds by the definition of V 𝒟 bound ∗ ( s , γ ) V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) ( definition 5.1 ). Equation 93 holds because 𝐟 π , s ( γ ) = 𝐞 s + γ 𝔼 s ′ ∼ T ( s , π ( s ) ) [ 𝐟 π , s ′ ( γ ) ] \mathbf{f}^{\pi,s}(\gamma)=\mathbf{e}_{s}+\gamma\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\mathbf{f}^{\pi,s^{\prime}}(\gamma)\right] by the definition of a visit distribution function ( definition 3.3 ). ∎
###### Definition E.37 (Discount-normalized value function) .
Let π \pi be a policy, R R a reward function, and s s a state. For γ ∈ [ 0 , 1 ] \gamma\in[0,1] , V R , norm π ( s , γ ) ≔ lim γ ∗ → γ ( 1 − γ ∗ ) V R π ( s , γ ∗ ) V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)\coloneqq\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{\pi}_{R}(s,\gamma^{*}) .
###### Lemma E.38 (Normalized value functions have uniformly bounded derivative) .
There exists K ≥ 0 K\geq 0 such that for all reward functions 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ] | d d γ V R , norm π ( s , γ ) | ≤ K ‖ 𝐫 ‖ 1 \sup_{\begin{subarray}{c}s\in\mathcal{S},\pi\in\Pi,\gamma\in[0,1]\end{subarray}}\left|\frac{d}{d\gamma}V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)\right|\leq K\left\lVert\mathbf{r}\right\rVert_{1} .
###### Proof.
Let π \pi be any policy, s s a state, and R R a reward function. Since V R , norm π ( s , γ ) = lim γ ∗ → γ ( 1 − γ ∗ ) 𝐟 π , s ( γ ∗ ) ⊤ 𝐫 V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)=\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*})^{\top}\mathbf{r} , d d γ V R , norm π ( s , γ ) \frac{d}{d\gamma}V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right) is controlled by the behavior of lim γ ∗ → γ ( 1 − γ ∗ ) 𝐟 π , s ( γ ∗ ) \lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*}) . We show that this function’s gradient is bounded in infinity norm.
By Lemma E.4 , 𝐟 π , s ( γ ) \mathbf{f}^{\pi,s}(\gamma) is a multivariate rational function on γ \gamma . Therefore, for any state s ′ s^{\prime} , 𝐟 π , s ( γ ) ⊤ 𝐞 s ′ = P ( γ ) Q ( γ ) \mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}}=\frac{P(\gamma)}{Q(\gamma)} in reduced form. By Proposition E.3 , 0 ≤ 𝐟 π , s ( γ ) ⊤ 𝐞 s ′ ≤ 1 1 − γ 0\leq\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}}\leq\frac{1}{1-\gamma} . Thus, Q Q may only have a root of multiplicity 1 at γ = 1 \gamma=1 , and Q ( γ ) ≠ 0 Q(\gamma)\neq 0 for γ ∈ [ 0 , 1 ) \gamma\in[0,1) . Let f s ′ ( γ ) ≔ ( 1 − γ ) 𝐟 π , s ( γ ) ⊤ 𝐞 s ′ f_{s^{\prime}}(\gamma)\coloneqq(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}} .
If Q ( 1 ) ≠ 0 Q(1)\neq 0 , then the derivative f s ′ ′ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) because the polynomial ( 1 − γ ) P ( γ ) (1-\gamma)P(\gamma) cannot diverge on a bounded domain.
If Q ( 1 ) = 0 Q(1)=0 , then factor out the root as Q ( γ ) = ( 1 − γ ) Q ∗ ( γ ) Q(\gamma)=(1-\gamma)Q^{*}(\gamma) . f s ′ ′ ( γ ) \displaystyle f_{s^{\prime}}^{\prime}(\gamma) = d d γ ( ( 1 − γ ) P ( γ ) Q ( γ ) ) \displaystyle=\frac{d}{d\gamma}\left(\frac{(1-\gamma)P(\gamma)}{Q(\gamma)}\right) (95) = d d γ ( P ( γ ) Q ∗ ( γ ) ) \displaystyle=\frac{d}{d\gamma}\left(\frac{P(\gamma)}{Q^{*}(\gamma)}\right) (96) = P ′ ( γ ) Q ∗ ( γ ) − ( Q ∗ ) ′ ( γ ) P ( γ ) ( Q ∗ ( γ ) ) 2 . \displaystyle=\frac{P^{\prime}(\gamma)Q^{*}(\gamma)-(Q^{*})^{\prime}(\gamma)P(\gamma)}{(Q^{*}(\gamma))^{2}}. (97)
Since Q ∗ ( γ ) Q^{*}(\gamma) is a polynomial with no roots on γ ∈ [ 0 , 1 ] \gamma\in[0,1] , f s ′ ′ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) .
Therefore, whether or not Q ( γ ) Q(\gamma) has a root at γ = 1 \gamma=1 , f s ′ ′ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) . Furthermore, sup γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ ∞ = sup γ ∈ [ 0 , 1 ) max s ′ ∈ 𝒮 | f s ′ ′ ( γ ) | \sup_{\gamma\in[0,1)}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{\infty}=\sup_{\gamma\in[0,1)}\max_{s^{\prime}\in\mathcal{S}}\left|f_{s^{\prime}}^{\prime}(\gamma)\right| is finite since there are only finitely many states.
There are finitely many π ∈ Π \pi\in\Pi , and finitely many states s s , and so there exists some K ′ K^{\prime} such that sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ ∞ ≤ K ′ \sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{\infty}\leq K^{\prime} . Then ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ 1 ≤ | 𝒮 | K ′ ≕ K \left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1}\leq\left|\mathcal{S}\right|K^{\prime}\eqqcolon K . sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d γ V R , norm π ( s , γ ) | ≔ \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}V^{\pi}_{R,\text{norm}}\left(s,\gamma\right)\right|\coloneqq\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d γ lim γ ∗ → γ ( 1 − γ ∗ ) V R π ( s , γ ∗ ) | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{\pi}_{R}\left(s,\gamma^{*}\right)\right| (98) = \displaystyle=\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d γ ( 1 − γ ) V R π ( s , γ ) | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}(1-\gamma)V^{\pi}_{R}\left(s,\gamma\right)\right| (99) = \displaystyle=\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ⊤ 𝐫 | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r}\right| (100) ≤ \displaystyle\leq\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ 1 ‖ 𝐫 ‖ 1 \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1}\left\lVert\mathbf{r}\right\rVert_{1} (101) ≤ \displaystyle\leq\, K ‖ 𝐫 ‖ 1 . \displaystyle K\left\lVert\mathbf{r}\right\rVert_{1}. (102)
Equation 99 holds because V R π ( s , γ ) V^{\pi}_{R}\left(s,\gamma\right) is continuous on γ ∈ [ 0 , 1 ) \gamma\in[0,1) by Corollary E.5 . Equation 101 holds by the Cauchy-Schwarz inequality.
Since | d d γ V R , norm π ( s , γ ) | \left|\frac{d}{d\gamma}V^{\pi}_{R,\text{norm}}\left(s,\gamma\right)\right| is bounded for all γ ∈ [ 0 , 1 ) \gamma\in[0,1) , eq. 102 also holds for γ → 1 \gamma\to 1 . ∎
See 5.3
###### Proof.
Let b , c b,c be such that supp ( 𝒟 bound ) ⊆ [ b , c ] | 𝒮 | \optSupp(\mathcal{D}_{\text{bound}})\subseteq[b,c]^{\left|\mathcal{S}\right|} . For any 𝐫 ∈ supp ( 𝒟 bound ) \mathbf{r}\in\optSupp(\mathcal{D}_{\text{bound}}) and π ∈ Π \pi\in\Pi , V R , norm π ( s , γ ) V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right) has Lipschitz constant K ‖ 𝐫 ‖ 1 ≤ K | 𝒮 | ‖ 𝐫 ‖ ∞ ≤ K | 𝒮 | max ( | c | , | b | ) K\left\lVert\mathbf{r}\right\rVert_{1}\leq K\left|\mathcal{S}\right|\left\lVert\mathbf{r}\right\rVert_{\infty}\leq K\left|\mathcal{S}\right|\max(\left|c\right|,\left|b\right|) on γ ∈ ( 0 , 1 ) \gamma\in(0,1) by Lemma E.38 .
For γ ∈ ( 0 , 1 ) \gamma\in(0,1) , Power 𝒟 bound ( s , γ ) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) V R π ( s ′ , γ ) ] ] \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[(1-\gamma)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right] by eq. 94 . The expectation of the maximum of a set of functions which share a Lipschitz constant, also shares the Lipschitz constant. This shows that Power 𝒟 bound ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) is Lipschitz continuous on γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Thus, its limits are well-defined as γ → 0 \gamma\to 0 and γ → 1 \gamma\to 1 . So it is Lipschitz continuous on the closed unit interval. ∎
See 5.4
###### Proof.
Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) V R ∗ ( s ′ , γ ) ] ] \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T(s,\pi(s))}\left[(1-\gamma)V^{*}_{R}\left(s^{\prime},\gamma\right)\right]\right] (103) ≤ 𝔼 R ∼ 𝒟 bound [ max π ∈ Π 𝔼 s ′ ∼ T ( s , π ( s ) ) [ ( 1 − γ ) max s ′′ ∈ 𝒮 R ( s ′′ ) 1 − γ ] ] \displaystyle\leq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T(s,\pi(s))}\left[(1-\gamma)\frac{\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})}{1-\gamma}\right]\right] (104) = 𝔼 R ∼ 𝒟 bound [ max s ′′ ∈ 𝒮 R ( s ′′ ) ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})\right]. (105)