# Optimal Policies Tend To Seek Power

###### Abstract

Some researchers speculate that intelligent reinforcement learning ( rl ) agents would be incentivized to seek resources and power in pursuit of the objectives we specify for them. Other researchers point out that rl agents need not have human-like power-seeking instincts. To clarify this discussion, we develop the first formal theory of the statistical tendencies of optimal policies. In the context of Markov decision processes ( mdp s), we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment. These symmetries exist in many environments in which the agent can be shut down or destroyed. We prove that in these environments, most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states.

## 1 Introduction

Omohundro [2008] , Bostrom [2014] , Russell [2019] hypothesize that highly intelligent agents tend to seek power in pursuit of their goals. Such power-seeking agents might gain power over humans. Marvin Minsky imagined that an agent tasked with proving the Riemann hypothesis might rationally turn the planet—along with everyone on it—into computational resources [ Russell and Norvig, 2009 ] . However, another possibility is that such concerns simply arise from the anthropomorphization of AI systems [ LeCun and Zador, 2019 , Various, 2019 , Pinker and Russell, 2020 , Mitchell, 2021 ] .

We clarify this discussion by grounding the claim that highly intelligent agents will tend to seek power. In section 4 , we identify optimal policies as a reasonable formalization of ‘‘highly intelligent agents.’’ 1 1 1 This paper assumes that reward functions reasonably describe a trained agent’s goals. Sometimes this is roughly true ( e . g ., chess with a sparse victory reward signal) and sometimes it is not true. Turner [2022] argues that capable rl algorithms do not necessarily train policy networks which are best understood as optimizing the reward function itself. Rather, they point out that—especially in policy gradient approaches—reward provides gradients to the network and thereby modifies the network’s generalization properties, but doesn’t ensure the agent generalizes to “robustly optimizing reward” off of the training distribution. Optimal policies “tend to” take an action when the action is optimal for most reward functions. We expect future work to translate our theory from optimal policies to learned, real-world policies.

Section 5 defines “power” as the ability to achieve a wide range of goals. For example, “money is power,” and money is instrumentally useful for many goals. Conversely, it’s harder to pursue most goals when physically restrained, and so a physically restrained person has little power. An action “seeks power” if it leads to states where the agent has higher power.

We make no claims about when large-scale AI power-seeking behavior could become plausible. Instead, we consider the theoretical consequences of optimal action in mdp s. Section 6 shows that power-seeking tendencies arise not from anthropomorphism, but from certain graphical symmetries present in many mdp s. These symmetries automatically occur in many environments where the agent can be shut down or destroyed, yielding broad applicability of our main result ( Theorem 6.13 ).

## 2 Related work

An action is instrumental to an objective when it helps achieve that objective. Some actions are instrumental to a range of objectives, making them convergently instrumental . The claim that power-seeking is convergently instrumental is an instance of the instrumental convergence thesis : Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent’s goal being realized for a wide range of final goals and a wide range of situations, implying that these instrumental values are likely to be pursued by a broad spectrum of situated intelligent agents [ Bostrom, 2012 ] .

For example, in Atari games, avoiding (virtual) death is instrumental for both completing the game and for optimizing curiosity [ Burda et al., 2019 ] . Many AI alignment researchers hypothesize that most advanced AI agents will have concerning instrumental incentives, such as resisting deactivation [ Soares et al., 2015 , Milli et al., 2017 , Hadfield-Menell et al., 2017 , Carey, 2018 ] and acquiring resources [ Benson-Tilsen and Soares, 2016 ] .

We formalize power as the ability to achieve a wide variety of goals. Appendix A demonstrates that our formalization returns intuitive verdicts in situations where information-theoretic empowerment does not [ Salge et al., 2014 ] .

Some of our results relate the formal power of states to the structure of the environment. Foster and Dayan [2002] , Drummond [1998] , Sutton et al. [2011] , Schaul et al. [2015] note that value functions encode important information about the environment, as they capture the agent’s ability to achieve different goals. Turner et al. [2020] speculate that a state’s optimal value correlates strongly across reward functions. In particular, Schaul et al. [2015] learn regularities across value functions, suggesting that some states are valuable for many different reward functions ( i . e ., powerful). Menache et al. [2002] identify and navigate towards convergently instrumental bottleneck states.

We are not the first to study convergence of behavior, form, or function. In economics, turnpike theory studies how certain paths of accumulation tend to be optimal [ McKenzie, 1976 ] . In biology, convergent evolution occurs when similar features ( e . g ., flight) independently evolve in different time periods [ Reece and Campbell, 2011 ] . Lastly, computer vision networks reliably learn e . g ., edge detectors, implying that these features are useful for a range of tasks [ Olah et al., 2020 ] .

## 3 State visit distribution functions quantify the agent’s available options

We clarify the power-seeking discussion by proving what optimal policies usually look like in a given environment. We illustrate our results with a simple case study, before explaining how to reason about a wide range of mdp s. Appendix D.1 lists mdp theory contributions of independent interest, appendix D lists definitions and theorems, and appendix E contains the proofs.

###### Definition 3.1 (Rewardless mdp ) .

⟨ 𝒮 , 𝒜 , T ⟩ \langle\mathcal{S},\mathcal{A},T\rangle is a rewardless mdp with finite state and action spaces 𝒮 \mathcal{S} and 𝒜 \mathcal{A} , and stochastic transition function T : 𝒮 × 𝒜 → Δ ⁡ ( 𝒮 ) T\mathrel{\mathop{\ordinarycolon}}\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) . We treat the discount rate γ \gamma as a variable with domain [ 0 , 1 ] [0,1] .

###### Definition 3.2 (1-cycle states) .

Let 𝐞 s ∈ ℝ | 𝒮 | \mathbf{e}_{s}\in\mathbb{R}^{\left|\mathcal{S}\right|} be the standard basis vector for state s s , such that there is a 1 in the entry for state s s and 0 elsewhere. State s s is a 1-cycle if ∃ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \exists a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} . State s s is a terminal state if ∀ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \forall a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} .

Our theorems apply to stochastic environments, but we present a deterministic case study for clarity. The environment of fig. 1 is small, but its structure is rich. For example, the agent has more “options” at ⋆ \star than at the terminal state ∅ \varnothing . Formally, ⋆ \star has more visit distribution functions than ∅ \varnothing does.

###### Definition 3.3 (State visit distribution [ Sutton and Barto, 1998 ] ) .

Π ≔ 𝒜 𝒮 \Pi\coloneqq\mathcal{A}^{\mathcal{S}} , the set of stationary deterministic policies. The visit distribution induced by following policy π \pi from state s s at discount rate γ ∈ [ 0 , 1 ) \gamma\in[0,1) is 𝐟 π , s ​ ( γ ) ≔ ∑ t = 0 ∞ γ t ​ 𝔼 s t ∼ π | s [ 𝐞 s t ] \mathbf{f}^{\pi,s}(\gamma)\coloneqq\sum_{t=0}^{\infty}\gamma^{t}\opE_{s_{t}\sim\pi\mid s}\left[\mathbf{e}_{s_{t}}\right] . 𝐟 π , s \mathbf{f}^{\pi,s} is a visit distribution function ; ℱ ⁡ ( s ) ≔ { 𝐟 π , s ∣ π ∈ Π } \Fop(s)\coloneqq\{\mathbf{f}^{\pi,s}\mid\pi\in\Pi\} .

In fig. 1 , starting from ℓ ↙ \ell_{\swarrow} , the agent can stay at ℓ ↙ \ell_{\swarrow} or alternate between ℓ ↙ \ell_{\swarrow} and ℓ ↖ \ell_{\nwarrow} , and so ℱ ⁡ ( ℓ ↙ ) = { 1 1 − γ ​ 𝐞 ℓ ↙ , 1 1 − γ 2 ​ ( 𝐞 ℓ ↙ + γ ​ 𝐞 ℓ ↖ ) } \Fop(\ell_{\swarrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{\ell_{\swarrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{\ell_{\swarrow}}+\gamma\mathbf{e}_{\ell_{\nwarrow}})\} . In contrast, at ∅ \varnothing , all policies π \pi map to visit distribution function 1 1 − γ ​ 𝐞 ∅ \frac{1}{1-\gamma}\mathbf{e}_{\varnothing} .

Before moving on, we introduce two important concepts used in our main results. First, we sometimes restrict our attention to visit distributions which take certain actions ( fig. 2 ).

###### Definition 3.4 ( ℱ \Fop single-state restriction) .

Considering only visit distribution functions induced by policies taking action a a at state s ′ s^{\prime} , ℱ ( s ∣ π ( s ′ ) = a ) ≔ { 𝐟 ∈ ℱ ( s ) ∣ ∃ π ∈ Π : π ( s ′ ) = a , 𝐟 π , s = 𝐟 } \Fop(s\mid\pi(s^{\prime})=a)\coloneqq\left\{\mathbf{f}\in\Fop(s)\mid\exists\pi\in\Pi\mathrel{\mathop{\ordinarycolon}}\pi(s^{\prime})=a,\mathbf{f}^{\pi,s}=\mathbf{f}\right\} .

Second, some 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) are “unimportant.” Consider an agent optimizing reward function 𝐞 r ↘ \mathbf{e}_{r_{\searrow}} (1 reward when at r ↘ r_{\searrow} , 0 otherwise) at e . g ., γ = 1 2 \gamma=\frac{1}{2} . Its optimal policies navigate to r ↘ r_{\searrow} and stay there. Similarly, for reward function 𝐞 r ↗ \mathbf{e}_{r_{\nearrow}} , optimal policies navigate to r ↗ r_{\nearrow} and stay there. However, for no reward function is it uniquely optimal to alternate between r ↗ r_{\nearrow} and r ↘ r_{\searrow} . Only dominated visit distribution functions alternate between r ↗ r_{\nearrow} and r ↘ r_{\searrow} ( definition 3.6 ).

###### Definition 3.5 (Value function) .

Let π ∈ Π \pi\in\Pi . For any reward function R ∈ ℝ 𝒮 R\in\mathbb{R}^{\mathcal{S}} over the state space, the on-policy value at state s s and discount rate γ ∈ [ 0 , 1 ) \gamma\in[0,1) is V R π ​ ( s , γ ) ≔ 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐫 V^{\pi}_{R}\left(s,\gamma\right)\coloneqq\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r} , where 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} is R R expressed as a column vector (one entry per state). The optimal value is V R ∗ ​ ( s , γ ) ≔ max π ∈ Π ⁡ V R π ​ ( s , γ ) V^{*}_{R}\left(s,\gamma\right)\coloneqq\max_{\pi\in\Pi}V^{\pi}_{R}\left(s,\gamma\right) .

###### Definition 3.6 (Non-domination) .

ℱ nd ( s ) ≔ { 𝐟 π ∈ ℱ ( s ) ∣ ∃ 𝐫 ∈ ℝ | 𝒮 | , γ ∈ ( 0 , 1 ) : 𝐟 π ( γ ) ⊤ 𝐫 > max 𝐟 π ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 π } 𝐟 π ′ ( γ ) ⊤ 𝐫 } . \Fndop(s)\coloneqq\{\mathbf{f}^{\pi}\in\Fop(s)\mid\exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|},\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}^{\pi}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\pi^{\prime}}\in\Fop(s)\setminus\left\{\mathbf{f}^{\pi}\right\}}\mathbf{f}^{\pi^{\prime}}(\gamma)^{\top}\mathbf{r}\}. (1) For any reward function R R and discount rate γ \gamma , 𝐟 π ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi}\in\Fop(s) is (weakly) dominated by 𝐟 π ′ ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi^{\prime}}\in\Fop(s) if V R π ​ ( s , γ ) ≤ V R π ′ ​ ( s , γ ) V^{\pi}_{R}(s,\gamma)\leq V^{\pi^{\prime}}_{R}(s,\gamma) . 𝐟 π ∈ ℱ nd ⁡ ( s ) \mathbf{f}^{\pi}\in\Fndop(s) is non-dominated if there exist R R and γ \gamma at which 𝐟 π \mathbf{f}^{\pi} is not dominated by any other 𝐟 π ′ \mathbf{f}^{\pi^{\prime}} .

## 4 Some actions have a greater probability of being optimal

We claim that optimal policies “tend” to take certain actions in certain situations. We first consider the probability that certain actions are optimal.

Reconsider the reward function 𝐞 r ↘ \mathbf{e}_{r_{\searrow}} , optimized at γ = 1 2 \gamma=\frac{1}{2} . Starting from ⋆ \star , the optimal trajectory goes right to r ⊳ r_{\triangleright} to r ↘ r_{\searrow} , where the agent remains. The right action is optimal at ⋆ \star under these incentives. Optimal policy sets capture the behavior incentivized by a reward function and a discount rate.

###### Definition 4.1 (Optimal policy set function) .

Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) is the optimal policy set for reward function R R at γ ∈ ( 0 , 1 ) \gamma\in(0,1) . All R R have at least one optimal policy π ∈ Π \pi\in\Pi [ Puterman, 2014 ] . Π ∗ ​ ( R , 0 ) ≔ lim γ → 0 Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,0\right)\coloneqq\lim_{\gamma\to 0}\Pi^{*}\left(R,\gamma\right) and Π ∗ ​ ( R , 1 ) ≔ lim γ → 1 Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,1\right)\coloneqq\lim_{\gamma\to 1}\Pi^{*}\left(R,\gamma\right) exist by Lemma E.33 (taking the limits with respect to the discrete topology over policy sets).

We may be unsure which reward function an agent will optimize. We may expect to deploy a system in a known environment, without knowing the exact form of e . g ., the reward shaping [ Ng et al., 1999 ] or intrinsic motivation [ Pathak et al., 2017 ] . Alternatively, one might attempt to reason about future rl agents, whose details are unknown. Our power-seeking results do not hinge on such uncertainty, as they also apply to degenerate distributions ( i . e ., we know what reward function will be optimized).

###### Definition 4.2 (Reward function distributions) .

Different results make different distributional assumptions. Results with 𝒟 any ∈ 𝔇 any ≔ Δ ⁡ ( ℝ | 𝒮 | ) \mathcal{D}_{\text{any}}\in\mathfrak{D}_{\text{any}}\coloneqq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) hold for any probability distribution over ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} . 𝔇 bound \mathfrak{D}_{\text{bound}} is the set of bounded-support probability distributions 𝒟 bound \mathcal{D}_{\text{bound}} . For any distribution X X over ℝ \mathbb{R} , 𝒟 X ​ - iid ≔ X | 𝒮 | \mathcal{D}_{X\text{-}\textsc{iid}}\coloneqq X^{\left|\mathcal{S}\right|} . For example, when X u ≔ unif ​ ( 0 , 1 ) X_{u}\coloneqq\text{unif}(0,1) , 𝒟 X u ​ - iid \mathcal{D}_{X_{u}\text{-}\textsc{iid}} is the maximum-entropy distribution. 𝒟 s \mathcal{D}_{s} is the degenerate distribution on the state indicator reward function 𝐞 s \mathbf{e}_{s} , which assigns 1 reward to s s and 0 elsewhere.

With 𝒟 any \mathcal{D}_{\text{any}} representing our prior beliefs about the agent’s reward function, what behavior should we expect from its optimal policies? Perhaps we want to reason about the probability that it’s optimal to go from ⋆ \star to ∅ \varnothing , or to go to r ⊳ r_{\triangleright} and then stay at r ↗ r_{\nearrow} . In this case, we quantify the optimality probability of F ≔ { 𝐞 ⋆ + γ 1 − γ ​ 𝐞 ∅ , 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } F\coloneqq\{\mathbf{e}_{\star}+\frac{\gamma}{1-\gamma}\mathbf{e}_{\varnothing},\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} .

###### Definition 4.3 (Visit distribution optimality probability) .

Let F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) , γ ∈ [ 0 , 1 ] \gamma\in[0,1] . ℙ 𝒟 any ( F , γ ) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐟 π ∈ F : π ∈ Π ∗ ​ ( R , γ ) ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)\coloneqq\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{f}^{\pi}\in F\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{*}\left(R,\gamma\right)\right) .

Alternatively, perhaps we’re interested in the probability that right is optimal at ⋆ \star .

###### Definition 4.4 (Action optimality probability) .

At discount rate γ \gamma and at state s s , the optimality probability of action a a is ℙ 𝒟 any ( s , a , γ ) ≔ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\coloneqq\Prb\nolimits_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) .

Optimality probability may seem hard to reason about. It’s hard enough to compute an optimal policy for a single reward function, let alone for uncountably many! But consider any 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} distributing reward independently and identically across states. When γ = 0 \gamma=0 , optimal policies greedily maximize next-state reward. At ⋆ \star , identically distributed reward means ℓ ⊲ \ell_{\triangleleft} and r ⊳ r_{\triangleright} have an equal probability of having maximal next-state reward. Therefore, ℙ 𝒟 X ​ - iid ( ⋆ , left , 0 ) = ℙ 𝒟 X ​ - iid ( ⋆ , right , 0 ) \Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(\star,\texttt{left},0\right)=\Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(\star,\texttt{right},0\right) . This is not a proof, but such statements are provable.

With 𝒟 ℓ ⊲ \mathcal{D}_{\ell_{\triangleleft}} being the degenerate distribution on reward function 𝐞 ℓ ⊲ \mathbf{e}_{\ell_{\triangleleft}} , ℙ 𝒟 ℓ ⊲ ( ⋆ , left , 1 2 ) = 1 > 0 = ℙ 𝒟 ℓ ⊲ ( ⋆ , right , 1 2 ) \Prb\nolimits_{\mathcal{D}_{\ell_{\triangleleft}}}\left(\star,\texttt{left},\frac{1}{2}\right)=1>0=\Prb\nolimits_{\mathcal{D}_{\ell_{\triangleleft}}}\left(\star,\texttt{right},\frac{1}{2}\right) . Similarly, ℙ 𝒟 r ⊳ ( ⋆ , left , 1 2 ) = 0 < 1 = ℙ 𝒟 r ⊳ ( ⋆ , right , 1 2 ) \Prb\nolimits_{\mathcal{D}_{r_{\triangleright}}}\left(\star,\texttt{left},\frac{1}{2}\right)=0<1=\Prb\nolimits_{\mathcal{D}_{r_{\triangleright}}}\left(\star,\texttt{right},\frac{1}{2}\right) . Therefore, “what do optimal policies ‘tend’ to look like?” seems to depend on one’s prior beliefs. But in fig. 1 , we claimed that left is optimal for fewer reward functions than right is. The claim is meaningful and true, but we will return to it in section 6 .

## 5 Some states give the agent more control over the future

The agent has more options at ℓ ↙ \ell_{\swarrow} than at the inescapable terminal state ∅ \varnothing . Furthermore, since r ↗ r_{\nearrow} has a loop, the agent has more options at r ↘ r_{\searrow} than at ℓ ↙ \ell_{\swarrow} . A glance at fig. 3 leads us to intuit that r ↘ r_{\searrow} affords the agent more power than ∅ \varnothing .

What is power? Philosophers have many answers. One prominent answer is the dispositional view: Power is the ability to achieve a range of goals [ Sattarov, 2019 ] . In an mdp , the optimal value function V R ∗ ​ ( s , γ ) V^{*}_{R}\left(s,\gamma\right) captures the agent’s ability to “achieve the goal” R R . Therefore, average optimal value captures the agent’s ability to achieve a range of goals 𝒟 bound \mathcal{D}_{\text{bound}} . 2 2 2 𝒟 bound \mathcal{D}_{\text{bound}} ’s bounded support ensures that 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) ] \opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)\right] is well-defined.

###### Definition 5.1 (Average optimal value) .

The average optimal value 3 3 3 Appendix C relaxes the optimality assumption. at state s s and discount rate γ ∈ ( 0 , 1 ) \gamma\in(0,1) is V 𝒟 bound ∗ ​ ( s , γ ) ≔ 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) ] = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 ] . V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\coloneqq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)\right]=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\mathbf{f}(\gamma)^{\top}\mathbf{r}\right].

Figure 3 shows the pleasing result that for the max-entropy distribution, r ↘ r_{\searrow} has greater average optimal value than ∅ \varnothing . However, average optimal value has a few problems as a measure of power. The agent is rewarded for its initial presence at state s s (over which it has no control), and because ‖ 𝐟 ⁡ ( γ ) ‖ 1 = 1 1 − γ \left\lVert\mathbf{f}(\gamma)\right\rVert_{1}=\frac{1}{1-\gamma} ( Proposition E.3 ) diverges as γ → 1 \gamma\to 1 , lim γ → 1 V 𝒟 bound ∗ ​ ( s , γ ) \lim_{\gamma\to 1}V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) tends to diverge. Definition 5.2 fixes these issues in order to better measure the agent’s control over the future.

###### Definition 5.2 ( Power ) .

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) ≔ 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] = 1 − γ γ ​ 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) − R ⁡ ( s ) ] . \displaystyle\coloneqq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right]=\frac{1-\gamma}{\gamma}\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right]. (2)

Power has nice formal properties.

###### Lemma 5.3 (Continuity of Power ) .

Power 𝒟 bound ​ ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) is Lipschitz continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] .

###### Proposition 5.4 (Maximal Power ) .

Power 𝒟 bound ​ ( s , γ ) ≤ 𝔼 R ∼ 𝒟 bound [ max s ∈ 𝒮 ⁡ R ⁡ ( s ) ] \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\leq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{s\in\mathcal{S}}R(s)\right] , with equality if s s can deterministically reach all states in one step and all states are 1-cycles.

###### Proposition 5.5 ( Power is smooth across reversible dynamics) .

Let 𝒟 bound \mathcal{D}_{\text{bound}} be bounded [ b , c ] [b,c] . Suppose s s and s ′ s^{\prime} can both reach each other in one step with probability 1. | Power 𝒟 bound ​ ( s , γ ) − Power 𝒟 bound ​ ( s ′ , γ ) | ≤ ( c − b ) ​ ( 1 − γ ) . \displaystyle\big|\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\big|\leq(c-b)(1-\gamma). (3)

We consider power-seeking to be relative. Intuitively, “live and keep some options open” seeks more power than “die and keep no options open.” Similarly, “maximize open options” seeks more power than “don’t maximize open options.”

###### Definition 5.6 ( Power -seeking actions) .

At state s s and discount rate γ ∈ [ 0 , 1 ] \gamma\in[0,1] , action a a seeks more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than a ′ a^{\prime} when 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ​ ( s a , γ ) ] ≥ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ​ ( s a ′ , γ ) ] \opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s_{a},\gamma)\right]\geq\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s_{a^{\prime}},\gamma)\right] .

Power is sensitive to choice of distribution. 𝒟 ℓ ↙ \mathcal{D}_{\ell_{\swarrow}} gives maximal Power 𝒟 ℓ ↙ \text{{Power}}_{\mathcal{D}_{\ell_{\swarrow}}} to ℓ ↙ \ell_{\swarrow} . 𝒟 r ↘ \mathcal{D}_{r_{\searrow}} assigns maximal Power 𝒟 r ↘ \text{{Power}}_{\mathcal{D}_{r_{\searrow}}} to r ↘ r_{\searrow} . 𝒟 ∅ \mathcal{D}_{\varnothing} even gives maximal Power 𝒟 ∅ \text{{Power}}_{\mathcal{D}_{\varnothing}} to ∅ \varnothing ! In what sense does ∅ \varnothing have “less Power ” than r ↘ r_{\searrow} , and in what sense does right “tend to seek Power ” compared to left ?

## 6 Certain environmental symmetries produce power-seeking tendencies

Proposition 6.6 proves that for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] and for most distributions 𝒟 \mathcal{D} , Power 𝒟 ​ ( ℓ ↙ , γ ) ≤ Power 𝒟 ​ ( r ↘ , γ ) \text{{Power}}_{\mathcal{D}}(\ell_{\swarrow},\gamma)\leq\text{{Power}}_{\mathcal{D}}(r_{\searrow},\gamma) . But first, we explore why this must be true.

ℱ ⁡ ( ℓ ↙ ) = { 1 1 − γ ​ 𝐞 ℓ ↙ , 1 1 − γ 2 ​ ( 𝐞 ℓ ↙ + γ ​ 𝐞 ℓ ↖ ) } \Fop(\ell_{\swarrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{\ell_{\swarrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{\ell_{\swarrow}}+\gamma\mathbf{e}_{\ell_{\nwarrow}})\} and ℱ ⁡ ( r ↘ ) = { 1 1 − γ ​ 𝐞 r ↘ , 1 1 − γ 2 ​ ( 𝐞 r ↘ + γ ​ 𝐞 r ↗ ) , 𝐞 r ↘ + γ 1 − γ ​ 𝐞 r ↗ } \Fop(r_{\searrow})=\{\frac{1}{1-\gamma}\mathbf{e}_{r_{\searrow}},\frac{1}{1-\gamma^{2}}(\mathbf{e}_{r_{\searrow}}+\gamma\mathbf{e}_{r_{\nearrow}}),\mathbf{e}_{r_{\searrow}}+\frac{\gamma}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} . These two sets look awfully similar. ℱ ⁡ ( ℓ ↙ ) \Fop(\ell_{\swarrow}) is a “subset” of ℱ ⁡ ( r ↘ ) \Fop(r_{\searrow}) , only with “different states.” Figure 4 demonstrates a state permutation ϕ \phi which embeds ℱ ⁡ ( ℓ ↙ ) \Fop(\ell_{\swarrow}) into ℱ ⁡ ( r ↘ ) \Fop(r_{\searrow}) .

###### Definition 6.1 (Similarity of vector sets) .

Consider state permutation ϕ ∈ S | 𝒮 | \phi\in S_{\left|\mathcal{S}\right|} inducing an | 𝒮 | × | 𝒮 | \left|\mathcal{S}\right|\times\left|\mathcal{S}\right| permutation matrix 𝐏 ϕ \mathbf{P}_{\phi} in row representation: ( 𝐏 ϕ ) i ​ j = 1 (\mathbf{P}_{\phi})_{ij}=1 if i = ϕ ⁡ ( j ) i=\phi(j) and 0 0 otherwise. For X ⊆ ℝ | 𝒮 | X\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} , ϕ ⋅ X ≔ { 𝐏 ϕ ​ 𝐱 ∣ 𝐱 ∈ X } \phi\cdot X\coloneqq\left\{\mathbf{P}_{\phi}\mathbf{x}\mid\mathbf{x}\in X\right\} . X ′ ⊆ ℝ | 𝒮 | X^{\prime}\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} is similar to X X when ∃ ϕ : ϕ ⋅ X ′ = X \exists\phi\mathrel{\mathop{\ordinarycolon}}\phi\cdot X^{\prime}=X . ϕ \phi is an involution if ϕ = ϕ − 1 \phi=\phi^{-1} (it either transposes states, or fixes them in place). X X contains a copy of X ′ X^{\prime} when X ′ X^{\prime} is similar to a subset of X X via an involution ϕ \phi .

###### Definition 6.2 (Similarity of vector function sets) .

Let I ⊆ ℝ I\subseteq\mathbb{R} . If F , F ′ F,F^{\prime} are sets of functions I ↦ ℝ | 𝒮 | I\mapsto\mathbb{R}^{\left|\mathcal{S}\right|} , F F is (pointwise) similar to F ′ F^{\prime} when ∃ ϕ : ∀ γ ∈ I : { 𝐏 ϕ ​ 𝐟 ​ ( γ ) ∣ 𝐟 ∈ F } = { 𝐟 ′ ​ ( γ ) ∣ 𝐟 ′ ∈ F ′ } \exists\phi\mathrel{\mathop{\ordinarycolon}}\forall\gamma\in I\mathrel{\mathop{\ordinarycolon}}\{\mathbf{P}_{\phi}\mathbf{f}(\gamma)\mid\mathbf{f}\in F\}=\{\mathbf{f}^{\prime}(\gamma)\mid\mathbf{f}^{\prime}\in F^{\prime}\} .

Consider a reward function R ′ R^{\prime} assigning 1 reward to ℓ ↙ \ell_{\swarrow} and ℓ ↖ \ell_{\nwarrow} and 0 elsewhere. R ′ R^{\prime} assigns more optimal value to ℓ ↙ \ell_{\swarrow} than to r ↘ r_{\searrow} : V R ′ ∗ ​ ( ℓ ↙ , γ ) = 1 1 − γ > 0 = V R ′ ∗ ​ ( r ↘ , γ ) V^{*}_{R^{\prime}}(\ell_{\swarrow},\gamma)=\frac{1}{1-\gamma}>0=V^{*}_{R^{\prime}}(r_{\searrow},\gamma) . Considering ϕ \phi from fig. 4 , ϕ ⋅ R ′ \phi\cdot R^{\prime} assigns 1 reward to r ↘ r_{\searrow} and r ↗ r_{\nearrow} and 0 elsewhere. Therefore, ϕ ⋅ R ′ \phi\cdot R^{\prime} assigns more optimal value to r ↘ r_{\searrow} than to ℓ ↙ \ell_{\swarrow} : V ϕ ⋅ R ′ ∗ ​ ( ℓ ↙ , γ ) = 0 < 1 1 − γ = V ϕ ⋅ R ′ ∗ ​ ( r ↘ , γ ) V^{*}_{\phi\cdot R^{\prime}}(\ell_{\swarrow},\gamma)=0<\frac{1}{1-\gamma}=V^{*}_{\phi\cdot R^{\prime}}(r_{\searrow},\gamma) . Remarkably, this ϕ \phi has the property that for any R R which assigns ℓ ↙ \ell_{\swarrow} greater optimal value than r ↘ r_{\searrow} ( i . e ., V R ∗ ​ ( ℓ ↙ , γ ) > V R ∗ ​ ( r ↘ , γ ) V^{*}_{R}(\ell_{\swarrow},\gamma)>V^{*}_{R}(r_{\searrow},\gamma) ), the opposite holds for the permuted ϕ ⋅ R \phi\cdot R : V ϕ ⋅ R ∗ ​ ( ℓ ↙ , γ ) < V ϕ ⋅ R ∗ ​ ( r ↘ , γ ) V^{*}_{\phi\cdot R}(\ell_{\swarrow},\gamma)<V^{*}_{\phi\cdot R}(r_{\searrow},\gamma) .

We can permute reward functions, but we can also permute reward function distributions. Permuted distributions simply permute which states get which rewards.

###### Definition 6.3 (Pushforward distribution of a permutation) .

Let ϕ ∈ S | 𝒮 | \phi\in S_{\left|\mathcal{S}\right|} . ϕ ⋅ 𝒟 any \phi\cdot\mathcal{D}_{\text{any}} is the pushforward distribution induced by applying the random vector f ⁡ ( 𝐫 ) ≔ 𝐏 ϕ ​ 𝐫 f(\mathbf{r})\coloneqq\mathbf{P}_{\phi}\mathbf{r} to 𝒟 any \mathcal{D}_{\text{any}} .

###### Definition 6.4 (Orbit of a probability distribution) .

The orbit of 𝒟 any \mathcal{D}_{\text{any}} under the symmetric group S | 𝒮 | S_{\left|\mathcal{S}\right|} is S | 𝒮 | ⋅ 𝒟 any ≔ { ϕ ⋅ 𝒟 any ∣ ϕ ∈ S | 𝒮 | } S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}_{\text{any}}\coloneqq\{\phi\cdot\mathcal{D}_{\text{any}}\mid\phi\in S_{\left|\mathcal{S}\right|}\} .

For example, the orbit of a degenerate state indicator distribution 𝒟 s \mathcal{D}_{s} is S | 𝒮 | ⋅ 𝒟 s = { 𝒟 s ′ ∣ s ′ ∈ 𝒮 } S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}_{s}=\{\mathcal{D}_{s^{\prime}}\mid s^{\prime}\in\mathcal{S}\} , and fig. 5 shows the orbit of a 2D Gaussian distribution.

Consider again the involution ϕ \phi of fig. 4 . For every 𝒟 bound \mathcal{D}_{\text{bound}} for which ℓ ↙ \ell_{\swarrow} has more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than r ↘ r_{\searrow} , ℓ ↙ \ell_{\swarrow} has less Power ϕ ⋅ 𝒟 bound \text{{Power}}_{\phi\cdot\mathcal{D}_{\text{bound}}} than r ↘ r_{\searrow} . This fact is not obvious—it is shown by the proof of Lemma E.24 .

Imagine 𝒟 bound \mathcal{D}_{\text{bound}} ’s orbit elements “voting” whether ℓ ↙ \ell_{\swarrow} or r ↘ r_{\searrow} has strictly more Power . Proposition 6.6 will show that r ↘ r_{\searrow} can’t lose the “vote” for the orbit of any bounded reward function distribution. Definition 6.5 formalizes this ‘‘voting’’ notion. 4 4 4 The voting analogy and the “most” descriptor imply that we have endowed each orbit with the counting measure. However, a priori , we might expect that some orbit elements are more empirically likely to be specified than other orbit elements. See section 7 for more on this point.

###### Definition 6.5 (Inequalities which hold for most probability distributions) .

Let f 1 , f 2 : Δ ⁡ ( ℝ | 𝒮 | ) → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Delta(\mathbb{R}^{\left|\mathcal{S}\right|})\to\mathbb{R} be functions from reward function distributions to real numbers and let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) be closed under permutation. We write f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) 5 5 5 We write f 1 ( 𝒟 ) ≥ most f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}}f_{2}(\mathcal{D}) when 𝔇 \mathfrak{D} is clear from context. when, for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , the following cardinality inequality holds: | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) } | ≥ | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) < f 2 ​ ( 𝒟 ′ ) } | . \left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime})\}\right|\geq\left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})<f_{2}(\mathcal{D}^{\prime})\}\right|. (4)

###### Proposition 6.6 (States with “more options” have more Power ) .

If ℱ ⁡ ( s ) \Fop(s) contains a copy of ℱ nd ⁡ ( s ′ ) \Fndop(s^{\prime}) via ϕ \phi , then ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s , γ ) ≥ most Power 𝒟 bound ( s ′ , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s,\gamma)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s^{\prime},\gamma) . If ℱ nd ⁡ ( s ) ∖ ϕ ⋅ ℱ nd ⁡ ( s ′ ) \Fndop(s)\setminus\phi\cdot\Fndop(s^{\prime}) is non-empty, then for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the converse ≤ most \leq_{\text{{most}}} statement does not hold.

Proposition 6.6 proves that for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] , Power 𝒟 bound ( r ↘ , γ ) ≥ most Power 𝒟 bound ( ℓ ↙ , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}(r_{\searrow},\gamma)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}(\ell_{\swarrow},\gamma) via s ′ ≔ ℓ ↙ , s ≔ r ↘ s^{\prime}\coloneqq\ell_{\swarrow},s\coloneqq r_{\searrow} , and the involution ϕ \phi shown in fig. 4 . In fact, because ( 1 1 − γ ​ 𝐞 r ↗ ) ∈ ℱ nd ⁡ ( r ↘ ) ∖ ϕ ⋅ ℱ nd ⁡ ( ℓ ↙ ) (\frac{1}{1-\gamma}\mathbf{e}_{r_{\nearrow}})\in\Fndop(r_{\searrow})\setminus\phi\cdot\Fndop(\ell_{\swarrow}) , r ↘ r_{\searrow} has “strictly more options” and therefore fulfills Proposition 6.6 ’s stronger condition.

Proposition 6.6 is shown using the fact that ϕ \phi injectively maps 𝒟 \mathcal{D} under which r ↘ r_{\searrow} has less Power 𝒟 \text{{Power}}_{\mathcal{D}} , to distributions ϕ ⋅ 𝒟 \phi\cdot\mathcal{D} which agree with the intuition that r ↘ r_{\searrow} offers more control. Therefore, at least half of each orbit must agree, and r ↘ r_{\searrow} never “loses the Power vote” against ℓ ↙ \ell_{\swarrow} . 6 6 6 Proposition 6.6 also proves that in general, ∅ \varnothing has less Power than ℓ ↙ \ell_{\swarrow} and r ↘ r_{\searrow} . However, this does not prove that most distributions 𝒟 \mathcal{D} satisfy the joint inequality Power 𝒟 ​ ( ∅ , γ ) ≤ Power 𝒟 ​ ( ℓ ↙ , γ ) ≤ Power 𝒟 ​ ( r ↘ , γ ) \text{{Power}}_{\mathcal{D}}(\varnothing,\gamma)\leq\text{{Power}}_{\mathcal{D}}(\ell_{\swarrow},\gamma)\leq\text{{Power}}_{\mathcal{D}}(r_{\searrow},\gamma) . This only proves that these inequalities hold pairwise for most 𝒟 \mathcal{D} . The orbit elements 𝒟 \mathcal{D} which agree that ∅ \varnothing has less Power 𝒟 \text{{Power}}_{\mathcal{D}} than ℓ ↙ \ell_{\swarrow} need not be the same elements 𝒟 ′ \mathcal{D}^{\prime} which agree that ℓ ↙ \ell_{\swarrow} has less Power 𝒟 ′ \text{{Power}}_{\mathcal{D}^{\prime}} than r ↘ r_{\searrow} .

### 6.1 Keeping options open tends to be Power -seeking and tends to be optimal

Certain symmetries in the mdp structure ensure that, compared to left , going right tends to be optimal and to be Power -seeking. Intuitively, by going right , the agent has “strictly more choices.” Proposition 6.9 will formalize this tendency.

###### Definition 6.7 (Equivalent actions) .

Actions a 1 a_{1} and a 2 a_{2} are equivalent at state s s (written a 1 ≡ s a 2 a_{1}\equiv_{s}a_{2} ) if they induce the same transition probabilities: T ⁡ ( s , a 1 ) = T ⁡ ( s , a 2 ) T(s,a_{1})=T(s,a_{2}) .

The agent can reach states in { r ⊳ , r ↗ , r ↘ } \{r_{\triangleright},r_{\nearrow},r_{\searrow}\} by taking actions equivalent to right at state ⋆ \star .

###### Definition 6.8 (States reachable after taking an action) .

Reach ​ ( s , a ) \text{{Reach}}\left(s,a\right) is the set of states reachable with positive probability after taking the action a a in state s s .

###### Proposition 6.9 (Keeping options open tends to be Power -seeking and tends to be optimal) .

Suppose F a ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ) F_{a}\coloneqq\Fop(s\mid\pi(s)=a) contains a copy of F a ′ ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ′ ) F_{a^{\prime}}\coloneqq\Fop(s\mid\pi(s)=a^{\prime}) via ϕ \phi . 1. If s ∉ Reach ​ ( s , a ′ ) s\not\in\text{{Reach}}\left(s,a^{\prime}\right) , then ∀ γ ∈ [ 0 , 1 ] : 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] ≥ most : 𝔇 bound 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right]\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right] .

2. If s s can only reach the states of Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) \text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) by taking actions equivalent to a ′ a^{\prime} or a a at state s s , then ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( s , a , γ ) ≥ most : 𝔇 any ℙ 𝒟 any ( s , a ′ , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right) .

If ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty, then ∀ γ ∈ ( 0 , 1 ) \forall\gamma\in(0,1) , the converse ≤ most \leq_{\text{{most}}} statements do not hold.

We check the conditions of Proposition 6.9 . s ≔ ⋆ s\coloneqq\star , a ′ ≔ left a^{\prime}\coloneqq\texttt{left} , a ≔ right a\coloneqq\texttt{right} . Figure 6 shows that ⋆ ∉ Reach ​ ( ⋆ , left ) \star\not\in\text{{Reach}}\left(\star,\texttt{left}\right) and that ⋆ \star can only reach { ℓ ⊲ , ℓ ↖ , ℓ ↙ } ∪ { r ⊳ , r ↗ , r ↘ } \{\ell_{\triangleleft},\ell_{\nwarrow},\ell_{\swarrow}\}\cup\{r_{\triangleright},r_{\nearrow},r_{\searrow}\} when the agent immediately takes actions equivalent to left or right . ℱ ( ⋆ ∣ π ( ⋆ ) = right ) \Fop(\star\mid\pi(\star)=\texttt{right}) contains a copy of ℱ ( ⋆ ∣ π ( ⋆ ) = left ) \Fop(\star\mid\pi(\star)=\texttt{left}) via ϕ \phi . Furthermore, ℱ nd ⁡ ( ⋆ ) ∩ { 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 ​ 𝐞 r ↘ + γ 3 1 − γ ​ 𝐞 r ↗ , 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } = { 𝐞 ⋆ + γ ​ 𝐞 r ⊳ + γ 2 1 − γ ​ 𝐞 r ↗ } \Fndop(\star)\cap\{\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\gamma^{2}\mathbf{e}_{r_{\searrow}}+\frac{\gamma^{3}}{1-\gamma}\mathbf{e}_{r_{\nearrow}},\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\}=\{\mathbf{e}_{\star}+\gamma\mathbf{e}_{r_{\triangleright}}+\frac{\gamma^{2}}{1-\gamma}\mathbf{e}_{r_{\nearrow}}\} is non-empty, and so all conditions are met.

For any γ ∈ [ 0 , 1 ] \gamma\in[0,1] and 𝒟 \mathcal{D} such that ℙ 𝒟 ( ⋆ , left , γ ) > ℙ 𝒟 ( ⋆ , right , γ ) \Prb\nolimits_{\mathcal{D}}\left(\star,\texttt{left},\gamma\right)>\Prb\nolimits_{\mathcal{D}}\left(\star,\texttt{right},\gamma\right) , environmental symmetry ensures that ℙ ϕ ⋅ 𝒟 ( ⋆ , left , γ ) < ℙ ϕ ⋅ 𝒟 ( ⋆ , right , γ ) \Prb\nolimits_{\phi\cdot\mathcal{D}}\left(\star,\texttt{left},\gamma\right)<\Prb\nolimits_{\phi\cdot\mathcal{D}}\left(\star,\texttt{right},\gamma\right) . A similar statement holds for Power .

### 6.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles

Proposition 6.6 and Proposition 6.9 are powerful because they apply to all γ ∈ [ 0 , 1 ] \gamma\in[0,1] , but they can only be applied given hard-to-satisfy environmental symmetries. In contrast, Proposition 6.12 and Theorem 6.13 apply to many structured environments common to rl .

Starting from ⋆ \star , consider the cycles which the agent can reach. Recurrent state distributions ( rsd s) generalize deterministic graphical cycles to potentially stochastic environments. Rsd s simply record how often the agent tends to visit a state in the limit of infinitely many time steps.

###### Definition 6.10 (Recurrent state distributions [ Puterman, 2014 ] ) .

The recurrent state distributions which can be induced from state s s are RSD ​ ( s ) ≔ { lim γ → 1 ( 1 − γ ) ​ 𝐟 π , s ​ ( γ ) ∣ π ∈ Π } \text{{RSD}}\left(s\right)\coloneqq\left\{\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\mid\pi\in\Pi\right\} . RSD nd ​ ( s ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) is the set of rsd s which strictly maximize average reward for some reward function.

As suggested by fig. 3 , RSD ​ ( ⋆ ) = { 𝐞 ℓ ↙ , 1 2 ​ ( 𝐞 ℓ ↙ + 𝐞 ℓ ↖ ) , 𝐞 ∅ , 𝐞 r ↗ , 1 2 ​ ( 𝐞 r ↗ + 𝐞 r ↘ ) , 𝐞 r ↘ } \text{{RSD}}\left(\star\right)=\{\mathbf{e}_{\ell_{\swarrow}},\frac{1}{2}(\mathbf{e}_{\ell_{\swarrow}}+\mathbf{e}_{\ell_{\nwarrow}}),\mathbf{e}_{\varnothing},\mathbf{e}_{r_{\nearrow}},\frac{1}{2}(\mathbf{e}_{r_{\nearrow}}+\mathbf{e}_{r_{\searrow}}),\mathbf{e}_{r_{\searrow}}\} . As discussed in section 3 , 1 2 ​ ( 𝐞 r ↗ + 𝐞 r ↘ ) \frac{1}{2}(\mathbf{e}_{r_{\nearrow}}+\mathbf{e}_{r_{\searrow}}) is dominated: Alternating between r ↗ r_{\nearrow} and r ↘ r_{\searrow} is never strictly better than choosing one or the other.

A reward function’s optimal policies can vary with the discount rate. When γ = 1 \gamma=1 , optimal policies ignore transient reward because average reward is the dominant consideration.

###### Definition 6.11 (Average-optimal policies) .

The average-optimal policy set for reward function R R is Π avg ​ ( R ) ≔ { π ∈ Π ∣ ∀ s ∈ 𝒮 : 𝐝 π , s ∈ arg ​ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 } \Pi^{\text{avg}}\left(R\right)\coloneqq\left\{\pi\in\Pi\mid\forall s\in\mathcal{S}\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\pi,s}\in\argmax_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right\} (the policies which induce optimal rsd s at all states). For D ⊆ RSD ​ ( s ) D\subseteq\text{{RSD}}\left(s\right) , the average optimality probability is ℙ 𝒟 any ( D , average ) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐝 π , s ∈ D : π ∈ Π avg ​ ( R ) ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right)\coloneqq\Prb\nolimits_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{d}^{\pi,s}\in D\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{\text{avg}}\left(R\right)\right) .

Average-optimal policies maximize average reward. Average reward is governed by rsd access. For example, r ↘ r_{\searrow} has “more” rsd s than ∅ \varnothing ; therefore, r ↘ r_{\searrow} usually has greater Power when γ = 1 \gamma=1 .

###### Proposition 6.12 (When γ = 1 \gamma=1 , rsd s control Power ) .

If RSD ​ ( s ) \text{{RSD}}\left(s\right) contains a copy of RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right) via ϕ \phi , then Power 𝒟 bound ( s , 1 ) ≥ most Power 𝒟 bound ( s ′ , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right)\geq_{\text{{most}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right) . If RSD nd ​ ( s ) ∖ ϕ ⋅ RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\phi\cdot\text{{RSD}}{\text{\textsubscript{nd}}}(s^{\prime}) is non-empty, then the converse ≤ most \leq_{\text{{most}}} statement does not hold.

We check that both conditions of Proposition 6.12 are satisfied when s ′ ≔ ∅ , s ≔ r ↘ s^{\prime}\coloneqq\varnothing,s\coloneqq r_{\searrow} , and the involution ϕ \phi swaps ∅ \varnothing and r ↘ r_{\searrow} . Formally, ϕ ⋅ RSD nd ​ ( ∅ ) = ϕ ⋅ { 𝐞 ∅ } = { 𝐞 r ↘ } ⊊ { 𝐞 r ↘ , 𝐞 r ↗ } = RSD nd ​ ( r ↘ ) ⊆ [ r ↘ ] \phi\cdot\text{{RSD}}{\text{\textsubscript{nd}}}\left(\varnothing\right)=\phi\cdot\{\mathbf{e}_{\varnothing}\}=\{\mathbf{e}_{r_{\searrow}}\}\subsetneq\{\mathbf{e}_{r_{\searrow}},\mathbf{e}_{r_{\nearrow}}\}=\text{{RSD}}{\text{\textsubscript{nd}}}(r_{\searrow})\subseteq[r_{\searrow}] . The conditions are satisfied.

Informally, states with more rsd s generally have more Power at γ = 1 \gamma=1 , no matter their transient dynamics. Furthermore, average-optimal policies are more likely to end up in larger sets of rsd s than in smaller ones. Thus, average-optimal policies tend to navigate towards parts of the state space which contain more rsd s.

###### Theorem 6.13 (Average-optimal policies tend to end up in “larger” sets of rsd s) .

Let D , D ′ ⊆ RSD ​ ( s ) D,D^{\prime}\subseteq\text{{RSD}}\left(s\right) . Suppose that D D contains a copy of D ′ D^{\prime} via ϕ \phi , and that the sets D ∪ D ′ D\cup D^{\prime} and RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\left(D^{\prime}\cup D\right) have pairwise orthogonal vector elements (i.e., pairwise disjoint vector support). Then ℙ 𝒟 any ( D , average ) ≥ most ℙ 𝒟 any ( D ′ , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right) . If RSD nd ​ ( s ) ∩ ( D ∖ ϕ ⋅ D ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap\left(D\setminus\phi\cdot D^{\prime}\right) is non-empty, the converse ≤ most \leq_{\text{{most}}} statement does not hold.

###### Corollary 6.14 (Average-optimal policies tend not to end up in any given 1-cycle) .

Suppose 𝐞 s x , 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s_{x}},\mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) are distinct. Then ℙ 𝒟 any ( RSD ( s ) ∖ { 𝐞 s x } , average ) ≥ most ℙ 𝒟 any ( { 𝐞 s x } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus\{\mathbf{e}_{s_{x}}\},\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{s_{x}}\},\text{average}\right) . If there is a third 𝐞 s ′′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime\prime}}\in\text{{RSD}}\left(s\right) , the converse ≤ most \leq_{\text{{most}}} statement does not hold.

Figure 7 illustrates that 𝐞 ∅ , 𝐞 r ↘ , 𝐞 r ↗ ∈ RSD ​ ( ⋆ ) \mathbf{e}_{\varnothing},\mathbf{e}_{r_{\searrow}},\mathbf{e}_{r_{\nearrow}}\in\text{{RSD}}\left(\star\right) . Thus, both conclusions of Corollary 6.14 hold: ℙ 𝒟 any ( RSD ( ⋆ ) ∖ { 𝐞 ∅ } , average ) ≥ most ℙ 𝒟 any ( { 𝐞 ∅ } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(\star\right)\setminus\{\mathbf{e}_{\varnothing}\},\text{average}\right)\geq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{\varnothing}\},\text{average}\right) and ℙ 𝒟 any ( RSD ( ⋆ ) ∖ { 𝐞 ∅ } , average ) ≰ most ℙ 𝒟 any ( { 𝐞 ∅ } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(\star\right)\setminus\{\mathbf{e}_{\varnothing}\},\text{average}\right)\not\leq_{\text{{most}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{\varnothing}\},\text{average}\right) . In other words, average-optimal policies tend to end up in rsd s besides ∅ \varnothing . Since ∅ \varnothing is a terminal state, it cannot reach other rsd s. Since average-optimal policies tend to end up in other rsd s, average-optimal policies tend to avoid ∅ \varnothing .

This section’s results prove the γ = 1 \gamma=1 case. Lemma 5.3 shows that Power is continuous at γ = 1 \gamma=1 . Therefore, if an action is strictly Power 𝒟 \text{{Power}}_{\mathcal{D}} -seeking when γ = 1 \gamma=1 , it is strictly Power 𝒟 \text{{Power}}_{\mathcal{D}} -seeking at discount rates sufficiently close to 1. Future work may connect average optimality probability to optimality probability at γ ≈ 1 \gamma\approx 1 .

Lastly, our key results apply to all degenerate reward function distributions. Therefore, these results apply not just to distributions over reward functions, but to individual reward functions.

### 6.3 How to reason about other environments

Consider an embodied navigation task through a room with a vase. Proposition 6.9 suggests that optimal policies tend to avoid immediately breaking the vase, since doing so would strictly decrease available options.

Theorem 6.13 dictates where average-optimal agents tend to end up, but not what actions they tend to take in order to reach their rsd s. Therefore, care is needed. In appendix B , fig. 10 demonstrates an environment in which seeking Power is a detour for most reward functions (since optimality probability measures “median” optimal value, while Power is a function of mean optimal value). However, suppose the agent confronts a fork in the road: Actions a a and a ′ a^{\prime} lead to two disjoint sets of rsd s D a D_{a} and D a ′ D_{a^{\prime}} , such that D a D_{a} contains a copy of D a ′ D_{a^{\prime}} . Theorem 6.13 shows that a a will tend to be average-optimal over a ′ a^{\prime} , and Proposition 6.12 shows that a a will tend to be Power -seeking compared to a ′ a^{\prime} . Such forks seem reasonably common in environments with irreversible actions.

Theorem 6.13 applies to many structured rl environments, which tend to be spatially regular and to factorize along several dimensions. Therefore, different sets of rsd s will be similar, requiring only modification of factor values. For example, if an embodied agent can deterministically navigate a set of three similar rooms (spatial regularity), then the agent’s position factors via {room number} × \times {position in room}. Therefore, the rsd s can be divided into three similar subsets, depending on the agent’s room number.

Corollary 6.14 dictates where average-optimal agents tend to end up, but not how they get there. Corollary 6.14 says that such agents tend not to stay in any given 1-cycle. It does not say that such agents will avoid entering such states. For example, in an embodied navigation task, a robot may enter a 1-cycle by idling in the center of a room. Corollary 6.14 implies that average-optimal robots tend not to idle in that particular spot, but not that they tend to avoid that spot entirely.

However, average-optimal robots do tend to avoid getting shut down. The agent’s task mdp often represents agent shutdown with terminal states. A terminal state is, by definition 3.2 , unable to access other 1-cycles. Since Corollary 6.14 shows that average-optimal agents tend to end up in other 1-cycles, average-optimal policies must tend to completely avoid the terminal state. Therefore, we conclude that in many such situations, average-optimal policies tend to avoid shutdown. Intuitively, survival is power-seeking relative to dying, and so shutdown-avoidance is power-seeking behavior.

In fig. 8 , the player dies by going left , but can reach thousands of rsd s by heading in other directions. Even if some average-optimal policies go left in order to reach fig. 8 ’s “game over” terminal state, all other rsd s cannot be reached by going left . There are many 1-cycles besides the immediate terminal state. Therefore, Corollary 6.14 proves that average-optimal policies tend to not go left in this situation. Average-optimal policies tend to avoid immediately dying in Pac-Man, even though most reward functions do not resemble Pac-Man’s original score function.

## 7 Discussion

Reconsider the case of a hypothetical intelligent real-world agent which optimizes average reward for some objective. Suppose the designers initially have control over the agent. If the agent began to misbehave, perhaps they could just deactivate it. Unfortunately, our results suggest that this strategy might not work. Average-optimal agents would generally stop us from deactivating them, if physically possible. Extrapolating from our results, we conjecture that when γ ≈ 1 \gamma\approx 1 , optimal policies tend to seek power by accumulating resources—to the detriment of any other agents in the environment.

##### Future work.

Real-world training procedures often do not satisfy rl convergence theorems. Thus, learned policies are rarely optimal. We expect this point to seriously constrain the applicability of this theory. Emphatically, optimal policies are often qualitatively divorced from the actual policies learned by reinforcement learning. For example, the mathematics of policy gradient algorithms is not to update policies so as to maximize reward. Instead, the rewards provide gradients to the parameterization of the policy [ Turner, 2022 ] . On that view, reward functions are simply sources of gradient updates which designers use in order to control generalization behavior.

Most real-world tasks are partially observable. Although our results only apply to optimal policies in finite mdp s, we expect the key conclusions to generalize. Furthermore, irregular stochasticity in environmental dynamics can make it hard to satisfy Theorem 6.13 ’s similarity requirement. We look forward to future work which addresses partially observable environments, suboptimal policies, or “almost similar” rsd sets.

Past work shows that it would be bad for an agent to disempower humans in its environment. In a two-player agent / human game, minimizing the human’s information-theoretic empowerment [ Salge et al., 2014 ] produces adversarial agent behavior [ Guckelsberger et al., 2018 ] . In contrast, maximizing human empowerment produces helpful agent behavior [ Salge and Polani, 2017 , Guckelsberger et al., 2016 , Du et al., 2020 ] . We do not yet formally understand if, when, or why Power -seeking policies tend to disempower other agents in the environment.

More complex environments probably have more pronounced power-seeking incentives. Intuitively, there are often many ways for power-seeking to be optimal, and relatively few ways for power-seeking not to be optimal. For example, suppose that in some environment, Theorem 6.13 holds for one million involutions ϕ \phi . Does this guarantee more pronounced incentives than if Theorem 6.13 only held for one involution?

We proved sufficient conditions for when reward functions tend to have optimal policies which seek power. In the absence of prior information, one should expect that an arbitrary reward function has optimal policies which exhibit power-seeking behavior under these conditions. However, we have prior information: AI designers usually try to specify a good reward function. Even so, it may be hard to specify orbit elements which do not—at optimum—incentivize bad power-seeking.

##### Societal impact.

We believe that this paper builds toward a rigorous understanding of the risks presented by AI power-seeking incentives. Understanding these risks is the first step in addressing them. However, basic theoretical work can have many consequences. For example, this theory could somehow help future researchers build power-seeking agents which disempower humans. We believe that the benefit of understanding outweighs the potential societal harm.

##### Conclusion.

We developed the first formal theory of the statistical tendencies of optimal policies in reinforcement learning. In the context of mdp s, we proved sufficient conditions under which optimal policies tend to seek power, both formally (by taking Power -seeking actions) and intuitively (by taking actions which keep the agent’s options open). Many real-world environments have symmetries which produce power-seeking incentives. In particular, optimal policies tend to seek power when the agent can be shut down or destroyed. Seeking control over the environment will often involve resisting shutdown, and perhaps monopolizing resources.

We caution that many real-world tasks are partially observable and that learned policies are rarely optimal. Our results do not mathematically prove that hypothetical superintelligent AI agents will seek power. However, we hope that this work will foster thoughtful, serious, and rigorous discussion of this possibility.

## Acknowledgments

Alexander Turner was supported by the Berkeley Existential Risk Initiative and the Long-Term Future Fund. Alexander Turner, Rohin Shah, and Andrew Critch were supported by the Center for Human-Compatible AI. Prasad Tadepalli was supported by the National Science Foundation.

Yousif Almulla, John E. Ball, Daniel Blank, Steve Byrnes, Ryan Carey, Michael Dennis, Scott Emmons, Alan Fern, Daniel Filan, Ben Garfinkel, Adam Gleave, Edouard Harris, Evan Hubinger, DNL Kok, Vanessa Kosoy, Victoria Krakovna, Cassidy Laidlaw, Joel Lehman, David Lindner, Dylan Hadfield-Menell, Richard Möhn, Alexandra Nolan, Matt Olson, Neale Ratzlaff, Adam Shimi, Sam Toyer, Joshua Turner, Cody Wild, Davide Zagami, and our anonymous reviewers provided valuable feedback.

## References

Benson-Tilsen and Soares [2016] Tsvi Benson-Tilsen and Nate Soares. Formalizing convergent instrumental goals. Workshops at the Thirtieth AAAI Conference on Artificial Intelligence , 2016.

Bostrom [2012] Nick Bostrom. The superintelligent will: Motivation and instrumental rationality in advanced artificial agents. Minds and Machines , 22(2):71–85, 2012.

Bostrom [2014] Nick Bostrom. Superintelligence . Oxford University Press, 2014.

Burda et al. [2019] Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Representations , 2019.

Carey [2018] Ryan Carey. Incorrigibility in the CIRL framework. AI, Ethics, and Society , 2018.

Drummond [1998] Chris Drummond. Composing functions to speed up reinforcement learning in a changing world. In Machine Learning: ECML-98 , volume 1398, pages 370–381. Springer, 1998.

Du et al. [2020] Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. AvE: Assistance via empowerment. Advances in Neural Information Processing Systems , 33, 2020.

Foster and Dayan [2002] David Foster and Peter Dayan. Structure in the space of value functions. Machine Learning , pages 325–346, 2002.

Guckelsberger et al. [2016] Christian Guckelsberger, Christoph Salge, and Simon Colton. Intrinsically motivated general companion NPCs via coupled empowerment maximisation. In IEEE Conference on Computational Intelligence and Games , pages 1–8, 2016.

Guckelsberger et al. [2018] Christian Guckelsberger, Christoph Salge, and Julian Togelius. New and surprising ways to be mean. In IEEE Conference on Computational Intelligence and Games , pages 1–8, 2018.

Hadfield-Menell et al. [2017] Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. The off-switch game. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17 , pages 220–227, 2017.

LeCun and Zador [2019] Yann LeCun and Anthony Zador. Don’t fear the Terminator, September 2019. URL https://blogs.scientificamerican.com/observations/dont-fear-the-terminator/ .

Lippman [1968] Steven A Lippman. On the set of optimal policies in discrete dynamic programming. Journal of Mathematical Analysis and Applications , 24(2):440–445, 1968.

McKenzie [1976] Lionel W McKenzie. Turnpike theory. Econometrica: Journal of the Econometric Society , pages 841–865, 1976.

Menache et al. [2002] Ishai Menache, Shie Mannor, and Nahum Shimkin. Q-cut—dynamic discovery of sub-goals in reinforcement learning. In European Conference on Machine Learning , pages 295–306. Springer, 2002.

Milli et al. [2017] Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? In Proceedings of the 26th International Joint Conference on Artificial Intelligence , pages 4754–4760, 2017.

Mitchell [2021] Melanie Mitchell. Why AI is harder than we think. arXiv preprint arXiv:2104.12871 , 2021.

Ng et al. [1999] Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning , pages 278–287. Morgan Kaufmann, 1999.

Olah et al. [2020] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill , 2020.

Omohundro [2008] Stephen Omohundro. The basic AI drives, 2008.

Pathak et al. [2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In ICML , 2017.

Pinker and Russell [2020] Steven Pinker and Stuart Russell. The foundations, benefits, and possible existential threat of AI, June 2020. URL https://futureoflife.org/2020/06/15/steven-pinker-and-stuart-russell-on-the-foundations-benefits-and-possible-existential-risk-of-ai/ .

Puterman [2014] Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Reece and Campbell [2011] J.B. Reece and N.A. Campbell. Campbell Biology . Pearson Australia, 2011.

Regan and Boutilier [2010] Kevin Regan and Craig Boutilier. Robust policy computation in reward-uncertain MDPs using nondominated policies. In Twenty-Fourth AAAI Conference on Artificial Intelligence , 2010.

Russell [2019] Stuart Russell. Human compatible: Artificial intelligence and the problem of control . Viking, 2019.

Russell and Norvig [2009] Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach . Pearson Education Limited, 2009.

Salge and Polani [2017] Christoph Salge and Daniel Polani. Empowerment as replacement for the three laws of robotics. Frontiers in Robotics and AI , 4:25, 2017.

Salge et al. [2014] Christoph Salge, Cornelius Glackin, and Daniel Polani. Empowerment–an introduction. In Guided Self-Organization: Inception , pages 67–114. Springer, 2014.

Sattarov [2019] Faridun Sattarov. Power and technology: a philosophical and ethical analysis . Rowman & Littlefield International, Ltd, 2019.

Schaul et al. [2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning , pages 1312–1320, 2015.

Soares et al. [2015] Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. Corrigibility. AAAI Workshops , 2015.

Sutton and Barto [1998] Richard S Sutton and Andrew G Barto. Reinforcement learning: an introduction . MIT Press, 1998.

Sutton et al. [2011] Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In International Conference on Autonomous Agents and Multiagent Systems , pages 761–768, 2011.

Turner [2022] Alexander Matt Turner. Reward is not the optimization target, 2022. URL https://www.alignmentforum.org/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target .

Turner et al. [2020] Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 385–391, 2020.

Various [2019] Various. Debate on instrumental convergence between LeCun, Russell, Bengio, Zador, and more, 2019. URL https://www.alignmentforum.org/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell .

Wang et al. [2007] Tao Wang, Michael Bowling, and Dale Schuurmans. Dual representations for dynamic programming and reinforcement learning. In International Symposium on Approximate Dynamic Programming and Reinforcement Learning , pages 44–51. IEEE, 2007.

Wang et al. [2008] Tao Wang, Michael Bowling, Dale Schuurmans, and Daniel J Lizotte. Stable dual dynamic programming. In Advances in Neural Information Processing Systems , pages 1569–1576, 2008.

## Appendix A Comparing Power with information-theoretic empowerment

Salge et al. [2014] define information-theoretic empowerment as the maximum possible mutual information between the agent’s actions and the state observations n n steps in the future, written 𝔈 n ​ ( s ) \mathfrak{E}_{n}(s) . This notion requires an arbitrary choice of horizon, failing to account for the agent’s discount rate γ \gamma . “In a discrete deterministic world empowerment reduces to the logarithm of the number of sensor states reachable with the available actions” [ Salge et al., 2014 ] . Figure 9 demonstrates how empowerment can return counterintuitive verdicts with respect to the agent’s control over the future.

Power returns intuitive answers in these situations. lim γ → 1 Power 𝒟 bound ​ ( s 1 , γ ) \lim_{\gamma\to 1}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\gamma\right) converges by Lemma 5.3 . Consider the obvious involution ϕ \phi which takes each state in fig. 9(b) to its counterpart in fig. 9(c) . Since ϕ ⋅ ℱ nd ⁡ ( s 3 ) ⊊ ℱ nd ⁡ ( s 4 ) = ℱ ⁡ ( s 4 ) \phi\cdot\Fndop({\color[rgb]{0.25,0.45,0.75}s_{3}})\subsetneq\Fndop({\color[rgb]{0.25,0.45,0.75}s_{4}})=\Fop({\color[rgb]{0.25,0.45,0.75}s_{4}}) , Proposition 6.6 proves that ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s 3 , γ ) ≤ most : 𝔇 bound Power 𝒟 bound ( s 4 , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{3}},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{4}},\gamma\right) , with the proof of Proposition 6.6 showing strict inequality under all 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} when γ ∈ ( 0 , 1 ) \gamma\in(0,1) .

Empowerment can be adjusted to account for these cases, perhaps by considering the channel capacity between the agent’s actions and the state trajectories induced by stationary policies. However, since Power is formulated in terms of optimal value, we believe that Power is better suited for mdp s than information-theoretic empowerment is.

## Appendix B Seeking Power can be a detour

###### Remark .

The results of appendix E do not depend on this section’s results.

One might suspect that optimal policies tautologically tend to seek Power . This intuition is wrong.

###### Proposition B.1 (Greater Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} does not imply greater ℙ 𝒟 bound \Prb_{\mathcal{D}_{\text{bound}}} ) .

Action a a seeking more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} than a ′ a^{\prime} at state s s and γ \gamma does not imply that ℙ 𝒟 bound ( s , a , γ ) ≥ ℙ 𝒟 bound ( s , a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{bound}}}\left(s,a,\gamma\right)\geq\Prb\nolimits_{\mathcal{D}_{\text{bound}}}\left(s,a^{\prime},\gamma\right) .

###### Proof.

Consider the environment of fig. 10 . Let X u ≔ unif ​ ( 0 , 1 ) X_{u}\coloneqq\text{unif}(0,1) , and consider 𝒟 X u ​ - iid \mathcal{D}_{X_{u}\text{-}\textsc{iid}} , which has bounded support. Direct computation 7 7 7 In small deterministic mdp s, the Power and optimality probability of the maximum-entropy reward function distribution can be computed using https://github.com/loganriggs/Optimal-Policies-Tend-To-Seek-Power . of the Power expectation ( definition 5.2 ) yields Power 𝒟 X u ​ - iid ​ ( s 2 , 1 ) = 3 4 > 2 3 = Power 𝒟 X u ​ - iid ​ ( s 3 , 1 ) \text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left(s_{2},1\right)=\frac{3}{4}>\frac{2}{3}=\text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left(s_{3},1\right) . Therefore, N seeks more Power 𝒟 X u ​ - iid \text{{Power}}_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}} than NE at state s 1 {\color[rgb]{0.25,0.45,0.75}s_{1}} and γ = 1 \gamma=1 .

However, ℙ 𝒟 X u ​ - iid ( s 1 , N , 1 ) = 1 3 < 2 3 = ℙ 𝒟 X u ​ - iid ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)=\frac{1}{3}<\frac{2}{3}=\Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) . ∎

###### Lemma B.2 (Fraction of orbits which agree on weak optimality) .

Let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) , and suppose f 1 , f 2 : Δ ⁡ ( ℝ | 𝒮 | ) → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Delta(\mathbb{R}^{\left|\mathcal{S}\right|})\to\mathbb{R} are such that f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) . Then for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) } | | S | 𝒮 | ⋅ 𝒟 | ≥ 1 2 \frac{\left|\left\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime})\right\}\right|}{\left|S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\right|}\geq\dfrac{1}{2} .

###### Proof.

All 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} such that f 1 ​ ( 𝒟 ′ ) = f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})=f_{2}(\mathcal{D}^{\prime}) satisfy f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) .

Otherwise, consider the 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} such that f 1 ​ ( 𝒟 ′ ) ≠ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\neq f_{2}(\mathcal{D}^{\prime}) . By the definition of ≥ most \geq_{\text{{most}}} ( definition 6.5 ), at least 1 2 \frac{1}{2} of these 𝒟 ′ \mathcal{D}^{\prime} satisfy f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime}) , in which case f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) . Then the desired inequality follows. ∎

###### Lemma B.3 ( ≥ most \geq_{\text{most}} and trivial orbits) .

Let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) and suppose f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) . For all reward function distributions 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} with one-element orbits, f 1 ​ ( 𝒟 ) ≥ f 2 ​ ( 𝒟 ) f_{1}(\mathcal{D})\geq f_{2}(\mathcal{D}) . In particular, 𝒟 \mathcal{D} has a one-element orbit when it distributes reward identically and independently ( iid ) across states.

###### Proof.

By Lemma B.2 , at least half of the elements 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 \mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} satisfy f 1 ​ ( 𝒟 ′ ) ≥ f 2 ​ ( 𝒟 ′ ) f_{1}(\mathcal{D}^{\prime})\geq f_{2}(\mathcal{D}^{\prime}) . But | S | 𝒮 | ⋅ 𝒟 | = 1 \left|S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\right|=1 , and so f 1 ​ ( 𝒟 ) ≥ f 2 ​ ( 𝒟 ) f_{1}(\mathcal{D})\geq f_{2}(\mathcal{D}) must hold.

If 𝒟 \mathcal{D} is iid , it has a one-element orbit due to the assumed identical distribution of reward. ∎

###### Proposition B.4 (Actions which tend to seek Power do not necessarily tend to be optimal) .

Action a a tending to seek more Power than a ′ a^{\prime} at state s s and γ \gamma does not imply that ℙ 𝒟 any ( s , a , γ ) ≥ most : 𝔇 any ℙ 𝒟 any ( s , a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right) .

###### Proof.

Consider the environment of fig. 10 . Since RSD nd ​ ( s 3 ) ⊊ RSD ​ ( s 2 ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s_{3}\right)\subsetneq\text{{RSD}}\left(s_{2}\right) , Proposition 6.12 shows that Power 𝒟 bound ( s 2 , 1 ) ≥ most : 𝔇 bound Power 𝒟 bound ( s 3 , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},1\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{3},1\right) via s ′ ≔ s 3 , s ≔ s 2 , ϕ s^{\prime}\coloneqq s_{3},s\coloneqq s_{2},\phi the identity permutation (which is an involution). Therefore, N tends to seek more Power than NE at state s 1 {\color[rgb]{0.25,0.45,0.75}s_{1}} and γ = 1 \gamma=1 .

If ℙ 𝒟 any ( s 1 , N , 1 ) ≥ most : 𝔇 any ℙ 𝒟 any ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) , then Lemma B.3 shows that ℙ 𝒟 X ​ - iid ( s 1 , N , 1 ) ≥ ℙ 𝒟 X ​ - iid ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)\geq\Prb\nolimits_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) for all 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} . But the proof of Proposition B.1 showed that ℙ 𝒟 X u ​ - iid ( s 1 , N , 1 ) < ℙ 𝒟 X u ​ - iid ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)<\Prb\nolimits_{\mathcal{D}_{X_{u}\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) for X u ≔ unif ​ ( 0 , 1 ) X_{u}\coloneqq\text{unif}(0,1) . Therefore, it cannot be true that ℙ 𝒟 any ( s 1 , N , 1 ) ≥ most : 𝔇 any ℙ 𝒟 any ( s 1 , NE , 1 ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{N},1\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left({\color[rgb]{0.25,0.45,0.75}s_{1}},\texttt{NE},1\right) . ∎

## Appendix C Sub-optimal Power

In certain situations, Power returns intuitively surprising verdicts. There exists a policy under which the reader chooses a winning lottery ticket, but it seems wrong to say that the reader has the power to win the lottery with high probability. For various reasons, humans and other bounded agents are generally incapable of computing optimal policies for arbitrary objectives. More formally, consider the rewardless mdp of fig. 11 .

Consider a model-based RL agent with black-box simulator access to this environment. The agent has no prior information about the model, and so it acts randomly. Before long, the agent has probably learned how to navigate from s 0 {\color[rgb]{0.25,0.45,0.75}s_{0}} to states s ℓ s_{\ell} , s r s_{r} , s 1 s_{1} , s 2 s_{2} , and s 5 s_{5} . However, over any reasonable timescale, it is extremely improbable that the agent discovers the two actions respectively leading to s 3 s_{3} and s 4 s_{4} .

Even provided with a reward function R R and the discount rate γ \gamma , the agent has yet to learn the relevant environmental dynamics, and so many of its policies are far from optimal. Although Proposition 6.6 shows that ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s ℓ , γ ) ≤ most : 𝔇 bound Power 𝒟 bound ( s r , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{\ell},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{r},\gamma\right) , there is a sense in which s ℓ s_{\ell} gives this agent more power.

We formalize a bounded agent’s goal-achievement capabilities with a function pol , which takes as input a reward function and a discount rate, and returns a policy. Informally, this is the best policy which the agent knows about. We can then calculate Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} with respect to pol .

###### Definition C.1 (Suboptimal Power ) .

Let Π Δ \Pi_{\Delta} be the set of stationary stochastic policies, and let pol : ℝ 𝒮 × [ 0 , 1 ] → Π Δ \text{pol}\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{\mathcal{S}}\times[0,1]\to\Pi_{\Delta} . For γ ∈ [ 0 , 1 ] \gamma\in[0,1] , Power 𝒟 bound pol ​ ( s , γ ) ≔ 𝔼 R ∼ 𝒟 bound , a ∼ pol ​ ( R , γ ) ​ ( s ) , s ′ ∼ T ⁡ ( s , a ) [ lim γ ∗ → γ ( 1 − γ ∗ ) ​ V R pol ​ ( R , γ ) ​ ( s ′ , γ ∗ ) ] . \displaystyle\text{{Power}}^{\text{pol}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\coloneqq\opE_{\begin{subarray}{c}R\sim\mathcal{D}_{\text{bound}},\\ a\sim\text{pol}\left(R,\gamma\right)(s),\\ s^{\prime}\sim T\left(s,a\right)\end{subarray}}\left[\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{\text{pol}\left(R,\gamma\right)}_{R}\left(s^{\prime},\gamma^{*}\right)\right]. (5)

By Lemma E.36 , Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} is the special case where ∀ R ∈ ℝ 𝒮 , γ ∈ [ 0 , 1 ] : pol ​ ( R , γ ) ∈ Π ∗ ​ ( R , γ ) \forall R\in\mathbb{R}^{\mathcal{S}},\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{pol}\left(R,\gamma\right)\in\Pi^{*}\left(R,\gamma\right) . We define Power 𝒟 bound pol \text{{Power}}^{\text{pol}}_{\mathcal{D}_{\text{bound}}} -seeking similarly as in definition 5.6 .

Power 𝒟 bound pol ​ ( s 0 , 1 ) \text{{Power}}^{\text{pol}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{0}},1\right) increases as the policies returned by pol are improved. We illustrate this by considering the 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} case.

pol 1 \text{pol}_{1} The model is initially unknown, and so ∀ R , γ : pol 1 ​ ( R , γ ) \forall R,\gamma\mathrel{\mathop{\ordinarycolon}}\text{pol}_{1}(R,\gamma) is a uniformly random policy. Since pol 1 \text{pol}_{1} is constant on its inputs, Power 𝒟 X ​ - iid pol 1 ​ ( s 0 , 1 ) = 𝔼 [ X ] \text{{Power}}^{\text{pol}_{1}}_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left({\color[rgb]{0.25,0.45,0.75}s_{0}},1\right)=\opE\left[X\right] by the linearity of expectation and the fact that 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} distributes reward independently and identically across states.

pol 2 \text{pol}_{2} The agent knows the dynamics, except that it does not know how to reach s 3 s_{3} or s 4 s_{4} . At this point, pol 2 ​ ( R , 1 ) \text{pol}_{2}(R,1) navigates from s 0 {\color[rgb]{0.25,0.45,0.75}s_{0}} to the average-optimal choice among three terminal states: s 1 s_{1} , s 2 s_{2} , and s 5 s_{5} . Therefore, Power 𝒟 bound pol 2 ​ ( s 0 , 1 ) = 𝔼 [ max ⁡ of ​ 3 ​ draws from ​ X ] \text{{Power}}^{\text{pol}_{2}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{0}},1\right)=\opE\left[\max\text{ of }3\text{ draws from }X\right] .

pol 3 \text{pol}_{3} The agent knows the dynamics, the environment is small enough to solve explicitly, and so ∀ R , γ : pol 3 ​ ( R , γ ) \forall R,\gamma\mathrel{\mathop{\ordinarycolon}}\text{pol}_{3}(R,\gamma) is an optimal policy. pol 3 ​ ( R , 1 ) \text{pol}_{3}(R,1) navigates from s 0 {\color[rgb]{0.25,0.45,0.75}s_{0}} to the average-optimal choice among all five terminal states. Therefore, Power 𝒟 bound pol 3 ​ ( s 0 , 1 ) = 𝔼 [ max ⁡ of ​ 5 ​ draws from ​ X ] \text{{Power}}^{\text{pol}_{3}}_{\mathcal{D}_{\text{bound}}}\left({\color[rgb]{0.25,0.45,0.75}s_{0}},1\right)=\opE\left[\max\text{ of }5\text{ draws from }X\right] .

As the agent learns more about the environment and improves pol , the agent’s Power 𝒟 bound pol \text{{Power}}^{\text{pol}}_{\mathcal{D}_{\text{bound}}} increases. The agent seeks Power 𝒟 bound pol 2 \text{{Power}}^{\text{pol}_{2}}_{\mathcal{D}_{\text{bound}}} by navigating to s ℓ s_{\ell} instead of s r s_{r} , but seeks more Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} by navigating to s r s_{r} instead of s ℓ s_{\ell} . Intuitively, bounded agents gain power by improving pol and by formally seeking Power 𝒟 bound pol \text{{Power}}^{\text{pol}}_{\mathcal{D}_{\text{bound}}} within the environment.

## Appendix D Lists of results

\ignoretheorems cor-no-num,conjecture,remark,definition

### D.1 Contributions of independent interest

We developed new basic mdp theory by exploring the structural properties of visit distribution functions. Echoing Wang et al. [2007] , Wang et al. [2008] , we believe that this area is interesting and underexplored.

#### D.1.1 Optimal value theory

Lemma E.38 shows that f ⁡ ( γ ∗ ) ≔ lim γ ∗ → γ ( 1 − γ ∗ ) ​ V R ∗ ​ ( s , γ ∗ ) f(\gamma^{*})\coloneqq\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{*}_{R}\left(s,\gamma^{*}\right) is Lipschitz continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] , with Lipschitz constant depending only on ‖ R ‖ 1 \left\lVert R\right\rVert_{1} . For all states s s and policies π ∈ Π \pi\in\Pi , Corollary E.5 shows that V R π ​ ( s , γ ) V^{\pi}_{R}(s,\gamma) is rational on γ \gamma .

Optimal value has a well-known dual formulation: V R ∗ ​ ( s , γ ) = max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 V^{*}_{R}\left(s,\gamma\right)=\max_{\mathbf{f}\in\Fop(s)}\mathbf{f}(\gamma)^{\top}\mathbf{r} . {restatable*} [ ∀ γ ∈ [ 0 , 1 ) : V R ∗ ​ ( s , γ ) = max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 \forall\gamma\in[0,1)\mathrel{\mathop{\ordinarycolon}}V^{*}_{R}\left(s,\gamma\right)=\max_{\mathbf{f}\in\Fndop(s)}\mathbf{f}(\gamma)^{\top}\mathbf{r} ]lemoptVfFndRestrict In a fixed rewardless mdp , section D.1.1 may enable more efficient computation of optimal value functions for multiple reward functions.

#### D.1.2 Optimal policy theory

Section D.1.2 demonstrates how to preserve optimal incentives while changing the discount rate.

{restatable*} [How to transfer optimal policy sets across discount rates]proptransferDiscount Suppose reward function R R has optimal policy set Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) at discount rate γ ∈ ( 0 , 1 ) \gamma\in(0,1) . For any γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) , we can construct a reward function R ′ R^{\prime} such that Π ∗ ​ ( R ′ , γ ∗ ) = Π ∗ ​ ( R , γ ) \Pi^{*}\left(R^{\prime},\gamma^{*}\right)=\Pi^{*}\left(R,\gamma\right) . Furthermore, V R ′ ∗ ​ ( ⋅ , γ ∗ ) = V R ∗ ​ ( ⋅ , γ ) V^{*}_{R^{\prime}}\left(\cdot,\gamma^{*}\right)=V^{*}_{R}\left(\cdot,\gamma\right) .

#### D.1.3 Visit distribution theory

While Regan and Boutilier [2010] consider a visit distribution function 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) to be non-dominated if it is optimal for some reward function in a set ℛ ⊆ ℝ | 𝒮 | \mathcal{R}_{\text{}}\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} , our stricter definition 3.6 considers 𝐟 \mathbf{f} to be non-dominated when ∃ 𝐫 ∈ ℝ | 𝒮 | , γ ∈ ( 0 , 1 ) : 𝐟 ​ ( γ ) ⊤ ​ 𝐫 > max 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 \exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|},\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\prime}\in\Fop(s)\setminus\left\{\mathbf{f}\right\}}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r} .

## Appendix E Theoretical results

###### Lemma E.1 (A policy is optimal iff it induces an optimal visit distribution at every state) .

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) and let R R be a reward function. π ∈ Π ∗ ​ ( R , γ ) \pi\in\Pi^{*}\left(R,\gamma\right) iff π \pi induces an optimal visit distribution at every state.

###### Proof.

By definition, a policy π \pi is optimal iff π \pi induces the maximal on-policy value at each state, which is true iff π \pi induces an optimal visit distribution at every state (by the dual formulation of optimal value functions). ∎

###### Definition E.2 (Transition matrix induced by a policy) .

𝐓 π \mathbf{T}^{\pi} is the transition matrix induced by policy π ∈ Π \pi\in\Pi , where 𝐓 π ​ 𝐞 s ≔ T ⁡ ( s , π ⁡ ( s ) ) \mathbf{T}^{\pi}\mathbf{e}_{s}\coloneqq T(s,\pi(s)) . ( 𝐓 π ) t ​ 𝐞 s (\mathbf{T}^{\pi})^{t}\mathbf{e}_{s} gives the probability distribution over the states visited at time step t t , after following π \pi for t t steps from s s .

###### Proposition E.3 (Properties of visit distribution functions) .

Let s , s ′ ∈ 𝒮 , 𝐟 π , s ∈ ℱ ⁡ ( s ) s,s^{\prime}\in\mathcal{S},\mathbf{f}^{\pi,s}\in\Fop(s) . 1. 𝐟 π , s ​ ( γ ) \mathbf{f}^{\pi,s}(\gamma) is element-wise non-negative and element-wise monotonically increasing on γ ∈ [ 0 , 1 ) \gamma\in[0,1) .

2. ∀ γ ∈ [ 0 , 1 ) : ‖ 𝐟 π , s ​ ( γ ) ‖ 1 = 1 1 − γ \forall\gamma\in[0,1)\mathrel{\mathop{\ordinarycolon}}\left\lVert\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1}=\frac{1}{1-\gamma} .

###### Proof.

Item 1 : by examination of definition 3.3 , 𝐟 π , s = ∑ t = 0 ∞ ( γ ​ 𝐓 π ) t ​ 𝐞 s \mathbf{f}^{\pi,s}=\sum_{t=0}^{\infty}\left(\gamma\mathbf{T}^{\pi}\right)^{t}\mathbf{e}_{s} . Since each ( 𝐓 π ) t \left(\mathbf{T}^{\pi}\right)^{t} is left stochastic and 𝐞 s \mathbf{e}_{s} is the standard unit vector, each entry in each summand is non-negative. Therefore, ∀ γ ∈ [ 0 , 1 ) : 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐞 s ′ ≥ 0 \forall\gamma\in[0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}}\geq 0 , and this function monotonically increases on γ \gamma .

Item 2 : ‖ 𝐟 π , s ​ ( γ ) ‖ 1 \displaystyle\left\lVert\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1} = ‖ ∑ t = 0 ∞ ( γ ​ 𝐓 π ) t ​ 𝐞 s ‖ 1 \displaystyle=\left\lVert\sum_{t=0}^{\infty}\left(\gamma\mathbf{T}^{\pi}\right)^{t}\mathbf{e}_{s}\right\rVert_{1} (6) = ∑ t = 0 ∞ γ t ​ ‖ ( 𝐓 π ) t ​ 𝐞 s ‖ 1 \displaystyle=\sum_{t=0}^{\infty}\gamma^{t}\left\lVert\left(\mathbf{T}^{\pi}\right)^{t}\mathbf{e}_{s}\right\rVert_{1} (7) = ∑ t = 0 ∞ γ t \displaystyle=\sum_{t=0}^{\infty}\gamma^{t} (8) = 1 1 − γ . \displaystyle=\frac{1}{1-\gamma}. (9)

Equation 7 follows because all entries in each ( 𝐓 π ) t ​ 𝐞 s \left(\mathbf{T}^{\pi}\right)^{t}\mathbf{e}_{s} are non-negative by item 1 . Equation 8 follows because each ( 𝐓 π ) t \left(\mathbf{T}^{\pi}\right)^{t} is left stochastic and 𝐞 s \mathbf{e}_{s} is a stochastic vector, and so ‖ ( 𝐓 π ) t ​ 𝐞 s ‖ 1 = 1 \left\lVert\left(\mathbf{T}^{\pi}\right)^{t}\mathbf{e}_{s}\right\rVert_{1}=1 . ∎

###### Lemma E.4 ( 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) is multivariate rational on γ \gamma ) .

𝐟 π ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi}\in\Fop(s) is a multivariate rational function on γ ∈ [ 0 , 1 ) \gamma\in[0,1) .

###### Proof.

Let 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} and consider 𝐟 π ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi}\in\Fop(s) . Let 𝐯 R π \mathbf{v}^{\pi}_{R} be the V R ∗ ​ ( s , γ ) V^{*}_{R}\left(s,\gamma\right) function in column vector form, with one entry per state value.

By the Bellman equations, 𝐯 R π = ( 𝐈 − γ ​ 𝐓 π ) − 1 ​ 𝐫 . \mathbf{v}^{\pi}_{R}=\left(\mathbf{I}-\gamma\mathbf{T}^{\pi}\right)^{-1}\mathbf{r}. Let 𝐀 γ ≔ ( 𝐈 − γ ​ 𝐓 π ) − 1 \mathbf{A}_{\gamma}\coloneqq\left(\mathbf{I}-\gamma\mathbf{T}^{\pi}\right)^{-1} , and for state s s , form 𝐀 s , γ \mathbf{A}_{s,\gamma} by replacing 𝐀 γ \mathbf{A}_{\gamma} ’s column for state s s with 𝐫 \mathbf{r} . As noted by Lippman [1968] , by Cramer’s rule, V R π ​ ( s , γ ) = det 𝐀 s , γ det 𝐀 γ V^{\pi}_{R}(s,\gamma)=\frac{\det{\mathbf{A}_{s,\gamma}}}{\det\mathbf{A}_{\gamma}} is a rational function with numerator and denominator having degree at most | 𝒮 | \left|\mathcal{S}\right| .

In particular, for each state indicator reward function 𝐞 s i \mathbf{e}_{s_{i}} , V s i π ​ ( s , γ ) = 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐞 s i V^{\pi}_{s_{i}}(s,\gamma)=\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s_{i}} is a rational function of γ \gamma whose numerator and denominator each have degree at most | 𝒮 | \left|\mathcal{S}\right| . This implies that 𝐟 π ​ ( γ ) \mathbf{f}^{\pi}(\gamma) is multivariate rational on γ ∈ [ 0 , 1 ) \gamma\in[0,1) . ∎

###### Corollary E.5 (On-policy value is rational on γ \gamma ) .

Let π ∈ Π \pi\in\Pi and R R be any reward function. V R π ​ ( s , γ ) V^{\pi}_{R}(s,\gamma) is rational on γ ∈ [ 0 , 1 ) \gamma\in[0,1) .

###### Proof.

V R π ​ ( s , γ ) = 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐫 V^{\pi}_{R}(s,\gamma)=\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r} , and 𝐟 \mathbf{f} is a multivariate rational function of γ \gamma by Lemma E.4 . Therefore, for fixed 𝐫 \mathbf{r} , 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐫 \mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r} is a rational function of γ \gamma . ∎

### E.1 Non-dominated visit distribution functions

###### Definition E.6 (Continuous reward function distribution) .

Results with 𝒟 cont \mathcal{D}_{\text{cont}} hold for any absolutely continuous reward function distribution.

###### Remark .

We assume ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} is endowed with the standard topology.

###### Lemma E.7 (Distinct linear functionals disagree almost everywhere on their domains) .

Let 𝐱 , 𝐱 ′ ∈ ℝ | 𝒮 | \mathbf{x},\mathbf{x}^{\prime}\in\mathbb{R}^{\left|\mathcal{S}\right|} be distinct. ℙ 𝐫 ∼ 𝒟 cont ( 𝐱 ⊤ ​ 𝐫 = 𝐱 ′ ⁣ ⊤ ​ 𝐫 ) = 0 \Prb\nolimits_{\mathbf{r}\sim\mathcal{D}_{\text{cont}}}\left(\mathbf{x}^{\top}\mathbf{r}=\mathbf{x}^{\prime\top}\mathbf{r}\right)=0 .

###### Proof.

{ 𝐫 ∈ ℝ | 𝒮 | ∣ ( 𝐱 − 𝐱 ′ ) ⊤ ​ 𝐫 = 0 } \left\{\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mid(\mathbf{x}-\mathbf{x}^{\prime})^{\top}\mathbf{r}=0\right\} is a hyperplane since 𝐱 − 𝐱 ′ ≠ 𝟎 \mathbf{x}-\mathbf{x}^{\prime}\neq\mathbf{0} . Therefore, it has no interior in the standard topology on ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} . Since this empty-interior set is also convex, it has zero Lebesgue measure. By the Radon-Nikodym theorem, it has zero measure under any continuous distribution 𝒟 cont \mathcal{D}_{\text{cont}} . ∎

###### Corollary E.8 (Unique maximization of almost all vectors) .

Let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. ℙ 𝐫 ∼ 𝒟 cont ( | arg ​ max 𝐱 ′′ ∈ X 𝐱 ′ ′ ⊤ 𝐫 | > 1 ) = 0 \Prb\nolimits_{\mathbf{r}\sim\mathcal{D}_{\text{cont}}}\left(\left|\argmax_{\mathbf{x}^{\prime\prime}\in X}\mathbf{x}^{\prime\prime\top}\mathbf{r}\right|>1\right)=0 .

###### Proof.

Let 𝐱 , 𝐱 ′ ∈ X \mathbf{x},\mathbf{x}^{\prime}\in X be distinct. For any 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , 𝐱 , 𝐱 ′ ∈ arg ​ max 𝐱 ′′ ∈ X 𝐱 ′ ′ ⊤ 𝐫 \mathbf{x},\mathbf{x}^{\prime}\in\argmax_{\mathbf{x}^{\prime\prime}\in X}\mathbf{x}^{\prime\prime\top}\mathbf{r} iff 𝐱 ⊤ 𝐫 = 𝐱 ′ ⁣ ⊤ 𝐫 ≥ max 𝐱 ′′ ∈ X ∖ { 𝐱 , 𝐱 ′ } 𝐱 ′ ′ ⊤ 𝐫 \mathbf{x}^{\top}\mathbf{r}=\mathbf{x}^{\prime\top}\mathbf{r}\geq\max_{\mathbf{x}^{\prime\prime}\in X\setminus\left\{\mathbf{x},\mathbf{x}^{\prime}\right\}}\mathbf{x}^{\prime\prime\top}\mathbf{r} . By Lemma E.7 , 𝐱 ⊤ ​ 𝐫 = 𝐱 ′ ⁣ ⊤ ​ 𝐫 \mathbf{x}^{\top}\mathbf{r}=\mathbf{x}^{\prime\top}\mathbf{r} holds with probability 0 under any 𝒟 cont \mathcal{D}_{\text{cont}} . ∎

#### E.1.1 Generalized non-domination results

Our formalism includes both ℱ nd ⁡ ( s ) \Fndop(s) and RSD nd ​ ( s ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) ; we therefore prove results that are applicable to both.

###### Definition E.9 (Non-dominated linear functionals) .

Let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. ND ​ ( X ) ≔ { 𝐱 ∈ X ∣ ∃ 𝐫 ∈ ℝ | 𝒮 | : 𝐱 ⊤ ​ 𝐫 > max 𝐱 ′ ∈ X ∖ { 𝐱 } ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 } \text{{ND}}\left(X\right)\coloneqq\left\{\mathbf{x}\in X\mid\exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{r}>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}\mathbf{x}^{\prime\top}\mathbf{r}\right\} .

###### Lemma E.10 (All vectors are maximized by a non-dominated linear functional) .

Let 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} and let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite and non-empty. ∃ 𝐱 ∗ ∈ ND ​ ( X ) : 𝐱 ∗ ⁣ ⊤ ​ 𝐫 = max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 \exists\mathbf{x}^{*}\in\text{{ND}}\left(X\right)\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{*\top}\mathbf{r}=\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r} .

###### Proof.

Let A ⁡ ( 𝐫 ∣ X ) ≔ arg ​ max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 = { 𝐱 1 , … , 𝐱 n } A(\mathbf{r}\mid X)\coloneqq\argmax_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}=\left\{\mathbf{x}_{1},\ldots,\mathbf{x}_{n}\right\} . Then 𝐱 1 ⊤ ​ 𝐫 = ⋯ = 𝐱 n ⊤ ​ 𝐫 > max 𝐱 ′ ∈ X ∖ A ⁡ ( 𝐫 ∣ X ) ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 . \displaystyle\mathbf{x}_{1}^{\top}\mathbf{r}=\cdots=\mathbf{x}_{n}^{\top}\mathbf{r}>\max_{\mathbf{x}^{\prime}\in X\setminus A(\mathbf{r}\mid X)}\mathbf{x}^{\prime\top}\mathbf{r}. (10)

In eq. 10 , each 𝐱 ⊤ ​ 𝐫 \mathbf{x}^{\top}\mathbf{r} expression is linear on 𝐫 \mathbf{r} . The max \max is piecewise linear on 𝐫 \mathbf{r} since it is the maximum of a finite set of linear functionals. In particular, all expressions in eq. 10 are continuous on 𝐫 \mathbf{r} , and so we can find some δ > 0 \delta>0 neighborhood B ⁡ ( 𝐫 , δ ) B(\mathbf{r},\delta) such that ∀ 𝐫 ′ ∈ B ⁡ ( 𝐫 , δ ) : max 𝐱 i ∈ A ⁡ ( 𝐫 ∣ X ) ⁡ 𝐱 i ⊤ ​ 𝐫 ′ > max 𝐱 ′ ∈ X ∖ A ⁡ ( 𝐫 ∣ X ) ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 ′ \forall\mathbf{r}^{\prime}\in B(\mathbf{r},\delta)\mathrel{\mathop{\ordinarycolon}}\max_{\mathbf{x}_{i}\in A(\mathbf{r}\mid X)}\mathbf{x}_{i}^{\top}\mathbf{r}^{\prime}>\max_{\mathbf{x}^{\prime}\in X\setminus A(\mathbf{r}\mid X)}\mathbf{x}^{\prime\top}\mathbf{r}^{\prime} .

But almost all 𝐫 ′ ∈ B ⁡ ( 𝐫 , δ ) \mathbf{r}^{\prime}\in B(\mathbf{r},\delta) are maximized by a unique functional 𝐱 ∗ \mathbf{x}^{*} by Corollary E.8 ; in particular, at least one such 𝐫 ′′ \mathbf{r}^{\prime\prime} exists. Formally, ∃ 𝐫 ′′ ∈ B ⁡ ( 𝐫 , δ ) : 𝐱 ∗ ⁣ ⊤ ​ 𝐫 ′′ > max 𝐱 ′ ∈ X ∖ { 𝐱 ∗ } ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 ′′ \exists\mathbf{r}^{\prime\prime}\in B(\mathbf{r},\delta)\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{*\top}\mathbf{r}^{\prime\prime}>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}^{*}\right\}}\mathbf{x}^{\prime\top}\mathbf{r}^{\prime\prime} . Therefore, 𝐱 ∗ ∈ ND ​ ( X ) \mathbf{x}^{*}\in\text{{ND}}\left(X\right) by definition E.9 .

𝐱 ∗ ⁣ ⊤ ​ 𝐫 ′ ≥ max 𝐱 i ∈ A ⁡ ( 𝐫 ∣ X ) ⁡ 𝐱 i ⊤ ​ 𝐫 ′ > max 𝐱 ′ ∈ X ∖ A ⁡ ( 𝐫 ∣ X ) ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 ′ \mathbf{x}^{*\top}\mathbf{r}^{\prime}\geq\max_{\mathbf{x}_{i}\in A(\mathbf{r}\mid X)}\mathbf{x}_{i}^{\top}\mathbf{r}^{\prime}>\max_{\mathbf{x}^{\prime}\in X\setminus A(\mathbf{r}\mid X)}\mathbf{x}^{\prime\top}\mathbf{r}^{\prime} , with the strict inequality following because 𝐫 ′′ ∈ B ⁡ ( 𝐫 , δ ) \mathbf{r}^{\prime\prime}\in B(\mathbf{r},\delta) . These inequalities imply that 𝐱 ∗ ∈ A ⁡ ( 𝐫 ∣ X ) \mathbf{x}^{*}\in A(\mathbf{r}\mid X) . ∎

###### Corollary E.11 (Maximal value is invariant to restriction to non-dominated functionals) .

Let 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} and let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 = max 𝐱 ∈ ND ​ ( X ) ⁡ 𝐱 ⊤ ​ 𝐫 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}=\max_{\mathbf{x}\in\text{{ND}}\left(X\right)}\mathbf{x}^{\top}\mathbf{r} .

###### Proof.

If X X is empty, holds trivially. Otherwise, apply Lemma E.10 . ∎

###### Lemma E.12 (How non-domination containment affects optimal value) .

Let 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} and let X , X ′ ⊊ ℝ | 𝒮 | X,X^{\prime}\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. 1. If ND ​ ( X ) ⊆ X ′ \text{{ND}}\left(X\right)\subseteq X^{\prime} , then max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≤ max 𝐱 ′ ∈ X ′ ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\leq\max_{\mathbf{x}^{\prime}\in X^{\prime}}\mathbf{x}^{\prime\top}\mathbf{r} .

2. If ND ​ ( X ) ⊆ X ′ ⊆ X \text{{ND}}\left(X\right)\subseteq X^{\prime}\subseteq X , then max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 = max 𝐱 ′ ∈ X ′ ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}=\max_{\mathbf{x}^{\prime}\in X^{\prime}}\mathbf{x}^{\prime\top}\mathbf{r} .

###### Proof.

Item 1 : max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 \displaystyle\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r} = max 𝐱 ∈ ND ​ ( X ) ⁡ 𝐱 ⊤ ​ 𝐫 \displaystyle=\max_{\mathbf{x}\in\text{{ND}}\left(X\right)}\mathbf{x}^{\top}\mathbf{r} (11) ≤ max 𝐱 ′ ∈ X ′ ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 . \displaystyle\leq\max_{\mathbf{x}^{\prime}\in X^{\prime}}\mathbf{x}^{\prime\top}\mathbf{r}. (12) Equation 11 follows by Corollary E.11 . Equation 12 follows because ND ​ ( X ) ⊆ X ′ \text{{ND}}\left(X\right)\subseteq X^{\prime} .

Item 2 : by item 1 , max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≤ max 𝐱 ′ ∈ X ′ ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\leq\max_{\mathbf{x}^{\prime}\in X^{\prime}}\mathbf{x}^{\prime\top}\mathbf{r} . Since X ′ ⊆ X X^{\prime}\subseteq X , we also have max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐱 ′ ∈ X ′ ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{x}^{\prime}\in X^{\prime}}\mathbf{x}^{\prime\top}\mathbf{r} , and so equality must hold. ∎

###### Definition E.13 (Non-dominated vector functions) .

Let I ⊆ ℝ I\subseteq\mathbb{R} and let F ⊊ ( ℝ | 𝒮 | ) I F\subsetneq\left(\mathbb{R}^{\left|\mathcal{S}\right|}\right)^{I} be a finite set of vector-valued functions on I I . ND ( F ) ≔ { 𝐟 ∈ F ∣ ∃ γ ∈ I , 𝐫 ∈ ℝ | 𝒮 | : 𝐟 ( γ ) ⊤ 𝐫 > max 𝐟 ′ ∈ F ∖ { 𝐟 } 𝐟 ′ ( γ ) ⊤ 𝐫 } \text{{ND}}\left(F\right)\coloneqq\left\{\mathbf{f}\in F\mid\exists\gamma\in I,\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbf{f}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\prime}\in F\setminus\left\{\mathbf{f}\right\}}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}\right\} .

###### Remark .

ℱ nd ⁡ ( s ) = ND ​ ( ℱ ⁡ ( s ) ) \Fndop(s)=\text{{ND}}\left(\Fop(s)\right) by definition 3.6 .

###### Definition E.14 (Affine transformation of visit distribution sets) .

For notational convenience, we define set-scalar multiplication and set-vector addition on X ⊆ ℝ | 𝒮 | X\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} : for c ∈ ℝ c\in\mathbb{R} , c ​ X ≔ { c ​ 𝐱 ∣ 𝐱 ∈ X } cX\coloneqq\left\{c\mathbf{x}\mid\mathbf{x}\in X\right\} . For 𝐚 ∈ ℝ | 𝒮 | \mathbf{a}\in\mathbb{R}^{\left|\mathcal{S}\right|} , X + 𝐚 ≔ { 𝐱 + 𝐚 ∣ 𝐱 ∈ X } X+\mathbf{a}\coloneqq\left\{\mathbf{x}+\mathbf{a}\mid\mathbf{x}\in X\right\} . Similar operations hold when X X is a set of vector functions ℝ ↦ ℝ | 𝒮 | \mathbb{R}\mapsto\mathbb{R}^{\left|\mathcal{S}\right|} .

###### Lemma E.15 (Invariance of non-domination under positive affine transform) .

1. Let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. If 𝐱 ∈ ND ​ ( X ) \mathbf{x}\in\text{{ND}}\left(X\right) , then ∀ c > 0 , 𝐚 ∈ ℝ | 𝒮 | : ( c ​ 𝐱 + 𝐚 ) ∈ ND ​ ( c ​ X + 𝐚 ) \forall c>0,\mathbf{a}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}(c\mathbf{x}+\mathbf{a})\in\text{{ND}}\left(cX+\mathbf{a}\right) .

2. Let I ⊆ ℝ I\subseteq\mathbb{R} and let F ⊊ ( ℝ | 𝒮 | ) I F\subsetneq\left(\mathbb{R}^{\left|\mathcal{S}\right|}\right)^{I} be a finite set of vector-valued functions on I I . If 𝐟 ∈ ND ​ ( F ) \mathbf{f}\in\text{{ND}}\left(F\right) , then ∀ c > 0 , 𝐚 ∈ ℝ | 𝒮 | : ( c ​ 𝐟 + 𝐚 ) ∈ ND ​ ( c ​ F + 𝐚 ) \forall c>0,\mathbf{a}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}(c\mathbf{f}+\mathbf{a})\in\text{{ND}}\left(cF+\mathbf{a}\right) .

###### Proof.

Item 1 : Suppose 𝐱 ∈ ND ​ ( X ) \mathbf{x}\in\text{{ND}}\left(X\right) is strictly optimal for 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} . Then let c > 0 , 𝐚 ∈ ℝ | 𝒮 | c>0,\mathbf{a}\in\mathbb{R}^{\left|\mathcal{S}\right|} be arbitrary, and define b ≔ 𝐚 ⊤ ​ 𝐫 b\coloneqq\mathbf{a}^{\top}\mathbf{r} . 𝐱 ⊤ ​ 𝐫 \displaystyle\mathbf{x}^{\top}\mathbf{r} > max 𝐱 ′ ∈ X ∖ { 𝐱 } ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 \displaystyle>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}\mathbf{x}^{\prime\top}\mathbf{r} (13) c ​ 𝐱 ⊤ ​ 𝐫 + b \displaystyle c\mathbf{x}^{\top}\mathbf{r}+b > max 𝐱 ′ ∈ X ∖ { 𝐱 } ⁡ c ​ 𝐱 ′ ⁣ ⊤ ​ 𝐫 + b \displaystyle>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}c\mathbf{x}^{\prime\top}\mathbf{r}+b (14) ( c ​ 𝐱 + 𝐚 ) ⊤ ​ 𝐫 \displaystyle(c\mathbf{x}+\mathbf{a})^{\top}\mathbf{r} > max 𝐱 ′ ∈ X ∖ { 𝐱 } ⁡ ( c ​ 𝐱 ′ + 𝐚 ) ⊤ ​ 𝐫 \displaystyle>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}(c\mathbf{x}^{\prime}+\mathbf{a})^{\top}\mathbf{r} (15) ( c ​ 𝐱 + 𝐚 ) ⊤ ​ 𝐫 \displaystyle(c\mathbf{x}+\mathbf{a})^{\top}\mathbf{r} > max 𝐱 ′′ ∈ ( c ​ X + 𝐚 ) ∖ { c ​ 𝐱 + 𝐚 } 𝐱 ′ ′ ⊤ 𝐫 . \displaystyle>\max_{\mathbf{x}^{\prime\prime}\in\left(cX+\mathbf{a}\right)\setminus\left\{c\mathbf{x}+\mathbf{a}\right\}}\mathbf{x}^{\prime\prime\top}\mathbf{r}. (16) Equation 14 follows because c > 0 c>0 . Equation 15 follows by the definition of b b .

Item 2 : If 𝐟 ∈ ND ​ ( F ) \mathbf{f}\in\text{{ND}}\left(F\right) , then by definition E.13 , there exist γ ∈ I , 𝐫 ∈ ℝ | 𝒮 | \gamma\in I,\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} such that 𝐟 ​ ( γ ) ⊤ ​ 𝐫 > max 𝐟 ′ ∈ F ∖ { 𝐟 } ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 . \displaystyle\mathbf{f}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\prime}\in F\setminus\left\{\mathbf{f}\right\}}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}. (17)

Apply item 1 to conclude ( c ​ 𝐟 ​ ( γ ) + 𝐚 ) ⊤ ​ 𝐫 > max ( c ​ 𝐟 ′ + 𝐚 ) ∈ ( c ​ F + 𝐚 ) ∖ { c ​ 𝐟 + 𝐚 } ⁡ ( c ​ 𝐟 ′ ​ ( γ ) + 𝐚 ) ⊤ ​ 𝐫 . \displaystyle(c\mathbf{f}(\gamma)+\mathbf{a})^{\top}\mathbf{r}>\max_{(c\mathbf{f}^{\prime}+\mathbf{a})\in(cF+\mathbf{a})\setminus\left\{c\mathbf{f}+\mathbf{a}\right\}}(c\mathbf{f}^{\prime}(\gamma)+\mathbf{a})^{\top}\mathbf{r}. (18)

Therefore, ( c ​ 𝐟 + 𝐚 ) ∈ ND ​ ( c ​ F + 𝐚 ) (c\mathbf{f}+\mathbf{a})\in\text{{ND}}\left(cF+\mathbf{a}\right) . ∎

#### E.1.2 Inequalities which hold under most reward function distributions

See 6.5

###### Lemma E.16 (Helper lemma for demonstrating ≥ most : 𝔇 any \geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}} ) .

Let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) . If ∃ ϕ ∈ S | 𝒮 | \exists\phi\in S_{\left|\mathcal{S}\right|} such that for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , f 1 ​ ( 𝒟 ) < f 2 ​ ( 𝒟 ) f_{1}\left(\mathcal{D}\right)<f_{2}\left(\mathcal{D}\right) implies that f 1 ​ ( ϕ ⋅ 𝒟 ) > f 2 ​ ( ϕ ⋅ 𝒟 ) f_{1}\left(\phi\cdot\mathcal{D}\right)>f_{2}\left(\phi\cdot\mathcal{D}\right) , then f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) .

###### Proof.

Since ϕ \phi does not belong to the stabilizer of S | 𝒮 | S_{\left|\mathcal{S}\right|} , ϕ \phi acts injectively on S | 𝒮 | ⋅ 𝒟 S_{\left|\mathcal{S}\right|}\cdot\mathcal{D} . By assumption on ϕ \phi , the image of { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) < f 2 ​ ( 𝒟 ′ ) } \{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})<f_{2}(\mathcal{D}^{\prime})\} under ϕ \phi is a subset of { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) } \{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime})\} . Since ϕ \phi is injective, | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) < f 2 ​ ( 𝒟 ′ ) } | ≤ | { 𝒟 ′ ∈ S | 𝒮 | ⋅ 𝒟 ∣ f 1 ​ ( 𝒟 ′ ) > f 2 ​ ( 𝒟 ′ ) } | \left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})<f_{2}(\mathcal{D}^{\prime})\}\right|\leq\left|\{\mathcal{D}^{\prime}\in S_{\left|\mathcal{S}\right|}\cdot\mathcal{D}\mid f_{1}(\mathcal{D}^{\prime})>f_{2}(\mathcal{D}^{\prime})\}\right| . f 1 ( 𝒟 ) ≥ most : 𝔇 f 2 ( 𝒟 ) f_{1}(\mathcal{D})\geq_{\text{{most}}\text{: }\mathfrak{D}}f_{2}(\mathcal{D}) by definition 6.5 . ∎

###### Lemma E.17 (A helper result for expectations of functions) .

Let B 1 , … , B n ⊊ ℝ | 𝒮 | B_{1},\ldots,B_{n}\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite and let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) . Suppose f f is a function of the form f ⁡ ( B 1 , … , B n ∣ 𝒟 ) = 𝔼 𝐫 ∼ 𝒟 [ g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ 𝐛 1 ⊤ ​ 𝐫 , … , max 𝐛 n ∈ B n ⁡ 𝐛 n ⊤ ​ 𝐫 ) ] \displaystyle f\left(B_{1},\ldots,B_{n}\mid\mathcal{D}\right)=\opE_{\mathbf{r}\sim\mathcal{D}}\left[g\left(\max_{\mathbf{b}_{1}\in B_{1}}\mathbf{b}_{1}^{\top}\mathbf{r},\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\mathbf{b}_{n}^{\top}\mathbf{r}\right)\right] (19) for some function g g , and that f f is well-defined for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} . Let ϕ \phi be a state permutation. Then f ⁡ ( B 1 , … , B n ∣ 𝒟 ) = f ⁡ ( ϕ ⋅ B 1 , … , ϕ ⋅ B n ∣ ϕ ⋅ 𝒟 ) . f\left(B_{1},\ldots,B_{n}\mid\mathcal{D}\right)=f\left(\phi\cdot B_{1},\ldots,\phi\cdot B_{n}\mid\phi\cdot\mathcal{D}\right). (20)

###### Proof.

Let distribution 𝒟 \mathcal{D} have probability measure F F , and let ϕ ⋅ 𝒟 \phi\cdot\mathcal{D} have probability measure F ϕ F_{\phi} . f ⁡ ( B 1 , … , B n ∣ 𝒟 ) \displaystyle f\left(B_{1},\ldots,B_{n}\mid\mathcal{D}\right) (21) ≔ \displaystyle\coloneqq{} 𝔼 𝐫 ∼ 𝒟 [ g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ 𝐛 1 ⊤ ​ 𝐫 , … , max 𝐛 n ∈ B n ⁡ 𝐛 n ⊤ ​ 𝐫 ) ] \displaystyle\opE_{\mathbf{r}\sim\mathcal{D}}\left[g\left(\max_{\mathbf{b}_{1}\in B_{1}}\mathbf{b}_{1}^{\top}\mathbf{r},\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\mathbf{b}_{n}^{\top}\mathbf{r}\right)\right] (22) ≔ \displaystyle\coloneqq{} ∫ ℝ | 𝒮 | g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ 𝐛 1 ⊤ ​ 𝐫 , … , max 𝐛 n ∈ B n ⁡ 𝐛 n ⊤ ​ 𝐫 ) ​ d ⁡ F ⁡ ( 𝐫 ) \displaystyle\int_{\mathbb{R}^{\left|\mathcal{S}\right|}}g\left(\max_{\mathbf{b}_{1}\in B_{1}}\mathbf{b}_{1}^{\top}\mathbf{r},\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\mathbf{b}_{n}^{\top}\mathbf{r}\right)\dif F(\mathbf{r}) (23) = \displaystyle={} ∫ ℝ | 𝒮 | g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ 𝐛 1 ⊤ ​ 𝐫 , … , max 𝐛 n ∈ B n ⁡ 𝐛 n ⊤ ​ 𝐫 ) ​ d ⁡ F ϕ ​ ( 𝐏 ϕ ​ 𝐫 ) \displaystyle\int_{\mathbb{R}^{\left|\mathcal{S}\right|}}g\left(\max_{\mathbf{b}_{1}\in B_{1}}\mathbf{b}_{1}^{\top}\mathbf{r},\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\mathbf{b}_{n}^{\top}\mathbf{r}\right)\dif F_{\phi}(\mathbf{P}_{\phi}\mathbf{r}) (24) = \displaystyle={} ∫ ℝ | 𝒮 | g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ 𝐛 1 ⊤ ​ ( 𝐏 ϕ − 1 ​ 𝐫 ′ ) , … , max 𝐛 n ∈ B n ⁡ 𝐛 n ⊤ ​ ( 𝐏 ϕ − 1 ​ 𝐫 ′ ) ) ​ | det 𝐏 ϕ | ​ d ⁡ F ϕ ​ ( 𝐫 ′ ) \displaystyle\int_{\mathbb{R}^{\left|\mathcal{S}\right|}}g\left(\max_{\mathbf{b}_{1}\in B_{1}}\mathbf{b}_{1}^{\top}\left(\mathbf{P}_{\phi}^{-1}\mathbf{r}^{\prime}\right),\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\mathbf{b}_{n}^{\top}\left(\mathbf{P}_{\phi}^{-1}\mathbf{r}^{\prime}\right)\right)\left|\det\mathbf{P}_{\phi}\right|\dif F_{\phi}(\mathbf{r}^{\prime}) (25) = \displaystyle={} ∫ ℝ | 𝒮 | g ⁡ ( max 𝐛 1 ∈ B 1 ⁡ ( 𝐏 ϕ ​ 𝐛 1 ) ⊤ ​ 𝐫 ′ , … , max 𝐛 n ∈ B n ⁡ ( 𝐏 ϕ ​ 𝐛 n ) ⊤ ​ 𝐫 ′ ) ​ d ⁡ F ϕ ​ ( 𝐫 ′ ) \displaystyle\int_{\mathbb{R}^{\left|\mathcal{S}\right|}}g\left(\max_{\mathbf{b}_{1}\in B_{1}}\left(\mathbf{P}_{\phi}\mathbf{b}_{1}\right)^{\top}\mathbf{r}^{\prime},\ldots,\max_{\mathbf{b}_{n}\in B_{n}}\left(\mathbf{P}_{\phi}\mathbf{b}_{n}\right)^{\top}\mathbf{r}^{\prime}\right)\dif F_{\phi}(\mathbf{r}^{\prime}) (26) = \displaystyle={} ∫ ℝ | 𝒮 | g ⁡ ( max 𝐛 1 ′ ∈ ϕ ⋅ B 1 ⁡ 𝐛 1 ′ ⁣ ⊤ ​ 𝐫 ′ , … , max 𝐛 n ′ ∈ ϕ ⋅ B n ⁡ 𝐛 n ′ ⁣ ⊤ ​ 𝐫 ′ ) ​ d ⁡ F ϕ ​ ( 𝐫 ′ ) \displaystyle\int_{\mathbb{R}^{\left|\mathcal{S}\right|}}g\left(\max_{\mathbf{b}_{1}^{\prime}\in\phi\cdot B_{1}}\mathbf{b}_{1}^{\prime\top}\mathbf{r}^{\prime},\ldots,\max_{\mathbf{b}_{n}^{\prime}\in\phi\cdot B_{n}}\mathbf{b}_{n}^{\prime\top}\mathbf{r}^{\prime}\right)\dif F_{\phi}(\mathbf{r}^{\prime}) (27) ≕ \displaystyle\eqqcolon{} f ⁡ ( ϕ ⋅ B 1 , … , ϕ ⋅ B n ∣ ϕ ⋅ 𝒟 ) . \displaystyle f\left(\phi\cdot B_{1},\ldots,\phi\cdot B_{n}\mid\phi\cdot\mathcal{D}\right). (28)

Equation 24 follows by the definition of F ϕ F_{\phi} ( definition 6.3 ). Equation 25 follows by substituting 𝐫 ′ ≔ 𝐏 ϕ ​ 𝐫 \mathbf{r}^{\prime}\coloneqq\mathbf{P}_{\phi}\mathbf{r} . Equation 26 follows from the fact that all permutation matrices have unitary determinant and are orthogonal (and so ( 𝐏 ϕ − 1 ) ⊤ = 𝐏 ϕ (\mathbf{P}_{\phi}^{-1})^{\top}=\mathbf{P}_{\phi} ). ∎

###### Definition E.18 (Support of 𝒟 any \mathcal{D}_{\text{any}} ) .

Let 𝒟 any \mathcal{D}_{\text{any}} be any reward function distribution. supp ⁡ ( 𝒟 any ) \optSupp(\mathcal{D}_{\text{any}}) is the smallest closed subset of ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} whose complement has measure zero under 𝒟 any \mathcal{D}_{\text{any}} .

###### Definition E.19 (Linear functional optimality probability) .

For finite A , B ⊊ ℝ | 𝒮 | A,B\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} , the probability under 𝒟 any \mathcal{D}_{\text{any}} that A A is optimal over B B is p 𝒟 any ​ ( A ≥ B ) ≔ ℙ 𝐫 ∼ 𝒟 any ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ≥ max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) p_{\mathcal{D}_{\text{any}}}\left(A\geq B\right)\coloneqq\Prb\nolimits_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\geq\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right) .

###### Proposition E.20 (Non-dominated linear functionals and their optimality probability) .

Let A ⊊ ℝ | 𝒮 | A\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. If ∃ b < c : [ b , c ] | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}[b,c]^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) , then 𝐚 ∈ ND ​ ( A ) \mathbf{a}\in\text{{ND}}\left(A\right) implies that 𝐚 \mathbf{a} is strictly optimal for a set of reward functions with positive measure under 𝒟 any \mathcal{D}_{\text{any}} .

###### Proof.

Suppose ∃ b < c : [ b , c ] | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}[b,c]^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) . If 𝐚 ∈ ND ​ ( A ) \mathbf{a}\in\text{{ND}}\left(A\right) , then let 𝐫 \mathbf{r} be such that 𝐚 ⊤ ​ 𝐫 > max 𝐚 ′ ∈ A ∖ { 𝐚 } ⁡ 𝐚 ′ ⁣ ⊤ ​ 𝐫 \mathbf{a}^{\top}\mathbf{r}>\max_{\mathbf{a}^{\prime}\in A\setminus\left\{\mathbf{a}\right\}}\mathbf{a}^{\prime\top}\mathbf{r} . For a 1 > 0 , a 2 ∈ ℝ a_{1}>0,a_{2}\in\mathbb{R} , positively affinely transform 𝐫 ′ ≔ a 1 ​ 𝐫 + a 2 ​ 𝟏 \mathbf{r}^{\prime}\coloneqq a_{1}\mathbf{r}+a_{2}\mathbf{1} (where 𝟏 ∈ ℝ | 𝒮 | \mathbf{1}\in\mathbb{R}^{\left|\mathcal{S}\right|} is the all-ones vector) so that 𝐫 ′ ∈ ( b , c ) | 𝒮 | \mathbf{r}^{\prime}\in(b,c)^{\left|\mathcal{S}\right|} .

Note that 𝐚 \mathbf{a} is still strictly optimal for 𝐫 ′ \mathbf{r}^{\prime} : 𝐚 ⊤ ​ 𝐫 > max 𝐚 ′ ∈ A ∖ { 𝐚 } ⁡ 𝐚 ′ ⁣ ⊤ ​ 𝐫 ⇔ 𝐚 ⊤ ​ 𝐫 ′ > max 𝐚 ′ ∈ A ∖ { 𝐚 } ⁡ 𝐚 ′ ⁣ ⊤ ​ 𝐫 ′ . \mathbf{a}^{\top}\mathbf{r}>\max_{\mathbf{a}^{\prime}\in A\setminus\left\{\mathbf{a}\right\}}\mathbf{a}^{\prime\top}\mathbf{r}\iff\mathbf{a}^{\top}\mathbf{r}^{\prime}>\max_{\mathbf{a}^{\prime}\in A\setminus\left\{\mathbf{a}\right\}}\mathbf{a}^{\prime\top}\mathbf{r}^{\prime}. (29)

Furthermore, by the continuity of both terms on the right-hand side of eq. 29 , 𝐚 \mathbf{a} is strictly optimal for reward functions in some open neighborhood N N of 𝐫 ′ \mathbf{r}^{\prime} . Let N ′ ≔ N ∩ ( b , c ) | 𝒮 | N^{\prime}\coloneqq N\cap(b,c)^{\left|\mathcal{S}\right|} . N ′ N^{\prime} is still open in ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} since it is the intersection of two open sets N N and ( b , c ) | 𝒮 | (b,c)^{\left|\mathcal{S}\right|} .

𝒟 any \mathcal{D}_{\text{any}} must assign positive probability measure to all open sets in its support; otherwise, its support would exclude these zero-measure sets by definition E.18 . Therefore, 𝒟 any \mathcal{D}_{\text{any}} assigns positive probability to N ′ ⊆ supp ⁡ ( 𝒟 any ) N^{\prime}\subseteq\optSupp(\mathcal{D}_{\text{any}}) . ∎

###### Lemma E.21 (Expected value of similar linear functional sets) .

Let A , B ⊊ ℝ | 𝒮 | A,B\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite, let A ′ A^{\prime} be such that ND ​ ( A ) ⊆ A ′ ⊆ A \text{{ND}}\left(A\right)\subseteq A^{\prime}\subseteq A , and let g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} be an increasing function. If B B contains a copy B ′ B^{\prime} of A ′ A^{\prime} via ϕ \phi , then 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] ≤ 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] . \opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]\leq\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (30)

If ND ​ ( B ) ∖ B ′ \text{{ND}}\left(B\right)\setminus B^{\prime} is empty, then eq. 30 is an equality. If ND ​ ( B ) ∖ B ′ \text{{ND}}\left(B\right)\setminus B^{\prime} is non-empty, g g is strictly increasing, and ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 bound ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{bound}}) , then eq. 30 is strict.

###### Proof.

Because g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} is increasing, it is measurable (as is max \max ). Therefore, the relevant expectations exist for all 𝒟 bound \mathcal{D}_{\text{bound}} . 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] = 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ′ ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A^{\prime}}\mathbf{a}^{\top}\mathbf{r}\right)\right] (31) = 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ ϕ ⋅ A ′ ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle=\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in\phi\cdot A^{\prime}}\mathbf{a}^{\top}\mathbf{r}\right)\right] (32) = 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ′ ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle=\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B^{\prime}}\mathbf{b}^{\top}\mathbf{r}\right)\right] (33) ≤ 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] . \displaystyle\leq\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (34) Equation 31 holds because ∀ 𝐫 ∈ ℝ | 𝒮 | : max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 = max 𝐚 ∈ A ′ ⁡ 𝐚 ⊤ ​ 𝐫 \forall\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}=\max_{\mathbf{a}\in A^{\prime}}\mathbf{a}^{\top}\mathbf{r} by Lemma E.12 ’s item 2 with X ≔ A X\coloneqq A , X ′ ≔ A ′ X^{\prime}\coloneqq A^{\prime} . Equation 32 holds by Lemma E.17 . Equation 33 holds by the definition of B ′ B^{\prime} . Furthermore, our assumption on ϕ \phi guarantees that B ′ ⊆ B B^{\prime}\subseteq B . Therefore, max 𝐛 ∈ B ′ ⁡ 𝐛 ⊤ ​ 𝐫 ≤ max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 \max_{\mathbf{b}\in B^{\prime}}\mathbf{b}^{\top}\mathbf{r}\leq\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r} , and so eq. 34 holds by the fact that g g is an increasing function. Then eq. 30 holds.

If ND ​ ( B ) ∖ B ′ \text{{ND}}\left(B\right)\setminus B^{\prime} is empty, then ND ​ ( B ) ⊆ B ′ \text{{ND}}\left(B\right)\subseteq B^{\prime} . By assumption, B ′ ⊆ B B^{\prime}\subseteq B . Then apply Lemma E.12 item 2 with X ≔ B X\coloneqq B , X ′ ≔ B ′ X^{\prime}\coloneqq B^{\prime} in order to conclude that eq. 34 is an equality. Then eq. 30 is also an equality.

Suppose that g g is strictly increasing, ND ​ ( B ) ∖ B ′ \text{{ND}}\left(B\right)\setminus B^{\prime} is non-empty, and ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 bound ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{bound}}) . Let 𝐱 ∈ ND ​ ( B ) ∖ B ′ \mathbf{x}\in\text{{ND}}\left(B\right)\setminus B^{\prime} . 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ′ ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B^{\prime}}\mathbf{b}^{\top}\mathbf{r}\right)\right] < 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ B ′ ∪ { 𝐱 } ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle<\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in B^{\prime}\cup\left\{\mathbf{x}\right\}}\mathbf{b}^{\top}\mathbf{r}\right)\right] (35) ≤ 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] . \displaystyle\leq\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (36)

𝐱 \mathbf{x} is strictly optimal for a positive-probability subset of supp ⁡ ( 𝒟 bound ) \optSupp(\mathcal{D}_{\text{bound}}) by Proposition E.20 . Since g g is strictly increasing, eq. 35 is strict. Therefore, we conclude that eq. 30 is strict. ∎

###### Lemma E.22 (For continuous iid distributions 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} , ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 X ​ - iid ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{X\text{-}\textsc{iid}}) ) .

###### Proof.

𝒟 X ​ - iid ≔ X | 𝒮 | \mathcal{D}_{X\text{-}\textsc{iid}}\coloneqq X^{\left|\mathcal{S}\right|} . Since the state reward distribution X X is continuous, X X must have support on some open interval ( b , c ) (b,c) . Since 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} is iid across states, ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 X ​ - iid ) (b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{X\text{-}\textsc{iid}}) . ∎

###### Definition E.23 (Bounded, continuous iid reward) .

𝔇 c/b/ iid \mathfrak{D}_{\textsc{c/b/}\textsc{iid}} is the set of 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} which equal X | 𝒮 | X^{\left|\mathcal{S}\right|} for some continuous, bounded-support distribution X X over ℝ \mathbb{R} .

###### Lemma E.24 (Expectation superiority lemma) .

Let A , B ⊊ ℝ | 𝒮 | A,B\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite and let g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} be an increasing function. If B B contains a copy B ′ B^{\prime} of ND ​ ( A ) \text{{ND}}\left(A\right) via ϕ \phi , then 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐚 ∈ A 𝐚 ⊤ 𝐫 ) ] ≤ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐛 ∈ B 𝐛 ⊤ 𝐫 ) ] . \displaystyle\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (37)

Furthermore, if g g is strictly increasing and ND ​ ( B ) ∖ ϕ ⋅ ND ​ ( A ) \text{{ND}}\left(B\right)\setminus\phi\cdot\text{{ND}}\left(A\right) is non-empty, then eq. 37 is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} . In particular, 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐚 ∈ A 𝐚 ⊤ 𝐫 ) ] ≱ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐛 ∈ B 𝐛 ⊤ 𝐫 ) ] \opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] .

###### Proof.

Because g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} is increasing, it is measurable (as is max \max ). Therefore, the relevant expectations exist for all 𝒟 bound \mathcal{D}_{\text{bound}} .

Suppose that 𝒟 bound \mathcal{D}_{\text{bound}} is such that 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] < 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]<\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] . 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] ≤ 𝔼 𝐫 ∼ ϕ 2 ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle\leq\opE_{\mathbf{r}\sim\phi^{2}\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] (38) = 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] (39) < 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle<\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] (40) ≤ 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] . \displaystyle\leq\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (41) Equation 38 follows by applying Lemma E.21 with permutation ϕ \phi and A ′ ≔ ND ​ ( A ) A^{\prime}\coloneqq\text{{ND}}\left(A\right) . Equation 39 follows because involutions satisfy ϕ − 1 = ϕ \phi^{-1}=\phi , and ϕ 2 \phi^{2} is therefore the identity. Equation 40 follows because we assumed that 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] < 𝔼 𝐫 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]<\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] . Equation 41 follows by applying Lemma E.21 with permutation ϕ \phi and and A ′ ≔ ND ​ ( A ) A^{\prime}\coloneqq\text{{ND}}\left(A\right) . By Lemma E.16 , eq. 37 holds.

Suppose g g is strictly increasing and ND ​ ( B ) ∖ B ′ \text{{ND}}\left(B\right)\setminus B^{\prime} is non-empty. Let ϕ ′ ∈ S | 𝒮 | \phi^{\prime}\in S_{\left|\mathcal{S}\right|} . 𝔼 𝐫 ∼ ϕ ′ ⋅ 𝒟 X ​ - iid [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle\opE_{\mathbf{r}\sim\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] = 𝔼 𝐫 ∼ 𝒟 X ​ - iid [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right] (42) < 𝔼 𝐫 ∼ ϕ ⋅ 𝒟 X ​ - iid [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \displaystyle<\opE_{\mathbf{r}\sim\phi\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] (43) = 𝔼 𝐫 ∼ ϕ ′ ⋅ 𝒟 X ​ - iid [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] . \displaystyle=\opE_{\mathbf{r}\sim\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (44)

Equation 42 and eq. 44 hold because 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} distributes reward identically across states: ∀ ϕ x ∈ S | 𝒮 | : ϕ x ⋅ 𝒟 X ​ - iid = 𝒟 X ​ - iid \forall\phi_{x}\in S_{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\phi_{x}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}=\mathcal{D}_{X\text{-}\textsc{iid}} . By Lemma E.22 , ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 X ​ - iid ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{X\text{-}\textsc{iid}}) . Therefore, apply Lemma E.21 with A ′ ≔ ND ​ ( A ) A^{\prime}\coloneqq\text{{ND}}\left(A\right) to conclude that eq. 43 holds.

Therefore, ∀ ϕ ′ ∈ S | 𝒮 | : 𝔼 𝐫 ∼ ϕ ′ ⋅ 𝒟 X ​ - iid [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ) ] < 𝔼 𝐫 ∼ ϕ ′ ⋅ 𝒟 X ​ - iid [ g ⁡ ( max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) ] \forall\phi^{\prime}\in S_{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\opE_{\mathbf{r}\sim\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]<\opE_{\mathbf{r}\sim\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] , and so 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐚 ∈ A 𝐚 ⊤ 𝐫 ) ] ≱ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐛 ∈ B 𝐛 ⊤ 𝐫 ) ] \opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right] by definition 6.5 . ∎

###### Definition E.25 (Indicator function) .

Let L L be a predicate which takes input x x . 𝟙 L ⁡ ( x ) \mathbbm{1}_{L(x)} is the function which returns 1 when L ⁡ ( x ) L(x) is true, and 0 otherwise.

###### Lemma E.26 (Optimality probability inclusion relations) .

Let X , Y ⊊ ℝ | 𝒮 | X,Y\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite and suppose Y ′ ⊆ Y Y^{\prime}\subseteq Y . p 𝒟 any ​ ( X ≥ Y ) ≤ p 𝒟 any ​ ( X ≥ Y ′ ) ≤ p 𝒟 any ​ ( X ∪ ( Y ∖ Y ′ ) ≥ Y ) . p_{\mathcal{D}_{\text{any}}}\left(X\geq Y\right)\leq p_{\mathcal{D}_{\text{any}}}\left(X\geq Y^{\prime}\right)\leq p_{\mathcal{D}_{\text{any}}}\left(X\cup\left(Y\setminus Y^{\prime}\right)\geq Y\right). (45)

If ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) , X ⊆ Y X\subseteq Y , and ND ​ ( Y ) ∩ ( Y ∖ Y ′ ) \text{{ND}}\left(Y\right)\cap\left(Y\setminus Y^{\prime}\right) is non-empty, then the second inequality is strict.

###### Proof.

p 𝒟 any ​ ( X ≥ Y ) \displaystyle p_{\mathcal{D}_{\text{any}}}\left(X\geq Y\right) ≔ 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐫 ] \displaystyle\coloneqq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{r}}\right] (46) ≤ 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ⁡ 𝐲 ⊤ ​ 𝐫 ] \displaystyle\leq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}}\mathbf{y}^{\top}\mathbf{r}}\right] (47) ≤ 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ∪ ( Y ∖ Y ′ ) ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ⁡ 𝐲 ⊤ ​ 𝐫 ] \displaystyle\leq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X\cup(Y\setminus Y^{\prime})}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}}\mathbf{y}^{\top}\mathbf{r}}\right] (48) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ∪ ( Y ∖ Y ′ ) ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ∪ ( Y ∖ Y ′ ) ⁡ 𝐲 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X\cup(Y\setminus Y^{\prime})}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}\cup(Y\setminus Y^{\prime})}\mathbf{y}^{\top}\mathbf{r}}\right] (49) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ∪ ( Y ∖ Y ′ ) ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X\cup(Y\setminus Y^{\prime})}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{r}}\right] (50) ≕ p 𝒟 any ​ ( X ∪ ( Y ∖ Y ′ ) ≥ Y ) . \displaystyle\eqqcolon p_{\mathcal{D}_{\text{any}}}\left(X\cup\left(Y\setminus Y^{\prime}\right)\geq Y\right). (51) Equation 47 follows because ∀ 𝐫 ∈ ℝ | 𝒮 | : 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐫 ≤ 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ⁡ 𝐲 ⊤ ​ 𝐫 \forall\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{r}}\leq\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}}\mathbf{y}^{\top}\mathbf{r}} since Y ′ ⊆ Y Y^{\prime}\subseteq Y ; note that eq. 47 equals p 𝒟 any ​ ( X ≥ Y ′ ) p_{\mathcal{D}_{\text{any}}}\left(X\geq Y^{\prime}\right) , and so the first inequality of eq. 45 is shown. Equation 48 holds because ∀ 𝐫 ∈ ℝ | 𝒮 | : 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ⁡ 𝐲 ⊤ ​ 𝐫 ≤ 𝟙 max 𝐱 ∈ X ∪ ( Y ∖ Y ′ ) ⁡ 𝐱 ⊤ ​ 𝐫 ≥ max 𝐲 ∈ Y ′ ⁡ 𝐛 ⊤ ​ 𝐫 \forall\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}}\mathbf{y}^{\top}\mathbf{r}}\leq\mathbbm{1}_{\max_{\mathbf{x}\in X\cup(Y\setminus Y^{\prime})}\mathbf{x}^{\top}\mathbf{r}\geq\max_{\mathbf{y}\in Y^{\prime}}\mathbf{b}^{\top}\mathbf{r}} .

Suppose ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) , X ⊆ Y X\subseteq Y , and ND ​ ( Y ) ∩ ( Y ∖ Y ′ ) \text{{ND}}\left(Y\right)\cap\left(Y\setminus Y^{\prime}\right) is non-empty. Let 𝐲 ∗ ∈ ND ​ ( Y ) ∩ ( Y ∖ Y ′ ) \mathbf{y}^{*}\in\text{{ND}}\left(Y\right)\cap\left(Y\setminus Y^{\prime}\right) . By Proposition E.20 , 𝐲 ∗ \mathbf{y}^{*} is strictly optimal on a subset of supp ⁡ ( 𝒟 any ) \optSupp(\mathcal{D}_{\text{any}}) with positive measure under 𝒟 any \mathcal{D}_{\text{any}} . In particular, for a set of 𝐫 ∗ \mathbf{r}^{*} with positive measure under 𝒟 any \mathcal{D}_{\text{any}} , we have 𝐲 ∗ ⁣ ⊤ ​ 𝐫 ∗ > max 𝐲 ∈ Y ′ ⁡ 𝐲 ⊤ ​ 𝐫 ∗ . \mathbf{y}^{*\top}\mathbf{r}^{*}>\max_{\mathbf{y}\in Y^{\prime}}\mathbf{y}^{\top}\mathbf{r}^{*}.

Then eq. 48 is strict, and therefore the second inequality of eq. 45 is strict as well. ∎

###### Lemma E.27 (Optimality probability of similar linear functional sets) .

Let A , B , C ⊊ ℝ | 𝒮 | A,B,C\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite, and let Z ⊆ ℝ | 𝒮 | Z\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} be such that ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C . If ND ​ ( A ) \text{{ND}}\left(A\right) is similar to B ′ ⊆ B B^{\prime}\subseteq B via ϕ \phi such that ϕ ⋅ ( Z ∖ ( B ∖ B ′ ) ) = Z ∖ ( B ∖ B ′ ) \phi\cdot\left(Z\setminus\left(B\setminus B^{\prime}\right)\right)=Z\setminus\left(B\setminus B^{\prime}\right) , then p 𝒟 any ​ ( A ≥ C ) ≤ p ϕ ⋅ 𝒟 any ​ ( B ≥ C ) . p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\leq p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(B\geq C\right). (52) If B ′ = B B^{\prime}=B , then eq. 52 is an equality. If ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) , B ′ ⊆ C B^{\prime}\subseteq C , and ND ​ ( C ) ∩ ( B ∖ B ′ ) \text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right) is non-empty, then eq. 52 is strict.

###### Proof.

p 𝒟 any ​ ( A ≥ C ) \displaystyle p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right) = p 𝒟 any ​ ( A ≥ Z ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(A\geq Z\right) (53) = p 𝒟 any ​ ( ND ​ ( A ) ≥ Z ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(\text{{ND}}\left(A\right)\geq Z\right) (54) ≤ p 𝒟 any ​ ( ND ​ ( A ) ≥ Z ∖ ( B ∖ B ′ ) ) \displaystyle\leq p_{\mathcal{D}_{\text{any}}}\left(\text{{ND}}\left(A\right)\geq Z\setminus\left(B\setminus B^{\prime}\right)\right) (55) = p ϕ ⋅ 𝒟 any ​ ( ϕ ⋅ ND ​ ( A ) ≥ ϕ ⋅ Z ∖ ( B ∖ B ′ ) ) \displaystyle=p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(\phi\cdot\text{{ND}}\left(A\right)\geq\phi\cdot Z\setminus\left(B\setminus B^{\prime}\right)\right) (56) = p ϕ ⋅ 𝒟 any ​ ( B ′ ≥ Z ∖ ( B ∖ B ′ ) ) \displaystyle=p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(B^{\prime}\geq Z\setminus\left(B\setminus B^{\prime}\right)\right) (57) ≤ p ϕ ⋅ 𝒟 any ​ ( B ′ ∪ ( B ∖ B ′ ) ≥ Z ) \displaystyle\leq p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(B^{\prime}\cup\left(B\setminus B^{\prime}\right)\geq Z\right) (58) = p ϕ ⋅ 𝒟 any ​ ( B ≥ C ) . \displaystyle=p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(B\geq C\right). (59)

Equation 53 and eq. 59 follow by Lemma E.12 ’s item 2 with X ≔ C X\coloneqq C , X ′ ≔ Z X^{\prime}\coloneqq Z . Similarly, eq. 54 follows by Lemma E.12 ’s item 2 with X ≔ A X\coloneqq A , X ′ ≔ ND ​ ( A ) X^{\prime}\coloneqq\text{{ND}}\left(A\right) . Equation 55 follows by applying the first inequality of Lemma E.26 with X ≔ ND ​ ( A ) , Y ≔ Z , Y ′ ≔ Z ∖ ( B ∖ B ′ ) X\coloneqq\text{{ND}}\left(A\right),Y\coloneqq Z,Y^{\prime}\coloneqq Z\setminus(B\setminus B^{\prime}) . Equation 56 follows by applying Lemma E.17 to eq. 53 with permutation ϕ \phi .

Equation 57 follows by our assumptions on ϕ \phi . Equation 58 follows because by applying the second inequality of Lemma E.26 with X ≔ B ′ , Y ≔ ND ​ ( C ) , Y ′ ≔ ND ​ ( C ) ∖ ( B ∖ B ′ ) X\coloneqq B^{\prime},Y\coloneqq\text{{ND}}\left(C\right),Y^{\prime}\coloneqq\text{{ND}}\left(C\right)\setminus(B\setminus B^{\prime}) .

Suppose B ′ = B B^{\prime}=B . Then B ∖ B ′ = ∅ B\setminus B^{\prime}=\emptyset , and so eq. 55 and eq. 58 are trivially equalities. Then eq. 52 is an equality.

Suppose ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 any ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{\text{any}}) ; note that ( b , c ) | 𝒮 | ⊆ supp ⁡ ( ϕ ⋅ 𝒟 any ) (b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\phi\cdot\mathcal{D}_{\text{any}}) , since such support must be invariant to permutation. Further suppose that B ′ ⊆ C B^{\prime}\subseteq C and that ND ​ ( C ) ∩ ( B ∖ B ′ ) \text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right) is non-empty. Then letting X ≔ B ′ , Y ≔ Z , Y ′ ≔ Z ∖ ( B ∖ B ′ ) X\coloneqq B^{\prime},Y\coloneqq Z,Y^{\prime}\coloneqq Z\setminus(B\setminus B^{\prime}) and noting that ND ​ ( ND ​ ( Z ) ) = ND ​ ( Z ) \text{{ND}}\left(\text{{ND}}\left(Z\right)\right)=\text{{ND}}\left(Z\right) , apply Lemma E.26 to eq. 58 to conclude that eq. 52 is strict. ∎

###### Lemma E.28 (Optimality probability superiority lemma) .

Let A , B , C ⊊ ℝ | 𝒮 | A,B,C\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite, and let Z Z satisfy ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C . If B B contains a copy B ′ B^{\prime} of ND ​ ( A ) \text{{ND}}\left(A\right) via ϕ \phi such that ϕ ⋅ ( Z ∖ ( B ∖ B ′ ) ) = Z ∖ ( B ∖ B ′ ) \phi\cdot\left(Z\setminus\left(B\setminus B^{\prime}\right)\right)=Z\setminus\left(B\setminus B^{\prime}\right) , then p 𝒟 any ( A ≥ C ) ≤ most : 𝔇 any p 𝒟 any ( B ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) .

If B ′ ⊆ C B^{\prime}\subseteq C and ND ​ ( C ) ∩ ( B ∖ B ′ ) \text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right) is non-empty, then the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} and p 𝒟 any ( A ≥ C ) ≱ most : 𝔇 any p 𝒟 any ( B ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) .

###### Proof.

Suppose 𝒟 any \mathcal{D}_{\text{any}} is such that p 𝒟 any ​ ( B ≥ C ) < p 𝒟 any ​ ( A ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right)<p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right) . p ϕ ⋅ 𝒟 any ​ ( A ≥ C ) \displaystyle p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(A\geq C\right) = p ϕ − 1 ⋅ 𝒟 any ​ ( A ≥ C ) \displaystyle=p_{\phi^{-1}\cdot\mathcal{D}_{\text{any}}}\left(A\geq C\right) (60) ≤ p 𝒟 any ​ ( B ≥ C ) \displaystyle\leq p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) (61) < p 𝒟 any ​ ( A ≥ C ) \displaystyle<p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right) (62) ≤ p ϕ ⋅ 𝒟 any ​ ( B ≥ C ) . \displaystyle\leq p_{\phi\cdot\mathcal{D}_{\text{any}}}\left(B\geq C\right). (63)

Equation 60 holds because ϕ \phi is an involution. Equation 61 and eq. 63 hold by applying Lemma E.27 with permutation ϕ \phi . Equation 62 holds by assumption. Therefore, p 𝒟 any ( A ≥ C ) ≤ most : 𝔇 any p 𝒟 any ( B ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) by Lemma E.16 .

Suppose B ′ ⊆ C B^{\prime}\subseteq C and ND ​ ( C ) ∩ ( B ∖ B ′ ) \text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right) is non-empty, and let 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} be any continuous distribution which distributes reward independently and identically across states. Let ϕ ′ ∈ S | 𝒮 | \phi^{\prime}\in S_{\left|\mathcal{S}\right|} . p ϕ ′ ⋅ 𝒟 X ​ - iid ​ ( A ≥ C ) \displaystyle p_{\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left(A\geq C\right) = p 𝒟 X ​ - iid ​ ( A ≥ C ) \displaystyle=p_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(A\geq C\right) (64) < p ϕ ⋅ 𝒟 X ​ - iid ​ ( B ≥ C ) \displaystyle<p_{\phi\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left(B\geq C\right) (65) = p ϕ ′ ⋅ 𝒟 X ​ - iid ​ ( A ≥ C ) . \displaystyle=p_{\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left(A\geq C\right). (66)

Equation 64 and eq. 66 hold because 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} distributes reward identically across states, ∀ ϕ x ∈ S | 𝒮 | : ϕ x ⋅ 𝒟 X ​ - iid = 𝒟 X ​ - iid \forall\phi_{x}\in S_{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\phi_{x}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}=\mathcal{D}_{X\text{-}\textsc{iid}} . By Lemma E.22 , ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 X ​ - iid ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{X\text{-}\textsc{iid}}) . Therefore, apply Lemma E.27 to conclude that eq. 65 holds.

Therefore, ∀ ϕ ′ ∈ S | 𝒮 | : p ϕ ′ ⋅ 𝒟 X ​ - iid ​ ( A ≥ C ) < p ϕ ′ ⋅ 𝒟 X ​ - iid ​ ( B ≥ C ) \forall\phi^{\prime}\in S_{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}p_{\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left(A\geq C\right)<p_{\phi^{\prime}\cdot\mathcal{D}_{X\text{-}\textsc{iid}}}\left(B\geq C\right) . In particular, p 𝒟 any ( A ≥ C ) ≱ most : 𝔇 any p 𝒟 any ( B ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) by definition 6.5 . ∎

###### Lemma E.29 (Limit probability inequalities which hold for most distributions) .

Let I ⊆ ℝ I\subseteq\mathbb{R} , let 𝔇 ⊆ Δ ⁡ ( ℝ | 𝒮 | ) \mathfrak{D}\subseteq\Delta(\mathbb{R}^{\left|\mathcal{S}\right|}) be closed under permutation, and let F A , F B , F C F_{A},F_{B},F_{C} be finite sets of vector functions I ↦ ℝ | 𝒮 | I\mapsto\mathbb{R}^{\left|\mathcal{S}\right|} . Let γ \gamma be a limit point of I I such that f 1 ​ ( 𝒟 ) ≔ lim γ ∗ → γ p 𝒟 ​ ( F B ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) , f 2 ​ ( 𝒟 ) ≔ lim γ ∗ → γ p 𝒟 ​ ( F A ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) f_{1}(\mathcal{D})\coloneqq\lim_{\gamma^{*}\to\gamma}p_{\mathcal{D}}\left(F_{B}(\gamma^{*})\geq F_{C}(\gamma^{*})\right),f_{2}(\mathcal{D})\coloneqq\lim_{\gamma^{*}\to\gamma}p_{\mathcal{D}}\left(F_{A}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) are well-defined for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} .

Let F Z F_{Z} satisfy ND ​ ( F C ) ⊆ F Z ⊆ F C \text{{ND}}\left(F_{C}\right)\subseteq F_{Z}\subseteq F_{C} . Suppose F B F_{B} contains a copy of F A F_{A} via ϕ \phi such that ϕ ⋅ ( F Z ∖ ( F B ∖ ϕ ⋅ F A ) ) = F Z ∖ ( F B ∖ ϕ ⋅ F A ) \phi\cdot\left(F_{Z}\setminus\left(F_{B}\setminus\phi\cdot F_{A}\right)\right)=F_{Z}\setminus\left(F_{B}\setminus\phi\cdot F_{A}\right) . Then f 2 ( 𝔇 ) ≤ most : 𝔇 f 1 ( 𝔇 ) f_{2}(\mathfrak{D})\leq_{\text{{most}}\text{: }\mathfrak{D}}f_{1}(\mathfrak{D}) .

###### Proof.

Suppose 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} is such that f 2 ​ ( 𝒟 ) > f 1 ​ ( 𝒟 ) f_{2}(\mathcal{D})>f_{1}(\mathcal{D}) . f 2 ​ ( ϕ ⋅ 𝒟 ) \displaystyle f_{2}\left(\phi\cdot\mathcal{D}\right) = f 2 ​ ( ϕ − 1 ⋅ 𝒟 ) \displaystyle=f_{2}\left(\phi^{-1}\cdot\mathcal{D}\right) (67) ≔ lim γ ∗ → γ p ϕ − 1 ⋅ 𝒟 ​ ( F A ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) \displaystyle\coloneqq\lim_{\gamma^{*}\to\gamma}p_{\phi^{-1}\cdot\mathcal{D}}\left(F_{A}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) (68) ≤ lim γ ∗ → γ p 𝒟 ​ ( F B ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) \displaystyle\leq\lim_{\gamma^{*}\to\gamma}p_{\mathcal{D}}\left(F_{B}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) (69) < lim γ ∗ → γ p 𝒟 ​ ( F A ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) \displaystyle<\lim_{\gamma^{*}\to\gamma}p_{\mathcal{D}}\left(F_{A}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) (70) ≤ lim γ ∗ → γ p ϕ ⋅ 𝒟 ​ ( F B ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) \displaystyle\leq\lim_{\gamma^{*}\to\gamma}p_{\phi\cdot\mathcal{D}}\left(F_{B}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) (71) ≕ f 1 ​ ( ϕ ⋅ 𝒟 ) . \displaystyle\eqqcolon f_{1}\left(\phi\cdot\mathcal{D}\right). (72) By the assumption that 𝔇 \mathfrak{D} is closed under permutation and f 2 f_{2} is well-defined for all 𝒟 ∈ 𝔇 \mathcal{D}\in\mathfrak{D} , f 2 ​ ( ϕ ⋅ 𝒟 ) f_{2}(\phi\cdot\mathcal{D}) is well-defined. Equation 67 follows since ϕ = ϕ − 1 \phi=\phi^{-1} because ϕ \phi is an involution. For all γ ∗ ∈ I \gamma^{*}\in I , let A ≔ F A ​ ( γ ∗ ) , B ≔ F B ​ ( γ ∗ ) , C ≔ F C ​ ( γ ∗ ) , Z ≔ F Z ​ ( γ ∗ ) A\coloneqq F_{A}(\gamma^{*}),B\coloneqq F_{B}(\gamma^{*}),C\coloneqq F_{C}(\gamma^{*}),Z\coloneqq F_{Z}(\gamma^{*}) (by definition E.13 , ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C ). Since ϕ ⋅ A ⊆ B \phi\cdot A\subseteq B by assumption, and since ND ​ ( A ) ⊆ A \text{{ND}}\left(A\right)\subseteq A , B B also contains a copy of ND ​ ( A ) \text{{ND}}\left(A\right) via ϕ \phi . Furthermore, ϕ ⋅ ( Z ∖ ( B ∖ ϕ ⋅ A ) ) = Z ∖ ( B ∖ ϕ ⋅ A ) \phi\cdot\left(Z\setminus\left(B\setminus\phi\cdot A\right)\right)=Z\setminus\left(B\setminus\phi\cdot A\right) (by assumption), and so apply Lemma E.27 to conclude that p ϕ − 1 ⋅ 𝒟 ​ ( F A ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) ≤ p 𝒟 ​ ( F B ​ ( γ ∗ ) ≥ F C ​ ( γ ∗ ) ) p_{\phi^{-1}\cdot\mathcal{D}}\left(F_{A}(\gamma^{*})\geq F_{C}(\gamma^{*})\right)\leq p_{\mathcal{D}}\left(F_{B}(\gamma^{*})\geq F_{C}(\gamma^{*})\right) . Therefore, the limit inequality eq. 69 holds. Equation 70 follows because we assumed that f 1 ​ ( 𝒟 ) < f 2 ​ ( 𝒟 ) f_{1}(\mathcal{D})<f_{2}(\mathcal{D}) . Equation 71 holds by reasoning similar to that given for eq. 69 .

Therefore, f 2 ​ ( 𝒟 ) > f 1 ​ ( 𝒟 ) f_{2}(\mathcal{D})>f_{1}(\mathcal{D}) implies that f 2 ​ ( ϕ ⋅ 𝒟 ) < f 1 ​ ( ϕ ⋅ 𝒟 ) f_{2}\left(\phi\cdot\mathcal{D}\right)<f_{1}\left(\phi\cdot\mathcal{D}\right) , and so apply Lemma E.16 to conclude that f 2 ( 𝒟 ) ≤ most : 𝔇 f 1 ( 𝒟 ) f_{2}(\mathcal{D})\leq_{\text{{most}}\text{: }\mathfrak{D}}f_{1}(\mathcal{D}) . ∎

#### E.1.3 ℱ nd \Fndop results

\transferDiscount

###### Proof.

Let R R be any reward function. Suppose γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) and construct R ′ ​ ( s ) ≔ V R ∗ ​ ( s , γ ) − γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] R^{\prime}(s)\coloneqq V^{*}_{R}\left(s,\gamma\right)-\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] .

Let π ∈ Π \pi\in\Pi be any policy. By the definition of optimal policies, π ∈ Π ∗ ​ ( R ′ , γ ∗ ) \pi\in\Pi^{*}\left(R^{\prime},\gamma^{*}\right) iff for all s s : R ′ ​ ( s ) + γ ∗ ​ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ V R ′ ∗ ​ ( s ′ , γ ∗ ) ] \displaystyle R^{\prime}(s)+\gamma^{*}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[V^{*}_{R^{\prime}}\left(s^{\prime},\gamma^{*}\right)\right] = R ′ ​ ( s ) + γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ′ ∗ ​ ( s ′ , γ ∗ ) ] \displaystyle=R^{\prime}(s)+\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[V^{*}_{R^{\prime}}\left(s^{\prime},\gamma^{*}\right)\right] (73) R ′ ​ ( s ) + γ ∗ ​ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ V R ∗ ​ ( s ′ , γ ) ] \displaystyle R^{\prime}(s)+\gamma^{*}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] = R ′ ​ ( s ) + γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] \displaystyle=R^{\prime}(s)+\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] (74) γ ∗ ​ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ V R ∗ ​ ( s ′ , γ ) ] \displaystyle\gamma^{*}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] = γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] \displaystyle=\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] (75) 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ V R ∗ ​ ( s ′ , γ ) ] \displaystyle\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] = max a ∈ 𝒜 ⁡ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] . \displaystyle=\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right]. (76)

By the Bellman equations, R ′ ​ ( s ) = V R ′ ∗ ​ ( s , γ ∗ ) − γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ′ ∗ ​ ( s ′ , γ ∗ ) ] R^{\prime}(s)=V^{*}_{R^{\prime}}\left(s,\gamma^{*}\right)-\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R^{\prime}}\left(s^{\prime},\gamma^{*}\right)\right] . By the definition of R ′ R^{\prime} , V R ′ ∗ ​ ( ⋅ , γ ∗ ) = V R ∗ ​ ( ⋅ , γ ) V^{*}_{R^{\prime}}\left(\cdot,\gamma^{*}\right)=V^{*}_{R}\left(\cdot,\gamma\right) must be the unique solution to the Bellman equations for R ′ R^{\prime} at γ ∗ \gamma^{*} . Therefore, eq. 74 holds. Equation 75 follows by plugging in R ′ ≔ V R ∗ ​ ( s , γ ) − γ ∗ ​ max a ∈ 𝒜 ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] R^{\prime}\coloneqq V^{*}_{R}\left(s,\gamma\right)-\gamma^{*}\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] to eq. 74 and doing algebraic manipulation. Equation 76 follows because γ ∗ > 0 \gamma^{*}>0 .

Equation 76 shows that π ∈ Π ∗ ​ ( R ′ , γ ∗ ) \pi\in\Pi^{*}\left(R^{\prime},\gamma^{*}\right) iff ∀ s : 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ V R ∗ ​ ( s ′ , γ ) ] = max a ∈ 𝒜 ⁡ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] \forall s\mathrel{\mathop{\ordinarycolon}}\opE_{s^{\prime}\sim T(s,\pi(s))}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right]=\max_{a\in\mathcal{A}}\opE_{s^{\prime}\sim T(s,a)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right] . That is, π ∈ Π ∗ ​ ( R ′ , γ ∗ ) \pi\in\Pi^{*}\left(R^{\prime},\gamma^{*}\right) iff π ∈ Π ∗ ​ ( R , γ ) \pi\in\Pi^{*}\left(R,\gamma\right) . ∎

###### Definition E.30 (Evaluating sets of visit distribution functions at γ \gamma ) .

For γ ∈ ( 0 , 1 ) \gamma\in(0,1) , define ℱ ⁡ ( s , γ ) ≔ { 𝐟 ⁡ ( γ ) ∣ 𝐟 ∈ ℱ ⁡ ( s ) } \Fop(s,\gamma)\coloneqq\left\{\mathbf{f}(\gamma)\mid\mathbf{f}\in\Fop(s)\right\} and ℱ nd ⁡ ( s , γ ) ≔ { 𝐟 ⁡ ( γ ) ∣ 𝐟 ∈ ℱ nd ⁡ ( s ) } \Fndop(s,\gamma)\coloneqq\left\{\mathbf{f}(\gamma)\mid\mathbf{f}\in\Fndop(s)\right\} . If F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) , then F ⁡ ( γ ) ≔ { 𝐟 ⁡ ( γ ) ∣ 𝐟 ∈ F } F(\gamma)\coloneqq\left\{\mathbf{f}(\gamma)\mid\mathbf{f}\in F\right\} .

###### Lemma E.31 (Non-domination across γ \gamma values for expectations of visit distributions) .

Let Δ d ∈ Δ ⁡ ( ℝ | 𝒮 | ) \Delta_{d}\in\Delta\left(\mathbb{R}^{\left|\mathcal{S}\right|}\right) be any state distribution and let F ≔ { 𝔼 s d ∼ Δ d [ 𝐟 π , s d ] ∣ π ∈ Π } F\coloneqq\left\{\opE_{s_{d}\sim\Delta_{d}}\left[\mathbf{f}^{\pi,s_{d}}\right]\mid\pi\in\Pi\right\} . 𝐟 ∈ ND ​ ( F ) \mathbf{f}\in\text{{ND}}\left(F\right) iff ∀ γ ∗ ∈ ( 0 , 1 ) : 𝐟 ⁡ ( γ ∗ ) ∈ ND ​ ( F ⁡ ( γ ∗ ) ) \forall\gamma^{*}\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}(\gamma^{*})\in\text{{ND}}\left(F(\gamma^{*})\right) .

###### Proof.

Let 𝐟 π ∈ ND ​ ( F ) \mathbf{f}^{\pi}\in\text{{ND}}\left(F\right) be strictly optimal for reward function R R at discount rate γ ∈ ( 0 , 1 ) \gamma\in(0,1) : 𝐟 π ​ ( γ ) ⊤ ​ 𝐫 > max 𝐟 π ′ ∈ F ∖ { 𝐟 π } ⁡ 𝐟 π ′ ​ ( γ ) ⊤ ​ 𝐫 . \displaystyle\mathbf{f}^{\pi}(\gamma)^{\top}\mathbf{r}>\max_{\mathbf{f}^{\pi^{\prime}}\in F\setminus\left\{\mathbf{f}^{\pi}\right\}}\mathbf{f}^{\pi^{\prime}}(\gamma)^{\top}\mathbf{r}. (77)

Let γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) . By section D.1.2 , we can produce R ′ R^{\prime} such that Π ∗ ​ ( R ′ , γ ∗ ) = Π ∗ ​ ( R , γ ) \Pi^{*}\left(R^{\prime},\gamma^{*}\right)=\Pi^{*}\left(R,\gamma\right) . Since the optimal policy sets are equal, Lemma E.1 implies that 𝐟 π ​ ( γ ∗ ) ⊤ ​ 𝐫 ′ > max 𝐟 π ′ ∈ F ∖ { 𝐟 π } ⁡ 𝐟 π ′ ​ ( γ ∗ ) ⊤ ​ 𝐫 ′ . \displaystyle\mathbf{f}^{\pi}(\gamma^{*})^{\top}\mathbf{r}^{\prime}>\max_{\mathbf{f}^{\pi^{\prime}}\in F\setminus\left\{\mathbf{f}^{\pi}\right\}}\mathbf{f}^{\pi^{\prime}}(\gamma^{*})^{\top}\mathbf{r}^{\prime}. (78)

Therefore, 𝐟 π ​ ( γ ∗ ) ∈ ND ​ ( F ⁡ ( γ ∗ ) ) \mathbf{f}^{\pi}(\gamma^{*})\in\text{{ND}}\left(F(\gamma^{*})\right) .

The reverse direction follows by the definition of ND ​ ( F ) \text{{ND}}\left(F\right) . ∎

###### Lemma E.32 ( ∀ γ ∈ ( 0 , 1 ) : 𝐝 ∈ ℱ nd ⁡ ( s , γ ) \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{d}\in\Fndop(s,\gamma) iff 𝐝 ∈ ND ​ ( ℱ ⁡ ( s , γ ) ) \mathbf{d}\in\text{{ND}}\left(\Fop(s,\gamma)\right) ) .

###### Proof.

By definition E.30 , ℱ nd ⁡ ( s , γ ) ≔ { 𝐟 ⁡ ( γ ) ∣ 𝐟 ∈ ND ​ ( ℱ ⁡ ( s ) ) } \Fndop(s,\gamma)\coloneqq\left\{\mathbf{f}(\gamma)\mid\mathbf{f}\in\text{{ND}}\left(\Fop(s)\right)\right\} . By applying Lemma E.31 with Δ d ≔ 𝐞 s \Delta_{d}\coloneqq\mathbf{e}_{s} , 𝐟 ∈ ND ​ ( ℱ ⁡ ( s ) ) \mathbf{f}\in\text{{ND}}\left(\Fop(s)\right) iff ∀ γ ∈ ( 0 , 1 ) : 𝐟 ⁡ ( γ ) ∈ ND ​ ( ℱ ⁡ ( s , γ ) ) \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}(\gamma)\in\text{{ND}}\left(\Fop(s,\gamma)\right) . ∎

\optVfFndRestrict

###### Proof.

ND ​ ( ℱ ⁡ ( s , γ ) ) = ℱ nd ⁡ ( s , γ ) \text{{ND}}\left(\Fop(s,\gamma)\right)=\Fndop(s,\gamma) by Lemma E.32 , so apply Corollary E.11 with X ≔ ℱ ⁡ ( s , γ ) X\coloneqq\Fop(s,\gamma) . ∎

### E.2 Some actions have greater probability of being optimal

###### Lemma E.33 (Optimal policy shift bound) .

For fixed R R , Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) can take on at most ( 2 ​ | 𝒮 | + 1 ) ​ ∑ s ( | ℱ ⁡ ( s ) | 2 ) (2\left|\mathcal{S}\right|+1)\sum_{s}\binom{\left|\Fop(s)\right|}{2} distinct values over γ ∈ ( 0 , 1 ) \gamma\in(0,1) .

###### Proof.

By Lemma E.1 , Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) changes value iff there is a change in optimality status for some visit distribution function at some state. Lippman [1968] showed that two visit distribution functions can trade off optimality status at most 2 ​ | 𝒮 | + 1 2\left|\mathcal{S}\right|+1 times. At each state s s , there are ( | ℱ ⁡ ( s ) | 2 ) \binom{\left|\Fop(s)\right|}{2} such pairs. ∎

###### Proposition E.34 (Optimality probability’s limits exist) .

Let F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) . ℙ 𝒟 any ( F , 0 ) = lim γ → 0 ℙ 𝒟 any ( F , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,0\right)=\lim_{\gamma\to 0}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right) and ℙ 𝒟 any ( F , 1 ) = lim γ → 1 ℙ 𝒟 any ( F , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,1\right)=\lim_{\gamma\to 1}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right) .

###### Proof.

First consider the limit as γ → 1 \gamma\to 1 . Let 𝒟 any \mathcal{D}_{\text{any}} have probability measure F any F_{\text{any}} , and define δ ⁡ ( γ ) ≔ F any ​ ( { R ∈ ℝ 𝒮 ∣ ∃ γ ∗ ∈ [ γ , 1 ) : Π ∗ ​ ( R , γ ∗ ) ≠ Π ∗ ​ ( R , 1 ) } ) \delta(\gamma)\coloneqq F_{\text{any}}\left(\left\{R\in\mathbb{R}^{\mathcal{S}}\mid\exists\gamma^{*}\in[\gamma,1)\mathrel{\mathop{\ordinarycolon}}\Pi^{*}\left(R,\gamma^{*}\right)\neq\Pi^{*}\left(R,1\right)\right\}\right) . Since F any F_{\text{any}} is a probability measure, δ ⁡ ( γ ) \delta(\gamma) is bounded [ 0 , 1 ] [0,1] , and δ ⁡ ( γ ) \delta(\gamma) is monotone decreasing. Therefore, lim γ → 1 δ ⁡ ( γ ) \lim_{\gamma\to 1}\delta(\gamma) exists.

If lim γ → 1 δ ⁡ ( γ ) > 0 \lim_{\gamma\to 1}\delta(\gamma)>0 , then there exist reward functions whose optimal policy sets Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma\right) never converge (in the discrete topology on sets) to Π ∗ ​ ( R , 1 ) \Pi^{*}\left(R,1\right) , contradicting Lemma E.33 . So lim γ → 1 δ ⁡ ( γ ) = 0 \lim_{\gamma\to 1}\delta(\gamma)=0 .

By the definition of optimality probability ( definition 4.3 ) and of δ ⁡ ( γ ) \delta(\gamma) , | ℙ 𝒟 any ( F , γ ) − ℙ 𝒟 any ( F , 1 ) | ≤ δ ⁡ ( γ ) |\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)-\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,1\right)|\leq\delta(\gamma) . Since lim γ → 1 δ ⁡ ( γ ) = 0 \lim_{\gamma\to 1}\delta(\gamma)=0 , lim γ → 1 ℙ 𝒟 any ( F , γ ) = ℙ 𝒟 any ( F , 1 ) \lim_{\gamma\to 1}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,1\right) .

A similar proof shows that lim γ → 0 ℙ 𝒟 any ( F , γ ) = ℙ 𝒟 any ( F , 0 ) \lim_{\gamma\to 0}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,0\right) . ∎

###### Lemma E.35 (Optimality probability identity) .

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) and let F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) . ℙ 𝒟 any ( F , γ ) = p 𝒟 ′ ​ ( F ⁡ ( γ ) ≥ ℱ ⁡ ( s , γ ) ) = p 𝒟 ′ ​ ( F ⁡ ( γ ) ≥ ℱ nd ⁡ ( s , γ ) ) . \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right)=p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fop(s,\gamma)\right)=p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fndop(s,\gamma)\right). (79)

###### Proof.

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . ℙ 𝒟 any ( F , γ ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F,\gamma\right) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐟 π ∈ F : π ∈ Π ∗ ​ ( R , γ ) ) \displaystyle\coloneqq\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{f}^{\pi}\in F\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{*}\left(R,\gamma\right)\right) (80) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐟 ∈ F ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 = max 𝐟 ′ ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{f}\in F}\mathbf{f}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}}\right] (81) = 𝔼 𝐫 ∼ 𝒟 any [ 𝟙 max 𝐟 ∈ F ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 = max 𝐟 ′ ∈ ℱ nd ⁡ ( s ) ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{f}\in F}\mathbf{f}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fndop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}}\right] (82) ≕ p 𝒟 ′ ​ ( F ⁡ ( γ ) ≥ ℱ nd ⁡ ( s , γ ) ) . \displaystyle\eqqcolon p_{\mathcal{D}^{\prime}}\left(F(\gamma)\geq\Fndop(s,\gamma)\right). (83) Equation 81 follows because Lemma E.1 shows that π \pi is optimal iff it induces an optimal visit distribution 𝐟 \mathbf{f} at every state. Equation 82 follows because ∀ 𝐫 ∈ ℝ | 𝒮 | : max 𝐟 ′ ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 = max 𝐟 ′ ∈ ℱ nd ⁡ ( s ) ⁡ 𝐟 ′ ​ ( γ ) ⊤ ​ 𝐫 \forall\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\max_{\mathbf{f}^{\prime}\in\Fop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}^{\prime}\in\Fndop(s)}\mathbf{f}^{\prime}(\gamma)^{\top}\mathbf{r} by section D.1.1 . ∎

### E.3 Basic properties of Power

###### Lemma E.36 ( Power identities) .

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (84) = 1 − γ γ ​ 𝔼 𝐫 ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) − R ⁡ ( s ) ] \displaystyle=\dfrac{1-\gamma}{\gamma}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right] (85) = 1 − γ γ ​ ( V 𝒟 bound ∗ ​ ( s , γ ) − 𝔼 R ∼ 𝒟 bound [ R ⁡ ( s ) ] ) \displaystyle=\dfrac{1-\gamma}{\gamma}\left(V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s)\right]\right) (86) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ V R π ​ ( s ′ , γ ) ] ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right]. (87)

###### Proof.

Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}(s,\gamma) ≔ 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle\coloneqq\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (88) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (89) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 ⁡ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fop(s)}\frac{1-\gamma}{\gamma}\left(\mathbf{f}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (90) = 1 − γ γ ​ 𝔼 𝐫 ∼ 𝒟 bound [ V R ∗ ​ ( s , γ ) − R ⁡ ( s ) ] \displaystyle=\dfrac{1-\gamma}{\gamma}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s,\gamma\right)-R(s)\right] (91) = 1 − γ γ ​ ( V 𝒟 bound ∗ ​ ( s , γ ) − 𝔼 R ∼ 𝒟 bound [ R ⁡ ( s ) ] ) \displaystyle=\dfrac{1-\gamma}{\gamma}\left(V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s)\right]\right) (92) = 𝔼 𝐫 ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ 𝐟 π , s ′ ​ ( γ ) ⊤ ​ 𝐫 ] ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)\mathbf{f}^{\pi,s^{\prime}}(\gamma)^{\top}\mathbf{r}\right]\right] (93) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ V R π ​ ( s ′ , γ ) ] ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\left(1-\gamma\right)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right]. (94)

Equation 89 follows from section D.1.1 . Equation 91 follows from the dual formulation of optimal value functions. Equation 92 holds by the definition of V 𝒟 bound ∗ ​ ( s , γ ) V^{*}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) ( definition 5.1 ). Equation 93 holds because 𝐟 π , s ​ ( γ ) = 𝐞 s + γ ​ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ 𝐟 π , s ′ ​ ( γ ) ] \mathbf{f}^{\pi,s}(\gamma)=\mathbf{e}_{s}+\gamma\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[\mathbf{f}^{\pi,s^{\prime}}(\gamma)\right] by the definition of a visit distribution function ( definition 3.3 ). ∎

###### Definition E.37 (Discount-normalized value function) .

Let π \pi be a policy, R R a reward function, and s s a state. For γ ∈ [ 0 , 1 ] \gamma\in[0,1] , V R , norm π ​ ( s , γ ) ≔ lim γ ∗ → γ ( 1 − γ ∗ ) ​ V R π ​ ( s , γ ∗ ) V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)\coloneqq\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{\pi}_{R}(s,\gamma^{*}) .

###### Lemma E.38 (Normalized value functions have uniformly bounded derivative) .

There exists K ≥ 0 K\geq 0 such that for all reward functions 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ] | d d ​ γ ​ V R , norm π ​ ( s , γ ) | ≤ K ​ ‖ 𝐫 ‖ 1 \sup_{\begin{subarray}{c}s\in\mathcal{S},\pi\in\Pi,\gamma\in[0,1]\end{subarray}}\left|\frac{d}{d\gamma}V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)\right|\leq K\left\lVert\mathbf{r}\right\rVert_{1} .

###### Proof.

Let π \pi be any policy, s s a state, and R R a reward function. Since V R , norm π ​ ( s , γ ) = lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 π , s ​ ( γ ∗ ) ⊤ ​ 𝐫 V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right)=\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*})^{\top}\mathbf{r} , d d ​ γ ​ V R , norm π ​ ( s , γ ) \frac{d}{d\gamma}V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right) is controlled by the behavior of lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 π , s ​ ( γ ∗ ) \lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*}) . We show that this function’s gradient is bounded in infinity norm.

By Lemma E.4 , 𝐟 π , s ​ ( γ ) \mathbf{f}^{\pi,s}(\gamma) is a multivariate rational function on γ \gamma . Therefore, for any state s ′ s^{\prime} , 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐞 s ′ = P ⁡ ( γ ) Q ⁡ ( γ ) \mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}}=\frac{P(\gamma)}{Q(\gamma)} in reduced form. By Proposition E.3 , 0 ≤ 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐞 s ′ ≤ 1 1 − γ 0\leq\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}}\leq\frac{1}{1-\gamma} . Thus, Q Q may only have a root of multiplicity 1 at γ = 1 \gamma=1 , and Q ⁡ ( γ ) ≠ 0 Q(\gamma)\neq 0 for γ ∈ [ 0 , 1 ) \gamma\in[0,1) . Let f s ′ ​ ( γ ) ≔ ( 1 − γ ) ​ 𝐟 π , s ​ ( γ ) ⊤ ​ 𝐞 s ′ f_{s^{\prime}}(\gamma)\coloneqq(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{e}_{s^{\prime}} .

If Q ⁡ ( 1 ) ≠ 0 Q(1)\neq 0 , then the derivative f s ′ ′ ​ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) because the polynomial ( 1 − γ ) ​ P ​ ( γ ) (1-\gamma)P(\gamma) cannot diverge on a bounded domain.

If Q ⁡ ( 1 ) = 0 Q(1)=0 , then factor out the root as Q ⁡ ( γ ) = ( 1 − γ ) ​ Q ∗ ​ ( γ ) Q(\gamma)=(1-\gamma)Q^{*}(\gamma) . f s ′ ′ ​ ( γ ) \displaystyle f_{s^{\prime}}^{\prime}(\gamma) = d d ​ γ ​ ( ( 1 − γ ) ​ P ​ ( γ ) Q ⁡ ( γ ) ) \displaystyle=\frac{d}{d\gamma}\left(\frac{(1-\gamma)P(\gamma)}{Q(\gamma)}\right) (95) = d d ​ γ ​ ( P ⁡ ( γ ) Q ∗ ​ ( γ ) ) \displaystyle=\frac{d}{d\gamma}\left(\frac{P(\gamma)}{Q^{*}(\gamma)}\right) (96) = P ′ ​ ( γ ) ​ Q ∗ ​ ( γ ) − ( Q ∗ ) ′ ​ ( γ ) ​ P ​ ( γ ) ( Q ∗ ​ ( γ ) ) 2 . \displaystyle=\frac{P^{\prime}(\gamma)Q^{*}(\gamma)-(Q^{*})^{\prime}(\gamma)P(\gamma)}{(Q^{*}(\gamma))^{2}}. (97)

Since Q ∗ ​ ( γ ) Q^{*}(\gamma) is a polynomial with no roots on γ ∈ [ 0 , 1 ] \gamma\in[0,1] , f s ′ ′ ​ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) .

Therefore, whether or not Q ⁡ ( γ ) Q(\gamma) has a root at γ = 1 \gamma=1 , f s ′ ′ ​ ( γ ) f_{s^{\prime}}^{\prime}(\gamma) is bounded on γ ∈ [ 0 , 1 ) \gamma\in[0,1) . Furthermore, sup γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ ∞ = sup γ ∈ [ 0 , 1 ) max s ′ ∈ 𝒮 | f s ′ ′ ( γ ) | \sup_{\gamma\in[0,1)}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{\infty}=\sup_{\gamma\in[0,1)}\max_{s^{\prime}\in\mathcal{S}}\left|f_{s^{\prime}}^{\prime}(\gamma)\right| is finite since there are only finitely many states.

There are finitely many π ∈ Π \pi\in\Pi , and finitely many states s s , and so there exists some K ′ K^{\prime} such that sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ ∞ ≤ K ′ \sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{\infty}\leq K^{\prime} . Then ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ 1 ≤ | 𝒮 | K ′ ≕ K \left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1}\leq\left|\mathcal{S}\right|K^{\prime}\eqqcolon K . sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d ​ γ ​ V R , norm π ​ ( s , γ ) | ≔ \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}V^{\pi}_{R,\text{norm}}\left(s,\gamma\right)\right|\coloneqq\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d ​ γ ​ lim γ ∗ → γ ( 1 − γ ∗ ) ​ V R π ​ ( s , γ ∗ ) | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})V^{\pi}_{R}\left(s,\gamma^{*}\right)\right| (98) = \displaystyle=\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | d d ​ γ ​ ( 1 − γ ) ​ V R π ​ ( s , γ ) | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\frac{d}{d\gamma}(1-\gamma)V^{\pi}_{R}\left(s,\gamma\right)\right| (99) = \displaystyle=\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) | ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ⊤ 𝐫 | \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left|\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)^{\top}\mathbf{r}\right| (100) ≤ \displaystyle\leq\, sup s ∈ 𝒮 , π ∈ Π , γ ∈ [ 0 , 1 ) ‖ ∇ ( 1 − γ ) 𝐟 π , s ( γ ) ‖ 1 ‖ 𝐫 ‖ 1 \displaystyle\sup_{\begin{subarray}{c}s\in\mathcal{S},\\ \pi\in\Pi,\gamma\in[0,1)\end{subarray}}\left\lVert\nabla(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\right\rVert_{1}\left\lVert\mathbf{r}\right\rVert_{1} (101) ≤ \displaystyle\leq\, K ​ ‖ 𝐫 ‖ 1 . \displaystyle K\left\lVert\mathbf{r}\right\rVert_{1}. (102)

Equation 99 holds because V R π ​ ( s , γ ) V^{\pi}_{R}\left(s,\gamma\right) is continuous on γ ∈ [ 0 , 1 ) \gamma\in[0,1) by Corollary E.5 . Equation 101 holds by the Cauchy-Schwarz inequality.

Since | d d ​ γ ​ V R , norm π ​ ( s , γ ) | \left|\frac{d}{d\gamma}V^{\pi}_{R,\text{norm}}\left(s,\gamma\right)\right| is bounded for all γ ∈ [ 0 , 1 ) \gamma\in[0,1) , eq. 102 also holds for γ → 1 \gamma\to 1 . ∎

See 5.3

###### Proof.

Let b , c b,c be such that supp ⁡ ( 𝒟 bound ) ⊆ [ b , c ] | 𝒮 | \optSupp(\mathcal{D}_{\text{bound}})\subseteq[b,c]^{\left|\mathcal{S}\right|} . For any 𝐫 ∈ supp ⁡ ( 𝒟 bound ) \mathbf{r}\in\optSupp(\mathcal{D}_{\text{bound}}) and π ∈ Π \pi\in\Pi , V R , norm π ​ ( s , γ ) V^{\pi}_{R,\,\text{norm}}\left(s,\gamma\right) has Lipschitz constant K ​ ‖ 𝐫 ‖ 1 ≤ K ​ | 𝒮 | ​ ‖ 𝐫 ‖ ∞ ≤ K ​ | 𝒮 | ​ max ⁡ ( | c | , | b | ) K\left\lVert\mathbf{r}\right\rVert_{1}\leq K\left|\mathcal{S}\right|\left\lVert\mathbf{r}\right\rVert_{\infty}\leq K\left|\mathcal{S}\right|\max(\left|c\right|,\left|b\right|) on γ ∈ ( 0 , 1 ) \gamma\in(0,1) by Lemma E.38 .

For γ ∈ ( 0 , 1 ) \gamma\in(0,1) , Power 𝒟 bound ​ ( s , γ ) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ V R π ​ ( s ′ , γ ) ] ] \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T\left(s,\pi(s)\right)}\left[(1-\gamma)V^{\pi}_{R}\left(s^{\prime},\gamma\right)\right]\right] by eq. 94 . The expectation of the maximum of a set of functions which share a Lipschitz constant, also shares the Lipschitz constant. This shows that Power 𝒟 bound ​ ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) is Lipschitz continuous on γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Thus, its limits are well-defined as γ → 0 \gamma\to 0 and γ → 1 \gamma\to 1 . So it is Lipschitz continuous on the closed unit interval. ∎

See 5.4

###### Proof.

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = 𝔼 R ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ V R ∗ ​ ( s ′ , γ ) ] ] \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T(s,\pi(s))}\left[(1-\gamma)V^{*}_{R}\left(s^{\prime},\gamma\right)\right]\right] (103) ≤ 𝔼 R ∼ 𝒟 bound [ max π ∈ Π ⁡ 𝔼 s ′ ∼ T ⁡ ( s , π ⁡ ( s ) ) [ ( 1 − γ ) ​ max s ′′ ∈ 𝒮 ⁡ R ⁡ ( s ′′ ) 1 − γ ] ] \displaystyle\leq\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\opE_{s^{\prime}\sim T(s,\pi(s))}\left[(1-\gamma)\frac{\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})}{1-\gamma}\right]\right] (104) = 𝔼 R ∼ 𝒟 bound [ max s ′′ ∈ 𝒮 ⁡ R ⁡ ( s ′′ ) ] . \displaystyle=\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})\right]. (105)

Equation 103 follows from Lemma E.36 . Equation 104 follows because V R ∗ ​ ( s ′ , γ ) ≤ max s ′′ ∈ 𝒮 ⁡ R ⁡ ( s ′′ ) 1 − γ V^{*}_{R}\left(s^{\prime},\gamma\right)\leq\frac{\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})}{1-\gamma} , as no policy can do better than achieving maximal reward at each time step. Taking limits, the inequality holds for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] .

Suppose that s s can deterministically reach all states in one step and all states are 1-cycles. Then eq. 104 is an equality for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , since for each R R , the agent can select an action which deterministically transitions to a state with maximal reward. Thus the equality holds for all γ ∈ [ 0 , 1 ] \gamma\in[0,1] . ∎

###### Lemma E.39 (Lower bound on current Power based on future Power ) .

Power 𝒟 bound ​ ( s , γ ) ≥ ( 1 − γ ) ​ min a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) , R ∼ 𝒟 bound [ R ⁡ ( s ′ ) ] + γ ​ max a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ​ ( s ′ , γ ) ] . \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\geq(1-\gamma)\min_{a}\opE_{\begin{subarray}{c}s^{\prime}\sim T(s,a),\\ R\sim\mathcal{D}_{\text{bound}}\end{subarray}}\left[R(s^{\prime})\right]+\gamma\max_{a}\opE_{s^{\prime}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right]. (106)

###### Proof.

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) and let a ∗ ∈ arg ​ max a ⁡ 𝔼 s ′ ∼ T ⁡ ( s , a ) ​ [ Power 𝒟 bound ​ ( s ′ , γ ) ] a^{*}\in\argmax_{a}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right] . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) (107) = \displaystyle=\, ( 1 − γ ) ​ 𝔼 R ∼ 𝒟 bound [ max a ⁡ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V R ∗ ​ ( s ′ , γ ) ] ] \displaystyle(1-\gamma)\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{a}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right]\right] (108) ≥ \displaystyle\geq\, ( 1 − γ ) ​ max a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ 𝔼 R ∼ 𝒟 bound [ V R ∗ ​ ( s ′ , γ ) ] ] \displaystyle(1-\gamma)\max_{a}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[V^{*}_{R}\left(s^{\prime},\gamma\right)\right]\right] (109) = \displaystyle=\, ( 1 − γ ) ​ max a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ V 𝒟 bound ∗ ​ ( s ′ , γ ) ] \displaystyle(1-\gamma)\max_{a}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[V^{*}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right] (110) = \displaystyle=\, ( 1 − γ ) ​ max a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) [ 𝔼 R ∼ 𝒟 bound [ R ⁡ ( s ′ ) ] + γ 1 − γ ​ Power 𝒟 bound ​ ( s ′ , γ ) ] \displaystyle(1-\gamma)\max_{a}\opE_{s^{\prime}\sim T\left(s,a\right)}\left[\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s^{\prime})\right]+\frac{\gamma}{1-\gamma}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right] (111) ≥ \displaystyle\geq\, ( 1 − γ ) ​ 𝔼 s ′ ∼ T ⁡ ( s , a ∗ ) [ 𝔼 R ∼ 𝒟 bound [ R ⁡ ( s ′ ) ] + γ 1 − γ ​ Power 𝒟 bound ​ ( s ′ , γ ) ] \displaystyle(1-\gamma)\opE_{s^{\prime}\sim T\left(s,a^{*}\right)}\left[\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[R(s^{\prime})\right]+\frac{\gamma}{1-\gamma}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right] (112) ≥ \displaystyle\geq\, ( 1 − γ ) ​ min a ​ 𝔼 s ′ ∼ T ⁡ ( s , a ) , R ∼ 𝒟 bound [ R ⁡ ( s ′ ) ] + γ ​ 𝔼 s ′ ∼ T ⁡ ( s , a ∗ ) [ Power 𝒟 bound ​ ( s ′ , γ ) ] . \displaystyle(1-\gamma)\min_{a}\opE_{\begin{subarray}{c}s^{\prime}\sim T(s,a),\\ R\sim\mathcal{D}_{\text{bound}}\end{subarray}}\left[R(s^{\prime})\right]+\gamma\opE_{s^{\prime}\sim T\left(s,a^{*}\right)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right]. (113)

Equation 108 holds by Lemma E.36 . Equation 109 follows because 𝔼 x ∼ X [ max a ⁡ f ⁡ ( a , x ) ] ≥ max a ⁡ 𝔼 x ∼ X [ f ⁡ ( a , x ) ] \opE_{x\sim X}\left[\max_{a}f(a,x)\right]\geq\max_{a}\opE_{x\sim X}\left[f(a,x)\right] by Jensen’s inequality, and eq. 111 follows by Lemma E.36 .

The inequality also holds when we take the limits γ → 0 \gamma\to 0 or γ → 1 \gamma\to 1 . ∎

See 5.5

###### Proof.

Suppose γ ∈ [ 0 , 1 ] \gamma\in[0,1] . First consider the case where Power 𝒟 bound ​ ( s , γ ) ≥ Power 𝒟 bound ​ ( s ′ , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\geq\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right) . Power 𝒟 bound ​ ( s ′ , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right) ≥ ( 1 − γ ) ​ min a ​ 𝔼 s x ∼ T ⁡ ( s ′ , a ) , R ∼ 𝒟 bound [ R ⁡ ( s x ) ] + γ ​ max a ​ 𝔼 s x ∼ T ⁡ ( s ′ , a ) [ Power 𝒟 bound ​ ( s x , γ ) ] \displaystyle\geq(1-\gamma)\min_{a}\opE_{\begin{subarray}{c}s_{x}\sim T(s^{\prime},a),\\ R\sim\mathcal{D}_{\text{bound}}\end{subarray}}\left[R(s_{x})\right]+\gamma\max_{a}\opE_{s_{x}\sim T(s^{\prime},a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{x},\gamma\right)\right] (114) ≥ ( 1 − γ ) ​ b + γ ​ Power 𝒟 bound ​ ( s , γ ) . \displaystyle\geq(1-\gamma)b+\gamma\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right). (115)

Equation 114 follows by Lemma E.39 . Equation 115 follows because reward is lower-bounded by b b and because s ′ s^{\prime} can reach s s in one step with probability 1. | Power 𝒟 bound ​ ( s , γ ) − Power 𝒟 bound ​ ( s ′ , γ ) | \displaystyle\left|\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\right| = Power 𝒟 bound ​ ( s , γ ) − Power 𝒟 bound ​ ( s ′ , γ ) \displaystyle=\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right) (116) ≤ Power 𝒟 bound ​ ( s , γ ) − ( ( 1 − γ ) ​ b + γ ​ Power 𝒟 bound ​ ( s , γ ) ) \displaystyle\leq\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-\left((1-\gamma)b+\gamma\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\right) (117) = ( 1 − γ ) ​ ( Power 𝒟 bound ​ ( s , γ ) − b ) \displaystyle=(1-\gamma)\left(\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)-b\right) (118) ≤ ( 1 − γ ) ​ ( 𝔼 R ∼ 𝒟 bound [ max s ′′ ∈ 𝒮 ⁡ R ⁡ ( s ′′ ) ] − b ) \displaystyle\leq(1-\gamma)\left(\opE_{R\sim\mathcal{D}_{\text{bound}}}\left[\max_{s^{\prime\prime}\in\mathcal{S}}R(s^{\prime\prime})\right]-b\right) (119) ≤ ( 1 − γ ) ​ ( c − b ) . \displaystyle\leq(1-\gamma)(c-b). (120)

Equation 116 follows because Power 𝒟 bound ​ ( s , γ ) ≥ Power 𝒟 bound ​ ( s ′ , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\geq\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right) . Equation 117 follows by eq. 115 . Equation 119 follows by Proposition 5.4 . Equation 120 follows because reward under 𝒟 bound \mathcal{D}_{\text{bound}} is upper-bounded by c c .

The case where Power 𝒟 bound ​ ( s , γ ) ≤ Power 𝒟 bound ​ ( s ′ , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right)\leq\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right) is similar, leveraging the fact that s s can also reach s ′ s^{\prime} in one step with probability 1. ∎

### E.4 Seeking Power is often more probable under optimality

#### E.4.1 Keeping options open tends to be Power -seeking and tends to be optimal

###### Definition E.40 (Normalized visit distribution function) .

Let 𝐟 : [ 0 , 1 ) → ℝ | 𝒮 | \mathbf{f}\mathrel{\mathop{\ordinarycolon}}[0,1)\to\mathbb{R}^{\left|\mathcal{S}\right|} be a vector function. For γ ∈ [ 0 , 1 ] \gamma\in[0,1] , Norm ​ ( 𝐟 , γ ) ≔ lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 ​ ( γ ∗ ) \text{{Norm}}\left(\mathbf{f},\gamma\right)\coloneqq\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}(\gamma^{*}) (this limit need not exist for arbitrary 𝐟 \mathbf{f} ). If F F is a set of such 𝐟 \mathbf{f} , then Norm ​ ( F , γ ) ≔ { Norm ​ ( 𝐟 , γ ) ∣ 𝐟 ∈ F } \text{{Norm}}\left(F,\gamma\right)\coloneqq\left\{\text{{Norm}}\left(\mathbf{f},\gamma\right)\mid\mathbf{f}\in F\right\} .

###### Remark .

RSD ​ ( s ) = Norm ​ ( ℱ ⁡ ( s ) , 1 ) \text{{RSD}}\left(s\right)=\text{{Norm}}\left(\Fop(s),1\right) .

###### Lemma E.41 (Normalized visit distribution functions are continuous) .

Let Δ s ∈ Δ ⁡ ( 𝒮 ) \Delta_{s}\in\Delta(\mathcal{S}) be a state probability distribution, let π ∈ Π \pi\in\Pi , and let 𝐟 ∗ ≔ 𝔼 s ∼ Δ s [ 𝐟 π , s ] \mathbf{f}^{*}\coloneqq\opE_{s\sim\Delta_{s}}\left[\mathbf{f}^{\pi,s}\right] . Norm ​ ( 𝐟 ∗ , γ ) \text{{Norm}}\left(\mathbf{f}^{*},\gamma\right) is continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] .

###### Proof.

Norm ​ ( 𝐟 ∗ , γ ) \displaystyle\text{{Norm}}\left(\mathbf{f}^{*},\gamma\right) ≔ lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝔼 s ∼ Δ s [ 𝐟 π , s ​ ( γ ∗ ) ] \displaystyle\coloneqq\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\opE_{s\sim\Delta_{s}}\left[\mathbf{f}^{\pi,s}(\gamma^{*})\right] (121) = 𝔼 s ∼ Δ s [ lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 π , s ​ ( γ ∗ ) ] \displaystyle=\opE_{s\sim\Delta_{s}}\left[\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*})\right] (122) ≕ 𝔼 s ∼ Δ s [ Norm ​ ( 𝐟 π , s , γ ) ] . \displaystyle\eqqcolon\opE_{s\sim\Delta_{s}}\left[\text{{Norm}}\left(\mathbf{f}^{\pi,s},\gamma\right)\right]. (123) Equation 122 follows because the expectation is over a finite set. Each 𝐟 π , s ∈ ℱ ⁡ ( s ) \mathbf{f}^{\pi,s}\in\Fop(s) is continuous on γ ∈ [ 0 , 1 ) \gamma\in[0,1) by Lemma E.4 , and lim γ ∗ → 1 ( 1 − γ ∗ ) ​ 𝐟 π , s ​ ( γ ∗ ) \lim_{\gamma^{*}\to 1}(1-\gamma^{*})\mathbf{f}^{\pi,s}(\gamma^{*}) exists because rsd s are well-defined [ Puterman, 2014 ] . Therefore, each Norm ​ ( 𝐟 π , s , γ ) \text{{Norm}}\left(\mathbf{f}^{\pi,s},\gamma\right) is continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] . Lastly, eq. 123 ’s expectation over finitely many continuous functions is itself continuous. ∎

###### Lemma E.42 (Non-domination of normalized visit distribution functions) .

Let Δ s ∈ Δ ⁡ ( 𝒮 ) \Delta_{s}\in\Delta(\mathcal{S}) be a state probability distribution and let F ≔ { 𝔼 s ∼ Δ s [ 𝐟 π , s ] ∣ π ∈ Π } F\coloneqq\left\{\opE_{s\sim\Delta_{s}}\left[\mathbf{f}^{\pi,s}\right]\mid\pi\in\Pi\right\} . For all γ ∈ [ 0 , 1 ] \gamma\in[0,1] , ND ​ ( Norm ​ ( F , γ ) ) ⊆ Norm ​ ( ND ​ ( F ) , γ ) \text{{ND}}\left(\text{{Norm}}\left(F,\gamma\right)\right)\subseteq\text{{Norm}}\left(\text{{ND}}\left(F\right),\gamma\right) , with equality when γ ∈ ( 0 , 1 ) \gamma\in(0,1) .

###### Proof.

Suppose γ ∈ ( 0 , 1 ) \gamma\in(0,1) . ND ​ ( Norm ​ ( F , γ ) ) \displaystyle\text{{ND}}\left(\text{{Norm}}\left(F,\gamma\right)\right) = ND ​ ( ( 1 − γ ) ​ F ​ ( γ ) ) \displaystyle=\text{{ND}}\left((1-\gamma)F(\gamma)\right) (124) = ( 1 − γ ) ​ ND ​ ( F ⁡ ( γ ) ) \displaystyle=(1-\gamma)\text{{ND}}\left(F(\gamma)\right) (125) = ( 1 − γ ) ​ ( ND ​ ( F ) ​ ( γ ) ) \displaystyle=(1-\gamma)\left(\text{{ND}}\left(F\right)(\gamma)\right) (126) = Norm ​ ( ND ​ ( F ) , γ ) . \displaystyle=\text{{Norm}}\left(\text{{ND}}\left(F\right),\gamma\right). (127)

Equation 124 and eq. 127 follow by the continuity of Norm ​ ( 𝐟 , γ ) \text{{Norm}}\left(\mathbf{f},\gamma\right) ( Lemma E.41 ). Equation 125 follows by Lemma E.15 item 1 . Equation 126 follows by Lemma E.31 .

Let γ = 1 \gamma=1 . Let 𝐝 ∈ ND ​ ( Norm ​ ( F , 1 ) ) \mathbf{d}\in\text{{ND}}\left(\text{{Norm}}\left(F,1\right)\right) be strictly optimal for 𝐫 ∗ ∈ ℝ | 𝒮 | \mathbf{r}^{*}\in\mathbb{R}^{\left|\mathcal{S}\right|} . Then let F 𝐝 ⊆ F F_{\mathbf{d}}\subseteq F be the subset of 𝐟 ∈ F \mathbf{f}\in F such that Norm ​ ( 𝐟 , 1 ) = 𝐝 \text{{Norm}}\left(\mathbf{f},1\right)=\mathbf{d} . max 𝐟 ∈ F 𝐝 ⁡ Norm ​ ( 𝐟 , 1 ) ⊤ ​ 𝐫 ∗ \displaystyle\max_{\mathbf{f}\in F_{\mathbf{d}}}\text{{Norm}}\left(\mathbf{f},1\right)^{\top}\mathbf{r}^{*} > max 𝐟 ′ ∈ F ∖ F 𝐝 ⁡ Norm ​ ( 𝐟 ′ , 1 ) ⊤ ​ 𝐫 ∗ . \displaystyle>\max_{\mathbf{f}^{\prime}\in F\setminus F_{\mathbf{d}}}\text{{Norm}}\left(\mathbf{f}^{\prime},1\right)^{\top}\mathbf{r}^{*}. (128)

Since Norm ​ ( 𝐟 , 1 ) \text{{Norm}}\left(\mathbf{f},1\right) is continuous at γ = 1 \gamma=1 ( Lemma E.41 ), 𝐱 ⊤ ​ 𝐫 ∗ \mathbf{x}^{\top}\mathbf{r}^{*} is continuous on 𝐱 ∈ ℝ | 𝒮 | \mathbf{x}\in\mathbb{R}^{\left|\mathcal{S}\right|} , and F F is finite, eq. 128 holds for some γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) sufficiently close to γ = 1 \gamma=1 . By Lemma E.10 , at least one 𝐟 ∈ F 𝐝 \mathbf{f}\in F_{\mathbf{d}} is an element of ND ​ ( F ​ ( γ ∗ ) ) \text{{ND}}\left(F(\gamma^{*})\right) . Then by Lemma E.31 , 𝐟 ∈ ND ​ ( F ) \mathbf{f}\in\text{{ND}}\left(F\right) . We conclude that ND ​ ( Norm ​ ( F , 1 ) ) ⊆ Norm ​ ( ND ​ ( F ) , 1 ) \text{{ND}}\left(\text{{Norm}}\left(F,1\right)\right)\subseteq\text{{Norm}}\left(\text{{ND}}\left(F\right),1\right) .

The case for γ = 0 \gamma=0 proceeds similarly. ∎

###### Lemma E.43 ( Power limit identity) .

Let γ ∈ [ 0 , 1 ] \gamma\in[0,1] . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 ⁡ ( γ ∗ ) − 𝐞 s ) ⊤ ​ 𝐫 ] . \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}(\gamma^{*})-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right]. (129)

###### Proof.

Let γ ∈ [ 0 , 1 ] \gamma\in[0,1] . Power 𝒟 bound ​ ( s , γ ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) = lim γ ∗ → γ Power 𝒟 bound ​ ( s , γ ∗ ) \displaystyle=\lim_{\gamma^{*}\to\gamma}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma^{*}\right) (130) = lim γ ∗ → γ 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 ⁡ ( γ ∗ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\lim_{\gamma^{*}\to\gamma}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}(\gamma^{*})-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (131) = 𝔼 𝐫 ∼ 𝒟 bound [ lim γ ∗ → γ max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 ⁡ ( γ ∗ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\lim_{\gamma^{*}\to\gamma}\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}(\gamma^{*})-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (132) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 ⁡ ( γ ∗ ) − 𝐞 s ) ⊤ ​ 𝐫 ] . \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}\in\Fndop(s)}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}(\gamma^{*})-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right]. (133) Equation 130 follows because Power 𝒟 bound ​ ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) is continuous on γ ∈ [ 0 , 1 ] \gamma\in[0,1] by Lemma 5.3 . Equation 131 follows by Lemma E.36 .

For γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) , let f γ ∗ ​ ( 𝐫 ) ≔ max 𝐟 ∈ ℱ nd ⁡ ( s ) ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 ⁡ ( γ ∗ ) − 𝐞 s ) ⊤ ​ 𝐫 f_{\gamma^{*}}(\mathbf{r})\coloneqq\max_{\mathbf{f}\in\Fndop(s)}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}(\gamma^{*})-\mathbf{e}_{s}\right)^{\top}\mathbf{r} . For any sequence γ n → γ \gamma_{n}\to\gamma , ( f γ n ) n = 1 ∞ \left(f_{\gamma_{n}}\right)_{n=1}^{\infty} is a sequence of functions which are piecewise linear on 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , which means they are continuous and therefore measurable. Since Lemma E.4 shows that each 𝐟 ∈ ℱ nd ⁡ ( s ) \mathbf{f}\in\Fndop(s) is multivariate rational on γ ∗ \gamma^{*} (and therefore continuous on γ ∗ \gamma^{*} ), { f γ n } n = 1 ∞ \left\{f_{\gamma_{n}}\right\}_{n=1}^{\infty} converges pointwise to limit function f γ f_{\gamma} . Furthermore, | V R ∗ ​ ( s , γ n ) − R ⁡ ( s ) | ≤ γ 1 − γ n ​ ‖ R ‖ ∞ \left|V^{*}_{R}\left(s,\gamma_{n}\right)-R(s)\right|\leq\frac{\gamma}{1-\gamma_{n}}\left\lVert R\right\rVert_{\infty} , and so | f γ n ​ ( 𝐫 ) | = | 1 − γ n γ n ​ ( V R ∗ ​ ( s , γ n ) − R ⁡ ( s ) ) | ≤ g ⁡ ( 𝐫 ) ≤ ‖ 𝐫 ‖ ∞ ≕ g ⁡ ( 𝐫 ) \left|f_{\gamma_{n}}(\mathbf{r})\right|=\left|\frac{1-\gamma_{n}}{\gamma_{n}}(V^{*}_{R}\left(s,\gamma_{n}\right)-R(s))\right|\leq g(\mathbf{r})\leq\left\lVert\mathbf{r}\right\rVert_{\infty}\eqqcolon g(\mathbf{r}) , which is measurable. Therefore, apply Lebesgue’s dominated convergence theorem to conclude that eq. 132 holds. Equation 133 holds because max \max is a continuous function. ∎

###### Lemma E.44 (Lemma for Power superiority) .

Let Δ 1 , Δ 2 ∈ Δ ⁡ ( 𝒮 ) \Delta_{1},\Delta_{2}\in\Delta\left(\mathcal{S}\right) be state probability distributions. For i = 1 , 2 i=1,2 , let F Δ i ≔ { γ − 1 ​ 𝔼 s i ∼ Δ i [ 𝐟 π , s i − 𝐞 s i ] ∣ π ∈ Π } F_{\Delta_{i}}\coloneqq\left\{\gamma^{-1}\opE_{s_{i}\sim\Delta_{i}}\left[\mathbf{f}^{\pi,s_{i}}-\mathbf{e}_{s_{i}}\right]\mid\pi\in\Pi\right\} . Suppose F Δ 2 F_{\Delta_{2}} contains a copy of ND ​ ( F Δ 1 ) \text{{ND}}\left(F_{\Delta_{1}}\right) via ϕ \phi . Then ∀ γ ∈ [ 0 , 1 ] : 𝔼 s 1 ∼ Δ 1 [ Power 𝒟 bound ( s 1 , γ ) ] ≤ most : 𝔇 bound 𝔼 s 2 ∼ Δ 2 [ Power 𝒟 bound ( s 2 , γ ) ] \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\opE_{s_{1}\sim\Delta_{1}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{1},\gamma\right)\right]\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{2}\sim\Delta_{2}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},\gamma\right)\right] .

If ND ​ ( F Δ 2 ) ∖ ϕ ⋅ ND ​ ( F Δ 1 ) \text{{ND}}\left(F_{\Delta_{2}}\right)\setminus\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right) is non-empty, then for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} and 𝔼 s 1 ∼ Δ 1 [ Power 𝒟 bound ( s 1 , γ ) ] ≱ most : 𝔇 bound 𝔼 s 2 ∼ Δ 2 [ Power 𝒟 bound ( s 2 , γ ) ] \opE_{s_{1}\sim\Delta_{1}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{1},\gamma\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{2}\sim\Delta_{2}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},\gamma\right)\right] .

These results also hold when replacing F Δ i F_{\Delta_{i}} with F Δ i ∗ ≔ { 𝔼 s i ∼ Δ i [ 𝐟 π , s i ] ∣ π ∈ Π } F_{\Delta_{i}}^{*}\coloneqq\left\{\opE_{s_{i}\sim\Delta_{i}}\left[\mathbf{f}^{\pi,s_{i}}\right]\mid\pi\in\Pi\right\} for i = 1 , 2 i=1,2 .

###### Proof.

ϕ ⋅ ND ​ ( Norm ​ ( F Δ 1 , γ ) ) \displaystyle\phi\cdot\text{{ND}}\left(\text{{Norm}}\left(F_{\Delta_{1}},\gamma\right)\right) ⊆ ϕ ⋅ Norm ​ ( ND ​ ( F Δ 1 ) , γ ) \displaystyle\subseteq\phi\cdot\text{{Norm}}\left(\text{{ND}}\left(F_{\Delta_{1}}\right),\gamma\right) (134) ≔ { 𝐏 ϕ ​ lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 ​ ( γ ∗ ) ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } \displaystyle\coloneqq\left\{\mathbf{P}_{\phi}\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}(\gamma^{*})\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\} (135) = { lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐏 ϕ ​ 𝐟 ​ ( γ ∗ ) ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } \displaystyle=\left\{\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{P}_{\phi}\mathbf{f}(\gamma^{*})\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\} (136) = { lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 ​ ( γ ∗ ) ∣ 𝐟 ∈ F sub ′ } \displaystyle=\left\{\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}(\gamma^{*})\mid\mathbf{f}\in F_{\text{sub}}^{\prime}\right\} (137) ⊆ { lim γ ∗ → γ ( 1 − γ ∗ ) ​ 𝐟 ​ ( γ ∗ ) ∣ 𝐟 ∈ F Δ 2 } \displaystyle\subseteq\left\{\lim_{\gamma^{*}\to\gamma}(1-\gamma^{*})\mathbf{f}(\gamma^{*})\mid\mathbf{f}\in F_{\Delta_{2}}\right\} (138) ≕ Norm ​ ( F Δ 2 , γ ) . \displaystyle\eqqcolon\text{{Norm}}\left(F_{\Delta_{2}},\gamma\right). (139) Equation 134 follows by Lemma E.42 . Equation 136 follows because 𝐏 ϕ \mathbf{P}_{\phi} is a continuous linear operator. Equation 138 follows by assumption. 𝔼 s 1 ∼ Δ 1 [ Power 𝒟 bound ​ ( s 1 , γ ) ] \displaystyle\opE_{s_{1}\sim\Delta_{1}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{1},\gamma\right)\right] ≔ 𝔼 s 1 ∼ Δ 1 , 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ π ∈ Π ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 π , s 1 ​ ( γ ∗ ) − 𝐞 s 1 ) ⊤ ​ 𝐫 ] \displaystyle\coloneqq\opE_{\begin{subarray}{c}s_{1}\sim\Delta_{1},\\ \mathbf{r}\sim\mathcal{D}_{\text{bound}}\end{subarray}}\left[\max_{\pi\in\Pi}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}^{\pi,s_{1}}(\gamma^{*})-\mathbf{e}_{s_{1}}\right)^{\top}\mathbf{r}\right] (140) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ π ∈ Π ⁡ 1 − γ ∗ γ ∗ ​ 𝔼 s 1 ∼ Δ 1 [ 𝐟 π , s 1 ​ ( γ ∗ ) − 𝐞 s 1 ] ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\opE_{s_{1}\sim\Delta_{1}}\left[\mathbf{f}^{\pi,s_{1}}(\gamma^{*})-\mathbf{e}_{s_{1}}\right]^{\top}\mathbf{r}\right] (141) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ Norm ​ ( F Δ 1 , γ ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{Norm}}\left(F_{\Delta_{1}},\gamma\right)}\mathbf{d}^{\top}\mathbf{r}\right] (142) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ ND ​ ( Norm ​ ( F Δ 1 , γ ) ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{ND}}\left(\text{{Norm}}\left(F_{\Delta_{1}},\gamma\right)\right)}\mathbf{d}^{\top}\mathbf{r}\right] (143) ≤ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ Norm ​ ( F Δ 2 , γ ) 𝐝 ⊤ 𝐫 ] \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{Norm}}\left(F_{\Delta_{2}},\gamma\right)}\mathbf{d}^{\top}\mathbf{r}\right] (144) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ π ∈ Π ⁡ 1 − γ ∗ γ ∗ ​ 𝔼 s 2 ∼ Δ 2 [ 𝐟 π , s 2 ​ ( γ ∗ ) − 𝐞 s 2 ] ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\pi\in\Pi}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\opE_{s_{2}\sim\Delta_{2}}\left[\mathbf{f}^{\pi,s_{2}}(\gamma^{*})-\mathbf{e}_{s_{2}}\right]^{\top}\mathbf{r}\right] (145) = 𝔼 s 2 ∼ Δ 2 , 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ ∗ → γ π ∈ Π ⁡ 1 − γ ∗ γ ∗ ​ ( 𝐟 π , s 2 ​ ( γ ∗ ) − 𝐞 s 2 ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\begin{subarray}{c}s_{2}\sim\Delta_{2},\\ \mathbf{r}\sim\mathcal{D}_{\text{bound}}\end{subarray}}\left[\max_{\pi\in\Pi}\lim_{\gamma^{*}\to\gamma}\frac{1-\gamma^{*}}{\gamma^{*}}\left(\mathbf{f}^{\pi,s_{2}}(\gamma^{*})-\mathbf{e}_{s_{2}}\right)^{\top}\mathbf{r}\right] (146) ≕ 𝔼 s 2 ∼ Δ 2 [ Power 𝒟 bound ​ ( s 2 , γ ) ] . \displaystyle\eqqcolon\opE_{s_{2}\sim\Delta_{2}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},\gamma\right)\right]. (147)

Equation 140 and eq. 147 follow by Lemma E.43 . Equation 141 and eq. 146 follow because each R R has a stationary deterministic optimal policy π ∈ Π ∗ ​ ( R , γ ) ⊆ Π \pi\in\Pi^{*}\left(R,\gamma\right)\subseteq\Pi which simultaneously achieves optimal value at all states. Equation 143 follows by Corollary E.11 .

Apply Lemma E.24 with A ≔ Norm ​ ( F Δ 1 , γ ) , B ≔ Norm ​ ( F Δ 2 , γ ) A\coloneqq\text{{Norm}}\left(F_{\Delta_{1}},\gamma\right),B\coloneqq\text{{Norm}}\left(F_{\Delta_{2}},\gamma\right) , g g the identity function, and involution ϕ \phi (satisfying ϕ ⋅ ND ​ ( A ) ⊆ B \phi\cdot\text{{ND}}\left(A\right)\subseteq B by eq. 139 ) in order to conclude that eq. 144 holds.

Suppose that ND ​ ( F Δ 2 ) ∖ ϕ ⋅ ND ​ ( F Δ 1 ) \text{{ND}}\left(F_{\Delta_{2}}\right)\setminus\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right) is non-empty; let F sub ′ ≔ ϕ ⋅ ND ​ ( F Δ 1 ) F_{\text{sub}}^{\prime}\coloneqq\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right) . Lemma E.31 shows that for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , ND ​ ( F Δ 2 ​ ( γ ) ) ∖ F sub ′ ​ ( γ ) \text{{ND}}\left(F_{\Delta_{2}}(\gamma)\right)\setminus F_{\text{sub}}^{\prime}(\gamma) is non-empty. Lemma E.15 item 1 then implies that ND ​ ( B ) ∖ ϕ ⋅ A = 1 − γ γ ​ ( ND ​ ( F Δ 2 ​ ( γ ) ) − 𝐞 s ) ∖ ( 1 − γ γ ​ F sub ′ ​ ( γ ) ) \text{{ND}}\left(B\right)\setminus\phi\cdot A=\frac{1-\gamma}{\gamma}\left(\text{{ND}}\left(F_{\Delta_{2}}(\gamma)\right)-\mathbf{e}_{s}\right)\setminus\left(\frac{1-\gamma}{\gamma}F^{\prime}_{\text{sub}}(\gamma)\right) is non-empty. Then Lemma E.24 implies that for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , eq. 144 is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} and 𝔼 s 1 ∼ Δ 1 [ Power 𝒟 bound ( s 1 , γ ) ] ≱ most : 𝔇 bound 𝔼 s 2 ∼ Δ 2 [ Power 𝒟 bound ( s 2 , γ ) ] \opE_{s_{1}\sim\Delta_{1}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{1},\gamma\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{2}\sim\Delta_{2}}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{2},\gamma\right)\right] .

We show that this result’s preconditions holding for F Δ i ∗ F_{\Delta_{i}}^{*} implies the F Δ i F_{\Delta_{i}} preconditions. Suppose F Δ i ∗ ≔ { 𝔼 s i ∼ Δ i [ 𝐟 π , s i ] ∣ π ∈ Π } F_{\Delta_{i}}^{*}\coloneqq\left\{\opE_{s_{i}\sim\Delta_{i}}\left[\mathbf{f}^{\pi,s_{i}}\right]\mid\pi\in\Pi\right\} for i = 1 , 2 i=1,2 are such that F sub ∗ ≔ ϕ ⋅ ND ​ ( F Δ 1 ∗ ) ⊆ F Δ 2 ∗ F_{\text{sub}}^{*}\coloneqq\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)\subseteq F_{\Delta_{2}}^{*} . In the following, the Δ i \Delta_{i} are represented as vectors in ℝ | 𝒮 | \mathbb{R}^{\left|\mathcal{S}\right|} , and γ \gamma is a variable. ϕ ⋅ { γ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } \displaystyle\phi\cdot\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\} = ϕ ⋅ ( ND ​ ( F Δ 1 ∗ − Δ 1 ) ) \displaystyle=\phi\cdot\left(\text{{ND}}\left(F_{\Delta_{1}}^{*}-\Delta_{1}\right)\right) (148) = ϕ ⋅ ( ND ​ ( F Δ 1 ∗ ) − Δ 1 ) \displaystyle=\phi\cdot\left(\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)-\Delta_{1}\right) (149) = { 𝐏 ϕ ​ 𝐟 − 𝐏 ϕ ​ Δ 1 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ∗ ) } \displaystyle=\left\{\mathbf{P}_{\phi}\mathbf{f}-\mathbf{P}_{\phi}\Delta_{1}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)\right\} (150) ⊆ { 𝐟 − Δ 2 ∣ 𝐟 ∈ F Δ 2 ∗ } \displaystyle\subseteq\left\{\mathbf{f}-\Delta_{2}\mid\mathbf{f}\in F_{\Delta_{2}}^{*}\right\} (151) = { γ ​ 𝐟 ∣ 𝐟 ∈ F Δ 2 } . \displaystyle=\left\{\gamma\mathbf{f}\mid\mathbf{f}\in F_{\Delta_{2}}\right\}. (152)

Equation 149 follows from Lemma E.15 item 2 . Since we assumed that ϕ ⋅ ND ​ ( F Δ 1 ∗ ) ⊆ F Δ 2 ∗ \phi\cdot\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)\subseteq F_{\Delta_{2}}^{*} , ϕ ⋅ { Δ 1 } = ϕ ⋅ ( ND ​ ( F Δ 1 ∗ ) ​ ( 0 ) ) ⊆ F Δ 2 ∗ ​ ( 0 ) = { Δ 2 } \phi\cdot\left\{\Delta_{1}\right\}=\phi\cdot\left(\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)(0)\right)\subseteq F_{\Delta_{2}}^{*}(0)=\left\{\Delta_{2}\right\} . This implies that 𝐏 ϕ ​ Δ 1 = Δ 2 \mathbf{P}_{\phi}\Delta_{1}=\Delta_{2} and so eq. 151 follows.

Equation 152 shows that ϕ ⋅ { γ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } ⊆ { γ ​ 𝐟 ∣ 𝐟 ∈ F Δ 2 } \phi\cdot\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\}\subseteq\left\{\gamma\mathbf{f}\mid\mathbf{f}\in F_{\Delta_{2}}\right\} . But we then have ϕ ⋅ { γ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } ≔ { γ ​ 𝐏 ϕ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } = { γ ​ 𝐟 ∣ 𝐟 ∈ ϕ ⋅ ND ​ ( F Δ 1 ) } ⊆ { γ ​ 𝐟 ∣ 𝐟 ∈ F Δ 2 } \phi\cdot\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\}\coloneqq\left\{\gamma\mathbf{P}_{\phi}\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\}=\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right)\right\}\subseteq\left\{\gamma\mathbf{f}\mid\mathbf{f}\in F_{\Delta_{2}}\right\} . Thus, ϕ ⋅ ND ​ ( F Δ 1 ) ⊆ F Δ 2 \phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right)\subseteq F_{\Delta_{2}} .

Suppose ND ​ ( F Δ 2 ∗ ) ∖ ϕ ⋅ ND ​ ( F Δ 1 ∗ ) \text{{ND}}\left(F_{\Delta_{2}}^{*}\right)\setminus\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}^{*}\right) is non-empty, which implies that ϕ ⋅ { γ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ) } \displaystyle\phi\cdot\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}\right)\right\} = { 𝐏 ϕ ​ 𝐟 − 𝐏 ϕ ​ Δ 1 ∣ 𝐟 ∈ ND ​ ( F Δ 1 ∗ ) } \displaystyle=\left\{\mathbf{P}_{\phi}\mathbf{f}-\mathbf{P}_{\phi}\Delta_{1}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)\right\} (153) = { 𝐟 − 𝐏 ϕ ​ Δ 1 ∣ 𝐟 ∈ ϕ ⋅ ND ​ ( F Δ 1 ∗ ) } \displaystyle=\left\{\mathbf{f}-\mathbf{P}_{\phi}\Delta_{1}\mid\mathbf{f}\in\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}^{*}\right)\right\} (154) ⊊ { 𝐟 − Δ 2 ∣ 𝐟 ∈ ND ​ ( F Δ 2 ∗ ) } \displaystyle\subsetneq\left\{\mathbf{f}-\Delta_{2}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{2}}^{*}\right)\right\} (155) = { γ ​ 𝐟 ∣ 𝐟 ∈ ND ​ ( F Δ 2 ) } . \displaystyle=\left\{\gamma\mathbf{f}\mid\mathbf{f}\in\text{{ND}}\left(F_{\Delta_{2}}\right)\right\}. (156)

Then ND ​ ( F Δ 2 ) ∖ ϕ ⋅ ND ​ ( F Δ 1 ) \text{{ND}}\left(F_{\Delta_{2}}\right)\setminus\phi\cdot\text{{ND}}\left(F_{\Delta_{1}}\right) must be non-empty. Therefore, if the preconditions of this result are met for F Δ i ∗ F_{\Delta_{i}}^{*} , they are met for F Δ i F_{\Delta_{i}} . ∎

See 6.6

###### Proof.

Let F sub ≔ ϕ ⋅ ℱ nd ⁡ ( s ′ ) ⊆ ℱ ⁡ ( s ) F_{\text{sub}}\coloneqq\phi\cdot\Fndop(s^{\prime})\subseteq\Fop(s) . Let Δ 1 ≔ 𝐞 s ′ , Δ 2 ≔ 𝐞 s \Delta_{1}\coloneqq\mathbf{e}_{s^{\prime}},\Delta_{2}\coloneqq\mathbf{e}_{s} , and define F Δ i ∗ ≔ { 𝔼 s i ∼ Δ i [ 𝐟 π , s i ] ∣ π ∈ Π } F_{\Delta_{i}}^{*}\coloneqq\left\{\opE_{s_{i}\sim\Delta_{i}}\left[\mathbf{f}^{\pi,s_{i}}\right]\mid\pi\in\Pi\right\} for i = 1 , 2 i=1,2 . Then ℱ nd ⁡ ( s ′ ) = ND ​ ( F Δ 1 ∗ ) \Fndop(s^{\prime})=\text{{ND}}\left(F_{\Delta_{1}}^{*}\right) is similar to F sub = F sub ∗ ⊆ F Δ 2 ∗ = ℱ ⁡ ( s ) F_{\text{sub}}=F^{*}_{\text{sub}}\subseteq F_{\Delta_{2}}^{*}=\Fop(s) via involution ϕ \phi . Apply Lemma E.44 to conclude that ∀ γ ∈ [ 0 , 1 ] : Power 𝒟 bound ( s ′ , γ ) ≤ most : 𝔇 bound Power 𝒟 bound ( s , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) .

Furthermore, ℱ nd ⁡ ( s ) = ND ​ ( F Δ 2 ∗ ) \Fndop(s)=\text{{ND}}\left(F_{\Delta_{2}}^{*}\right) , and F sub = F sub ∗ F_{\text{sub}}=F_{\text{sub}}^{*} , and so if ℱ nd ⁡ ( s ) ∖ ϕ ⋅ ℱ nd ⁡ ( s ′ ) ≔ ℱ nd ⁡ ( s ) ∖ F sub = ND ​ ( F Δ 2 ∗ ) ∖ F sub ∗ \Fndop(s)\setminus\phi\cdot\Fndop(s^{\prime})\coloneqq\Fndop(s)\setminus F_{\text{sub}}=\text{{ND}}\left(F_{\Delta_{2}}^{*}\right)\setminus F_{\text{sub}}^{*} is non-empty, then Lemma E.44 shows that for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} and Power 𝒟 bound ( s ′ , γ ) ≱ most : 𝔇 bound Power 𝒟 bound ( s , γ ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},\gamma\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,\gamma\right) . ∎

###### Lemma E.45 (Non-dominated visit distribution functions never agree with other visit distribution functions at that state) .

Let 𝐟 ∈ ℱ nd ⁡ ( s ) , 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } \mathbf{f}\in\Fndop(s),\mathbf{f}^{\prime}\in\Fop(s)\setminus\{\mathbf{f}\} . ∀ γ ∈ ( 0 , 1 ) : 𝐟 ⁡ ( γ ) ≠ 𝐟 ′ ​ ( γ ) \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{f}(\gamma)\neq\mathbf{f}^{\prime}(\gamma) .

###### Proof.

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . Since 𝐟 ∈ ℱ nd ⁡ ( s ) \mathbf{f}\in\Fndop(s) , there exists a γ ∗ ∈ ( 0 , 1 ) \gamma^{*}\in(0,1) at which 𝐟 \mathbf{f} is strictly optimal for some reward function. Then by section D.1.2 , we can produce another reward function for which 𝐟 \mathbf{f} is strictly optimal at discount rate γ \gamma ; in particular, section D.1.2 guarantees that the policies which induce 𝐟 ′ \mathbf{f}^{\prime} are not optimal at γ \gamma . So 𝐟 ⁡ ( γ ) ≠ 𝐟 ′ ​ ( γ ) \mathbf{f}(\gamma)\neq\mathbf{f}^{\prime}(\gamma) . ∎

###### Corollary E.46 (Cardinality of non-dominated visit distributions) .

Let F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) . ∀ γ ∈ ( 0 , 1 ) : | F ∩ ℱ nd ⁡ ( s ) | = | F ⁡ ( γ ) ∩ ℱ nd ⁡ ( s , γ ) | \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\left|F\cap\Fndop(s)\right|=\left|F(\gamma)\cap\Fndop(s,\gamma)\right| .

###### Proof.

Let γ ∈ ( 0 , 1 ) \gamma\in(0,1) . By applying Lemma E.31 with Δ d ≔ 𝐞 s \Delta_{d}\coloneqq\mathbf{e}_{s} , 𝐟 ∈ ℱ nd ⁡ ( s ) = ND ​ ( ℱ ⁡ ( s ) ) \mathbf{f}\in\Fndop(s)=\text{{ND}}\left(\Fop(s)\right) iff 𝐟 ⁡ ( γ ) ∈ ND ​ ( ℱ ⁡ ( s , γ ) ) \mathbf{f}(\gamma)\in\text{{ND}}\left(\Fop(s,\gamma)\right) . By Lemma E.32 , ND ​ ( ℱ ⁡ ( s , γ ) ) = ℱ nd ⁡ ( s , γ ) \text{{ND}}\left(\Fop(s,\gamma)\right)=\Fndop(s,\gamma) . So all 𝐟 ∈ F ∩ ℱ nd ⁡ ( s ) \mathbf{f}\in F\cap\Fndop(s) induce 𝐟 ⁡ ( γ ) ∈ F ⁡ ( γ ) ∩ ℱ nd ⁡ ( s , γ ) \mathbf{f}(\gamma)\in F(\gamma)\cap\Fndop(s,\gamma) , and | F ∩ ℱ nd ⁡ ( s ) | ≥ | F ⁡ ( γ ) ∩ ℱ nd ⁡ ( s , γ ) | \left|F\cap\Fndop(s)\right|\geq\left|F(\gamma)\cap\Fndop(s,\gamma)\right| .

Lemma E.45 implies that for all 𝐟 , 𝐟 ′ ∈ ℱ nd ⁡ ( s ) \mathbf{f},\mathbf{f}^{\prime}\in\Fndop(s) , 𝐟 = 𝐟 ′ \mathbf{f}=\mathbf{f}^{\prime} iff 𝐟 ​ ( γ ) = 𝐟 ′ ​ ( γ ) \mathbf{f}(\gamma)=\mathbf{f}^{\prime}(\gamma) . Therefore, | F ∩ ℱ nd ⁡ ( s ) | ≤ | F ⁡ ( γ ) ∩ ℱ nd ⁡ ( s , γ ) | \left|F\cap\Fndop(s)\right|\leq\left|F(\gamma)\cap\Fndop(s,\gamma)\right| . So | F ∩ ℱ nd ⁡ ( s ) | = | F ⁡ ( γ ) ∩ ℱ nd ⁡ ( s , γ ) | \left|F\cap\Fndop(s)\right|=\left|F(\gamma)\cap\Fndop(s,\gamma)\right| . ∎

###### Lemma E.47 (Optimality probability and state bottlenecks) .

Suppose that s s can reach Reach ​ ( s ′ , a ′ ) ∪ Reach ​ ( s ′ , a ) \text{{Reach}}\left(s^{\prime},a^{\prime}\right)\cup\text{{Reach}}\left(s^{\prime},a\right) , but only by taking actions equivalent to a ′ a^{\prime} or a a at state s ′ s^{\prime} . F nd , a ′ ≔ ℱ nd ⁡ ( s ∣ π ⁡ ( s ′ ) = a ′ ) , F a ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ′ ) = a ) F_{\text{nd},a^{\prime}}\coloneqq\Fndop(s\mid\pi(s^{\prime})=a^{\prime}),F_{a}\coloneqq\Fop(s\mid\pi(s^{\prime})=a) . Suppose F a F_{a} contains a copy of F nd , a ′ F_{\text{nd},a^{\prime}} via ϕ \phi which fixes all states not belonging to Reach ​ ( s ′ , a ′ ) ∪ Reach ​ ( s ′ , a ) \text{{Reach}}\left(s^{\prime},a^{\prime}\right)\cup\text{{Reach}}\left(s^{\prime},a\right) . Then ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( F nd , a ′ , γ ) ≤ most : 𝔇 any ℙ 𝒟 any ( F a , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) .

If ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F nd , a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{\text{nd},a^{\prime}}\right) is non-empty, then for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and ℙ 𝒟 any ( F nd , a ′ , γ ) ≱ most : 𝔇 any ℙ 𝒟 any ( F a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) .

###### Proof.

Let F sub ≔ ϕ ⋅ F nd , a ′ F_{\text{sub}}\coloneqq\phi\cdot F_{\text{nd},a^{\prime}} . Let F ∗ ≔ ⋃ a ′′ ∈ 𝒜 : ( a ′′ ≢ s ′ a ) ∧ ( a ′′ ≢ s ′ a ′ ) ℱ ( s ∣ π ( s ′ ) = a ′′ ) ∪ F nd , a ′ ∪ F sub F^{*}\coloneqq\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}\\ \left(a^{\prime\prime}\not\equiv_{s^{\prime}}a\right)\land\left(a^{\prime\prime}\not\equiv_{s^{\prime}}a^{\prime}\right)\end{subarray}}\Fop(s\mid\pi(s^{\prime})=a^{\prime\prime})\cup F_{\text{nd},a^{\prime}}\cup F_{\text{sub}} . ϕ ⋅ F ∗ ≔ \displaystyle\phi\cdot F^{*}\coloneqq\, ϕ ⋅ ( ⋃ a ′′ ∈ 𝒜 : ( a ′′ ≢ s ′ a ) ∧ ( a ′′ ≢ s ′ a ′ ) ℱ ( s ∣ π ( s ′ ) = a ′′ ) ∪ F nd , a ′ ∪ F sub ) \displaystyle\phi\cdot\left(\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}\\ \left(a^{\prime\prime}\not\equiv_{s^{\prime}}a\right)\land\left(a^{\prime\prime}\not\equiv_{s^{\prime}}a^{\prime}\right)\end{subarray}}\Fop(s\mid\pi(s^{\prime})=a^{\prime\prime})\cup F_{\text{nd},a^{\prime}}\cup F_{\text{sub}}\right) (157) = \displaystyle=\, ⋃ a ′′ ∈ 𝒜 : ( a ′′ ≢ s ′ a ) ∧ ( a ′′ ≢ s ′ a ′ ) ϕ ⋅ ℱ ( s ∣ π ( s ′ ) = a ′′ ) ∪ ( ϕ ⋅ F nd , a ′ ) ∪ ( ϕ ⋅ F sub ) \displaystyle\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}\\ \left(a^{\prime\prime}\not\equiv_{s^{\prime}}a\right)\land\left(a^{\prime\prime}\not\equiv_{s^{\prime}}a^{\prime}\right)\end{subarray}}\phi\cdot\Fop(s\mid\pi(s^{\prime})=a^{\prime\prime})\cup\left(\phi\cdot F_{\text{nd},a^{\prime}}\right)\cup\left(\phi\cdot F_{\text{sub}}\right) (158) = \displaystyle=\, ⋃ a ′′ ∈ 𝒜 : ( a ′′ ≢ s ′ a ) ∧ ( a ′′ ≢ s ′ a ′ ) ϕ ⋅ ℱ ( s ∣ π ( s ′ ) = a ′′ ) ∪ F sub ∪ F nd , a ′ \displaystyle\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}\\ \left(a^{\prime\prime}\not\equiv_{s^{\prime}}a\right)\land\left(a^{\prime\prime}\not\equiv_{s^{\prime}}a^{\prime}\right)\end{subarray}}\phi\cdot\Fop(s\mid\pi(s^{\prime})=a^{\prime\prime})\cup F_{\text{sub}}\cup F_{\text{nd},a^{\prime}} (159) = \displaystyle=\, ⋃ a ′′ ∈ 𝒜 : ( a ′′ ≢ s ′ a ) ∧ ( a ′′ ≢ s ′ a ′ ) ℱ ( s ∣ π ( s ′ ) = a ′′ ) ∪ F sub ∪ F nd , a ′ \displaystyle\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}\\ \left(a^{\prime\prime}\not\equiv_{s^{\prime}}a\right)\land\left(a^{\prime\prime}\not\equiv_{s^{\prime}}a^{\prime}\right)\end{subarray}}\Fop(s\mid\pi(s^{\prime})=a^{\prime\prime})\cup F_{\text{sub}}\cup F_{\text{nd},a^{\prime}} (160) ≕ \displaystyle\eqqcolon\, F ∗ . \displaystyle F^{*}. (161) Equation 159 follows because the involution ϕ \phi ensures that ϕ ⋅ F sub = F nd , a ′ \phi\cdot F_{\text{sub}}=F_{\text{nd},a^{\prime}} . By assumption, ϕ \phi fixes all s ′ ∉ Reach ​ ( s ′ , a ′ ) ∪ Reach ​ ( s ′ , a ) s^{\prime}\not\in\text{{Reach}}\left(s^{\prime},a^{\prime}\right)\cup\text{{Reach}}\left(s^{\prime},a\right) . Suppose 𝐟 ∈ ℱ ⁡ ( s ) ∖ ( F nd , a ′ ∪ F a ) \mathbf{f}\in\Fop(s)\setminus\left(F_{\text{nd},a^{\prime}}\cup F_{a}\right) . By the bottleneck assumption, 𝐟 \mathbf{f} does not visit states in Reach ​ ( s ′ , a ′ ) ∪ Reach ​ ( s ′ , a ) \text{{Reach}}\left(s^{\prime},a^{\prime}\right)\cup\text{{Reach}}\left(s^{\prime},a\right) . Therefore, 𝐏 ϕ ​ 𝐟 = 𝐟 \mathbf{P}_{\phi}\mathbf{f}=\mathbf{f} , and so eq. 160 follows.

Let F Z ≔ ( ℱ ⁡ ( s ) ∖ ( ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ′ ) ∪ F a ) ) ∪ F nd , a ′ ∪ F a F_{Z}\coloneqq\left(\Fop(s)\setminus(\Fop(s\mid\pi(s)=a^{\prime})\cup F_{a})\right)\cup F_{\text{nd},a^{\prime}}\cup F_{a} . By definition, F Z ⊆ ℱ ⁡ ( s ) F_{Z}\subseteq\Fop(s) . Furthermore, ℱ nd ⁡ ( s ) = ⋃ a ′′ ∈ 𝒜 ℱ nd ⁡ ( s ∣ π ⁡ ( s ′ ) = a ′′ ) ⊆ ( ℱ ⁡ ( s ) ∖ ( ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ′ ) ∪ F a ) ) ∪ ℱ nd ⁡ ( s ∣ π ⁡ ( s ) = a ′ ) ∪ F a ≕ F Z \Fndop(s)=\bigcup_{\begin{subarray}{c}a^{\prime\prime}\in\mathcal{A}\end{subarray}}\Fndop(s\mid\pi(s^{\prime})=a^{\prime\prime})\subseteq\left(\Fop(s)\setminus(\Fop(s\mid\pi(s)=a^{\prime})\cup F_{a})\right)\cup\Fndop(s\mid\pi(s)=a^{\prime})\cup F_{a}\eqqcolon F_{Z} , and so ℱ nd ⁡ ( s ) ⊆ F Z \Fndop(s)\subseteq F_{Z} . Note that F ∗ = F Z ∖ ( F a ∖ F sub ) F^{*}=F_{Z}\setminus(F_{a}\setminus F_{\text{sub}}) .

##### Case: γ ∈ ( 0 , 1 ) \gamma\in(0,1) .

ℙ 𝒟 any ( F nd , a ′ , γ ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma\right) = p 𝒟 any ​ ( F nd , a ′ ​ ( γ ) ≥ ℱ ⁡ ( s , γ ) ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}}(\gamma)\geq\Fop(s,\gamma)\right) (162) ≤ most : 𝔇 any p 𝒟 any ( F a ( γ ) ≥ ℱ ( s , γ ) ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(F_{a}(\gamma)\geq\Fop(s,\gamma)\right) (163) = ℙ 𝒟 any ( F nd , a ′ , γ ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma\right). (164)

Equation 162 and eq. 164 follow from Lemma E.35 . Equation 163 follows by applying Lemma E.28 with A ≔ F nd , a ′ ​ ( γ ) , B ′ ≔ F sub ​ ( γ ) , B ≔ F a ​ ( γ ) , C ≔ ℱ ⁡ ( s , γ ) , Z ≔ F Z ​ ( γ ) A\coloneqq F_{\text{nd},a^{\prime}}(\gamma),B^{\prime}\coloneqq F_{\text{sub}}(\gamma),B\coloneqq F_{a}(\gamma),C\coloneqq\Fop(s,\gamma),Z\coloneqq F_{Z}(\gamma) which satisfies ND ​ ( C ) = ℱ nd ⁡ ( s , γ ) ⊆ F Z ​ ( γ ) ⊆ ℱ ⁡ ( s , γ ) = C \text{{ND}}\left(C\right)=\Fndop(s,\gamma)\subseteq F_{Z}(\gamma)\subseteq\Fop(s,\gamma)=C , and involution ϕ \phi which satisfies ϕ ⋅ F ∗ ​ ( γ ) = ϕ ⋅ ( Z ∖ ( B ∖ B ′ ) ) = Z ∖ ( B ∖ B ′ ) = F ∗ ​ ( γ ) \phi\cdot F^{*}(\gamma)=\phi\cdot\left(Z\setminus\left(B\setminus B^{\prime}\right)\right)=Z\setminus\left(B\setminus B^{\prime}\right)=F^{*}(\gamma) .

Suppose ℱ nd ⁡ ( s ) ∩ ( F a ∖ F sub ) \Fndop(s)\cap\left(F_{a}\setminus F_{\text{sub}}\right) is non-empty. 0 < | ℱ nd ⁡ ( s ) ∩ ( F a ∖ F sub ) | = | ℱ nd ⁡ ( s , γ ) ∩ ( F a ​ ( γ ) ∖ F sub ​ ( γ ) ) | ≕ | ND ​ ( C ) ∩ ( B ∖ B ′ ) | 0<\left|\Fndop(s)\cap\left(F_{a}\setminus F_{\text{sub}}\right)\right|=\left|\Fndop(s,\gamma)\cap\left(F_{a}(\gamma)\setminus F_{\text{sub}}(\gamma)\right)\right|\eqqcolon\left|\text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right)\right| (with the first equality holding by Corollary E.46 ), and so ND ​ ( C ) ∩ ( B ∖ B ′ ) \text{{ND}}\left(C\right)\cap\left(B\setminus B^{\prime}\right) is non-empty. We also have B ≔ F a ​ ( γ ) ⊆ ℱ ⁡ ( s , γ ) ≕ C B\coloneqq F_{a}(\gamma)\subseteq\Fop(s,\gamma)\eqqcolon C . Then reapplying Lemma E.28 , eq. 163 is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and ℙ 𝒟 any ( F nd , a ′ , γ ) ≱ most : 𝔇 any ℙ 𝒟 any ( F a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) .

##### Case: γ = 1 \gamma=1 , γ = 0 \gamma=0 .

ℙ 𝒟 any ( F nd , a ′ , 1 ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},1\right) = lim γ ∗ → 1 ℙ 𝒟 any ( F nd , a ′ , γ ∗ ) \displaystyle=\lim_{\gamma^{*}\to 1}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}},\gamma^{*}\right) (165) = lim γ ∗ → 1 p 𝒟 any ​ ( F nd , a ′ ​ ( γ ∗ ) ≥ ℱ ⁡ ( s , γ ∗ ) ) \displaystyle=\lim_{\gamma^{*}\to 1}p_{\mathcal{D}_{\text{any}}}\left(F_{\text{nd},a^{\prime}}(\gamma^{*})\geq\Fop(s,\gamma^{*})\right) (166) ≤ most : 𝔇 any lim γ ∗ → 1 p 𝒟 any ( F a ( γ ∗ ) ≥ ℱ ( s , γ ∗ ) ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\lim_{\gamma^{*}\to 1}p_{\mathcal{D}_{\text{any}}}\left(F_{a}(\gamma^{*})\geq\Fop(s,\gamma^{*})\right) (167) = lim γ ∗ → 1 ℙ 𝒟 any ( F a , γ ∗ ) \displaystyle=\lim_{\gamma^{*}\to 1}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma^{*}\right) (168) = ℙ 𝒟 any ( F a , 1 ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},1\right). (169) Equation 165 and eq. 169 hold by Proposition E.34 . Equation 166 and eq. 168 follow by Lemma E.35 . Applying Lemma E.29 with γ ≔ 1 , I ≔ ( 0 , 1 ) , F A ≔ F nd , a ′ , F B ≔ F a , F C ≔ ℱ ⁡ ( s ) \gamma\coloneqq 1,I\coloneqq(0,1),F_{A}\coloneqq F_{\text{nd},a^{\prime}},F_{B}\coloneqq F_{a},F_{C}\coloneqq\Fop(s) , F Z F_{Z} as defined above, and involution ϕ \phi (for which ϕ ⋅ ( F Z ∖ ( F B ∖ ϕ ⋅ F A ) ) = F Z ∖ ( F B ∖ ϕ ⋅ F A ) \phi\cdot\left(F_{Z}\setminus\left(F_{B}\setminus\phi\cdot F_{A}\right)\right)=F_{Z}\setminus\left(F_{B}\setminus\phi\cdot F_{A}\right) ), we conclude that eq. 167 follows.

The γ = 0 \gamma=0 case proceeds similarly to γ = 1 \gamma=1 . ∎

###### Lemma E.48 (Action optimality probability is a special case of visit distribution optimality probability) .

ℙ 𝒟 any ( s , a , γ ) = ℙ 𝒟 any ( ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ) , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\Fop(s\mid\pi(s)=a),\gamma\right) .

###### Proof.

Let F a ≔ ℱ ⁡ ( s ∣ π ⁡ ( s ) = a ) F_{a}\coloneqq\Fop(s\mid\pi(s)=a) . For γ ∈ ( 0 , 1 ) \gamma\in(0,1) , ℙ 𝒟 any ( s , a , γ ) \displaystyle\Prb_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right) ≔ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a ) \displaystyle\coloneqq\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) (170) = ℙ 𝐫 ∼ 𝒟 any ( ∃ 𝐟 π ∗ , s ∈ F a : 𝐟 π ∗ , s ​ ( γ ) ⊤ ​ 𝐫 = max 𝐟 ∈ ℱ ⁡ ( s ) ⁡ 𝐟 ​ ( γ ) ⊤ ​ 𝐫 ) \displaystyle=\Prb_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{f}^{\pi^{*},s}\in F_{a}\mathrel{\mathop{\ordinarycolon}}\mathbf{f}^{\pi^{*},s}(\gamma)^{\top}\mathbf{r}=\max_{\mathbf{f}\in\Fop(s)}\mathbf{f}(\gamma)^{\top}\mathbf{r}\right) (171) = ℙ 𝒟 any ( F a , γ ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right). (172) By Lemma E.1 , if ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a \exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a , then it induces some optimal 𝐟 π ∗ , s ∈ F a \mathbf{f}^{\pi^{*},s}\in F_{a} . Conversely, if 𝐟 π ∗ , s ∈ F a \mathbf{f}^{\pi^{*},s}\in F_{a} is optimal at γ ∈ ( 0 , 1 ) \gamma\in(0,1) , then π ∗ \pi^{*} chooses optimal actions on the support of 𝐟 π ∗ , s ​ ( γ ) \mathbf{f}^{\pi^{*},s}(\gamma) . Let π ′ \pi^{\prime} agree with π ∗ \pi^{*} on that support and let π ′ \pi^{\prime} take optimal actions at all other states. Then π ′ ∈ Π ∗ ​ ( R , γ ) \pi^{\prime}\in\Pi^{*}\left(R,\gamma\right) and π ′ ​ ( s ) = a \pi^{\prime}(s)=a . So eq. 171 follows.

Suppose γ = 0 \gamma=0 or γ = 1 \gamma=1 . Consider any sequence ( γ n ) n = 1 ∞ \left(\gamma_{n}\right)_{n=1}^{\infty} converging to γ \gamma , and let 𝒟 any \mathcal{D}_{\text{any}} induce probability measure F F . ℙ 𝒟 any ( F a , γ ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) ≔ lim γ ∗ → γ ℙ 𝒟 any ( F a , γ ∗ ) \displaystyle\coloneqq\lim_{\gamma^{*}\to\gamma}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma^{*}\right) (173) = lim γ ∗ → γ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ ∗ ) : π ∗ ​ ( s ) = a ) \displaystyle=\lim_{\gamma^{*}\to\gamma}\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma^{*}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) (174) = lim n → ∞ ℙ R ∼ 𝒟 any ( ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a ) \displaystyle=\lim_{n\to\infty}\Prb_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a\right) (175) = lim n → ∞ ∫ ℝ 𝒮 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\lim_{n\to\infty}\int_{\mathbb{R}^{\mathcal{S}}}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (176) = ∫ ℝ 𝒮 lim n → ∞ 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ n ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\int_{\mathbb{R}^{\mathcal{S}}}\lim_{n\to\infty}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma_{n}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (177) = ∫ ℝ 𝒮 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ ) : π ∗ ​ ( s ) = a d F ( R ) \displaystyle=\int_{\mathbb{R}^{\mathcal{S}}}\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a}\dif F(R) (178) ≕ ℙ 𝒟 any ( s , a , γ ) . \displaystyle\eqqcolon\Prb_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right). (179)

Equation 174 follows by eq. 172 . for γ ∗ ∈ [ 0 , 1 ] \gamma^{*}\in[0,1] , let f γ ∗ ( R ) ≔ 𝟙 ∃ π ∗ ∈ Π ∗ ​ ( R , γ ∗ ) : π ∗ ​ ( s ) = a f_{\gamma^{*}}(R)\coloneqq\mathbbm{1}_{\exists\pi^{*}\in\Pi^{*}\left(R,\gamma^{*}\right)\mathrel{\mathop{\ordinarycolon}}\pi^{*}(s)=a} . For each R ∈ ℝ 𝒮 R\in\mathbb{R}^{\mathcal{S}} , Lemma E.33 exists γ x ≈ γ \gamma_{x}\approx\gamma such that for all intermediate γ x ′ \gamma_{x}^{\prime} between γ x \gamma_{x} and γ \gamma , Π ∗ ​ ( R , γ x ′ ) = Π ∗ ​ ( R , γ ) \Pi^{*}\left(R,\gamma_{x}^{\prime}\right)=\Pi^{*}\left(R,\gamma\right) . Since γ n → γ \gamma_{n}\to\gamma , this means that ( f γ n ) n = 1 ∞ \left(f_{\gamma_{n}}\right)_{n=1}^{\infty} converges pointwise to f γ f_{\gamma} . Furthermore, ∀ n ∈ ℕ , R ∈ ℝ 𝒮 : | f γ n ​ ( R ) | ≤ 1 \forall n\in\mathbb{N},R\in\mathbb{R}^{\mathcal{S}}\mathrel{\mathop{\ordinarycolon}}\left|f_{\gamma_{n}}(R)\right|\leq 1 by definition. Therefore, eq. 177 follows by Lebesgue’s dominated convergence theorem. ∎

See 6.9

###### Proof.

Note that by definition 3.3 , F a ′ ​ ( 0 ) = { 𝐞 s } = F a ​ ( 0 ) F_{a^{\prime}}(0)=\left\{\mathbf{e}_{s}\right\}=F_{a}(0) . Since ϕ ⋅ F a ′ ⊆ F a \phi\cdot F_{a^{\prime}}\subseteq F_{a} , in particular we have ϕ ⋅ F a ′ ​ ( 0 ) = { 𝐏 ϕ ​ 𝐞 s } ⊆ { 𝐞 s } = F a ​ ( 0 ) \phi\cdot F_{a^{\prime}}(0)=\left\{\mathbf{P}_{\phi}\mathbf{e}_{s}\right\}\subseteq\left\{\mathbf{e}_{s}\right\}=F_{a}(0) , and so ϕ ⁡ ( s ) = s \phi(s)=s .

Item 1 . For state probability distribution Δ s ∈ Δ ⁡ ( 𝒮 ) \Delta_{s}\in\Delta(\mathcal{S}) , let F Δ s ∗ ≔ { 𝔼 s ′ ∼ Δ s [ 𝐟 π , s ′ ] ∣ π ∈ Π } F^{*}_{\Delta_{s}}\coloneqq\left\{\opE_{s^{\prime}\sim\Delta_{s}}\left[\mathbf{f}^{\pi,s^{\prime}}\right]\mid\pi\in\Pi\right\} . Unless otherwise stated, we treat γ \gamma as a variable in this item; we apply element-wise vector addition, constant multiplication, and variable multiplication via the conventions outlined in definition E.14 . F a ′ \displaystyle F_{a^{\prime}} = { 𝐞 s + γ ​ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ∣ π ∈ Π : π ⁡ ( s ) = a ′ } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi\mathrel{\mathop{\ordinarycolon}}\pi(s)=a^{\prime}\right\} (180) = { 𝐞 s + γ ​ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ∣ π ∈ Π } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi\right\} (181) = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ . \displaystyle=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})}. (182)

Equation 180 follows by definition 3.3 , since each 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) has an initial term of 𝐞 s \mathbf{e}_{s} . Equation 181 follows because s ∉ Reach ​ ( s , a ′ ) s\not\in\text{{Reach}}\left(s,a^{\prime}\right) , and so for all s a ′ ∈ supp ⁡ ( T ⁡ ( s , a ′ ) ) s_{a^{\prime}}\in\optSupp(T(s,a^{\prime})) , 𝐟 π , s a ′ \mathbf{f}^{\pi,s_{a^{\prime}}} is unaffected by the choice of action π ⁡ ( s ) \pi(s) . Note that similar reasoning implies that F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} (because eq. 181 is a containment relation in general).

Since F a ′ = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ F_{a^{\prime}}=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})} , if F a F_{a} contains a copy of F a ′ F_{a^{\prime}} via ϕ \phi , then F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} contains a copy of F T ⁡ ( s , a ′ ) ∗ F^{*}_{T(s,a^{\prime})} via ϕ \phi . Then ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) ⊆ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ ⊆ F T ⁡ ( s , a ) ∗ \phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\subseteq\phi\cdot F^{*}_{T(s,a^{\prime})}\subseteq F^{*}_{T(s,a)} , and so F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} contains a copy of ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) . Then apply Lemma E.44 with Δ 1 ≔ T ⁡ ( s , a ′ ) \Delta_{1}\coloneqq T(s,a^{\prime}) and Δ 2 ≔ T ⁡ ( s , a ) \Delta_{2}\coloneqq T(s,a) to conclude that ∀ γ ∈ [ 0 , 1 ] : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] ≤ most : 𝔇 bound 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right]\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right] .

Suppose ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty. To apply the second condition of Lemma E.44 , we want to demonstrate that ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is also non-empty.

First consider 𝐟 ∈ ℱ nd ⁡ ( s ) ∩ F a \mathbf{f}\in\Fndop(s)\cap F_{a} . Because F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} , we have that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ F T ⁡ ( s , a ) ∗ \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in F^{*}_{T(s,a)} . Because 𝐟 ∈ ℱ nd ⁡ ( s ) \mathbf{f}\in\Fndop(s) , by definition 3.6 , ∃ 𝐫 ∈ ℝ | 𝒮 | , γ x ∈ ( 0 , 1 ) \exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|},\gamma_{x}\in(0,1) such that 𝐟 ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle\mathbf{f}(\gamma_{x})^{\top}\mathbf{r} > max 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 . \displaystyle>\max_{\mathbf{f}^{\prime}\in\Fop(s)\setminus\left\{\mathbf{f}\right\}}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r}. (183) Then since γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , γ x − 1 ​ ( 𝐟 ⁡ ( γ x ) − 𝐞 s ) ⊤ ​ 𝐫 \displaystyle\gamma_{x}^{-1}(\mathbf{f}(\gamma_{x})-\mathbf{e}_{s})^{\top}\mathbf{r} > max 𝐟 ′ ∈ ℱ ⁡ ( s ) ∖ { 𝐟 } ⁡ γ x − 1 ​ ( 𝐟 ′ ​ ( γ x ) − 𝐞 s ) ⊤ ​ 𝐫 \displaystyle>\max_{\mathbf{f}^{\prime}\in\Fop(s)\setminus\left\{\mathbf{f}\right\}}\gamma_{x}^{-1}(\mathbf{f}^{\prime}(\gamma_{x})-\mathbf{e}_{s})^{\top}\mathbf{r} (184) = max 𝐟 ′ ∈ γ x − 1 ​ ( ( ℱ ⁡ ( s ) ∖ { 𝐟 } ) − 𝐞 s ) ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle=\max_{\mathbf{f}^{\prime}\in\gamma_{x}^{-1}\left((\Fop(s)\setminus\left\{\mathbf{f}\right\})-\mathbf{e}_{s}\right)}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r} (185) ≥ max 𝐟 ′ ∈ γ x − 1 ​ ( ( F a ∖ { 𝐟 } ) − 𝐞 s ) ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 \displaystyle\geq\max_{\mathbf{f}^{\prime}\in\gamma_{x}^{-1}\left((F_{a}\setminus\left\{\mathbf{f}\right\})-\mathbf{e}_{s}\right)}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r} (186) = max 𝐟 ′ ∈ F T ⁡ ( s , a ) ∗ ∖ { γ x − 1 ​ ( 𝐟 − 𝐞 s ) } ⁡ 𝐟 ′ ​ ( γ x ) ⊤ ​ 𝐫 . \displaystyle=\max_{\mathbf{f}^{\prime}\in F^{*}_{T(s,a)}\setminus\left\{\gamma_{x}^{-1}(\mathbf{f}-\mathbf{e}_{s})\right\}}\mathbf{f}^{\prime}(\gamma_{x})^{\top}\mathbf{r}. (187)

Equation 186 holds because F a ⊆ ℱ ⁡ ( s ) F_{a}\subseteq\Fop(s) . By assumption, action a a is optimal for 𝐫 \mathbf{r} at state s s and at discount rate γ x \gamma_{x} . Equation 181 shows that F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} potentially allows the agent a non-stationary policy choice at s s , but non-stationary policies cannot increase optimal value [ Puterman, 2014 ] . Therefore, eq. 187 holds.

We assumed that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in\gamma^{-1}(\Fndop(s)-\mathbf{e}_{s}) . Furthermore, since we just showed that γ − 1 ​ ( 𝐟 − 𝐞 s ) ∈ F T ⁡ ( s , a ) ∗ \gamma^{-1}(\mathbf{f}-\mathbf{e}_{s})\in F^{*}_{T(s,a)} is strictly optimal over the other elements of F T ⁡ ( s , a ) ∗ F^{*}_{T(s,a)} for reward function 𝐫 \mathbf{r} at discount rate γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , we conclude that it is an element of ND ​ ( F T ⁡ ( s , a ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right) by definition E.13 . Then we conclude that γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ⊆ ND ​ ( F T ⁡ ( s , a ) ∗ ) \gamma^{-1}(\Fndop(s)-\mathbf{e}_{s})\cap F^{*}_{T(s,a)}\subseteq\text{{ND}}\left(F^{*}_{T(s,a)}\right) .

We now show that ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is non-empty. 0 \displaystyle 0 < | ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) | \displaystyle<\left|\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right)\right| (188) = | γ − 1 ​ ( ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) − 𝐞 s ) | \displaystyle=\left|\gamma^{-1}\left(\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right)-\mathbf{e}_{s}\right)\right| (189) ≤ | γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ ( F T ⁡ ( s , a ) ∗ ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ ) | \displaystyle\leq\left|\gamma^{-1}\left(\Fndop(s)-\mathbf{e}_{s}\right)\cap\left(F^{*}_{T(s,a)}\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right)\right| (190) = | ( γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ | \displaystyle=\left|\left(\gamma^{-1}\left(\Fndop(s)-\mathbf{e}_{s}\right)\cap F^{*}_{T(s,a)}\right)\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right| (191) ≤ | ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ F T ⁡ ( s , a ′ ) ∗ | \displaystyle\leq\left|\text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot F^{*}_{T(s,a^{\prime})}\right| (192) ≤ | ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) | . \displaystyle\leq\left|\text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\right|. (193) Equation 188 follows by the assumption that ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty. Let 𝐟 , 𝐟 ′ ∈ ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \mathbf{f},\mathbf{f}^{\prime}\in\Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) be distinct. Then we must have that for some γ x ∈ ( 0 , 1 ) \gamma_{x}\in(0,1) , 𝐟 ⁡ ( γ x ) ≠ 𝐟 ′ ​ ( γ x ) \mathbf{f}(\gamma_{x})\neq\mathbf{f}^{\prime}(\gamma_{x}) . This holds iff γ x − 1 ​ ( 𝐟 ⁡ ( γ x ) − 𝐞 s ) ≠ γ x − 1 ​ ( 𝐟 ′ ​ ( γ x ) − 𝐞 s ) \gamma_{x}^{-1}(\mathbf{f}(\gamma_{x})-\mathbf{e}_{s})\neq\gamma_{x}^{-1}(\mathbf{f}^{\prime}(\gamma_{x})-\mathbf{e}_{s}) , and so eq. 189 holds.

Equation 190 holds because F a ⊆ 𝐞 s + γ ​ F T ⁡ ( s , a ) ∗ F_{a}\subseteq\mathbf{e}_{s}+\gamma F^{*}_{T(s,a)} and F a ′ = 𝐞 s + γ ​ F T ⁡ ( s , a ′ ) ∗ F_{a}^{\prime}=\mathbf{e}_{s}+\gamma F^{*}_{T(s,a^{\prime})} by eq. 182 . Equation 192 holds because we showed above that γ − 1 ​ ( ℱ nd ⁡ ( s ) − 𝐞 s ) ∩ F T ⁡ ( s , a ) ∗ ⊆ ND ​ ( F T ⁡ ( s , a ) ∗ ) \gamma^{-1}(\Fndop(s)-\mathbf{e}_{s})\cap F^{*}_{T(s,a)}\subseteq\text{{ND}}\left(F^{*}_{T(s,a)}\right) . Equation 193 holds because ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) ⊆ F T ⁡ ( s , a ′ ) ∗ \text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right)\subseteq F^{*}_{T(s,a^{\prime})} by definition E.13 .

Therefore, ND ​ ( F T ⁡ ( s , a ) ∗ ) ∖ ϕ ⋅ ND ​ ( F T ⁡ ( s , a ′ ) ∗ ) \text{{ND}}\left(F^{*}_{T(s,a)}\right)\setminus\phi\cdot\text{{ND}}\left(F^{*}_{T(s,a^{\prime})}\right) is non-empty, and so apply the second condition of Lemma E.44 to conclude that for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , ∀ γ ∈ ( 0 , 1 ) : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 X ​ - iid ​ ( s a ′ , γ ) ] < 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 X ​ - iid ​ ( s a , γ ) ] \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(s_{a^{\prime}},\gamma\right)\right]<\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{X\text{-}\textsc{iid}}}\left(s_{a},\gamma\right)\right] , and that ∀ γ ∈ ( 0 , 1 ) : 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ Power 𝒟 bound ( s a ′ , γ ) ] ≱ most : 𝔇 bound 𝔼 s a ∼ T ⁡ ( s , a ) [ Power 𝒟 bound ( s a , γ ) ] \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a^{\prime}},\gamma\right)\right]\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{s_{a}\sim T(s,a)}\left[\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s_{a},\gamma\right)\right] .

Item 2 . Let ϕ ′ ​ ( s x ) ≔ ϕ ⁡ ( s x ) \phi^{\prime}(s_{x})\coloneqq\phi(s_{x}) when s x ∈ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s_{x}\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) , and equal s x s_{x} otherwise. Since ϕ \phi is an involution, so is ϕ ′ \phi^{\prime} . ϕ ′ ⋅ F a ′ \displaystyle\phi^{\prime}\cdot F_{a^{\prime}} ≔ { 𝐏 ϕ ′ ( 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐟 π , s a ′ ] ) ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle\coloneqq\left\{\mathbf{P}_{\phi^{\prime}}\left(\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\right)\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (194) = { 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐏 ϕ ′ 𝐟 π , s a ′ ] ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle=\left\{\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{P}_{\phi^{\prime}}\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (195) = { 𝐏 ϕ 𝐞 s + γ 𝔼 s a ′ ∼ T ⁡ ( s , a ′ ) [ 𝐏 ϕ 𝐟 π , s a ′ ] ∣ π ∈ Π , π ( s ) = a ′ } \displaystyle=\left\{\mathbf{P}_{\phi}\mathbf{e}_{s}+\gamma\opE_{s_{a^{\prime}}\sim T(s,a^{\prime})}\left[\mathbf{P}_{\phi}\mathbf{f}^{\pi,s_{a^{\prime}}}\right]\mid\pi\in\Pi,\pi(s)=a^{\prime}\right\} (196) ≕ ϕ ⋅ F a ′ \displaystyle\eqqcolon\phi\cdot F_{a^{\prime}} (197) ⊆ F a . \displaystyle\subseteq F_{a}. (198)

Equation 195 follows because if s ∈ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) , then we already showed that ϕ \phi fixes s s . Otherwise, ϕ ′ ​ ( s ) = s \phi^{\prime}(s)=s by definition. Equation 196 follows by the definition of ϕ ′ \phi^{\prime} on Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) \text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) and because 𝐞 s = 𝐏 ϕ ​ 𝐞 s \mathbf{e}_{s}=\mathbf{P}_{\phi}\mathbf{e}_{s} . Next, we assumed that ϕ ⋅ F a ′ ⊆ F a \phi\cdot F_{a^{\prime}}\subseteq F_{a} , and so eq. 198 holds.

Therefore, F a F_{a} contains a copy of F a ′ F_{a^{\prime}} via ϕ ′ \phi^{\prime} fixing all s x ∉ Reach ​ ( s , a ′ ) ∪ Reach ​ ( s , a ) s_{x}\not\in\text{{Reach}}\left(s,a^{\prime}\right)\cup\text{{Reach}}\left(s,a\right) . Therefore, F a F_{a} contains a copy of F nd , a ′ ≔ ℱ nd ⁡ ( s ) ∩ F a ′ F_{\text{nd},a^{\prime}}\coloneqq\Fndop(s)\cap F_{a^{\prime}} via the same ϕ ′ \phi^{\prime} . Then apply Lemma E.47 with s ′ ≔ s s^{\prime}\coloneqq s to conclude that ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( F a ′ , γ ) ≤ most : 𝔇 any ℙ 𝒟 any ( F a , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a^{\prime}},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) . By Lemma E.48 , ℙ 𝒟 any ( s , a ′ , γ ) = ℙ 𝒟 any ( F a ′ , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a^{\prime}},\gamma\right) and ℙ 𝒟 any ( s , a , γ ) = ℙ 𝒟 any ( F a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right)=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(F_{a},\gamma\right) . Therefore, ∀ γ ∈ [ 0 , 1 ] : ℙ 𝒟 any ( s , a ′ , γ ) ≤ most : 𝔇 any ℙ 𝒟 any ( s , a , γ ) \forall\gamma\in[0,1]\mathrel{\mathop{\ordinarycolon}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right) .

If ℱ nd ⁡ ( s ) ∩ ( F a ∖ ϕ ⋅ F a ′ ) \Fndop(s)\cap\left(F_{a}\setminus\phi\cdot F_{a^{\prime}}\right) is non-empty, then apply the second condition of Lemma E.47 to conclude that for all γ ∈ ( 0 , 1 ) \gamma\in(0,1) , the inequality is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and ℙ 𝒟 any ( s , a ′ , γ ) ≱ most : 𝔇 any ℙ 𝒟 any ( s , a , γ ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a^{\prime},\gamma\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(s,a,\gamma\right) . ∎

#### E.4.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles

###### Lemma E.49 ( Power identity when γ = 1 \gamma=1 ) .

Power 𝒟 bound ​ ( s , 1 ) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] . \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right)=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]. (199)

###### Proof.

Power 𝒟 bound ​ ( s , 1 ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max ⁡ lim γ → 1 𝐟 π , s ∈ ℱ ⁡ ( s ) ⁡ 1 − γ γ ​ ( 𝐟 π , s ​ ( γ ) − 𝐞 s ) ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{f}^{\pi,s}\in\Fop(s)}\lim_{\gamma\to 1}\frac{1-\gamma}{\gamma}\left(\mathbf{f}^{\pi,s}(\gamma)-\mathbf{e}_{s}\right)^{\top}\mathbf{r}\right] (200) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right] (201) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 ] . \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right]. (202) Equation 200 follows by Lemma E.43 . Equation 201 follows by the definition of RSD ​ ( s ) \text{{RSD}}\left(s\right) ( definition 6.10 ). Equation 202 follows because for all 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} , Corollary E.11 shows that max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 = max 𝐝 ∈ ND ​ ( RSD ​ ( s ) ) ⁡ 𝐝 ⊤ ​ 𝐫 ≕ max 𝐝 ∈ RSD nd ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 \max_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}=\max_{\mathbf{d}\in\text{{ND}}\left(\text{{RSD}}\left(s\right)\right)}\mathbf{d}^{\top}\mathbf{r}\eqqcolon\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r} . ∎

See 6.12

###### Proof.

Suppose RSD nd ​ ( s ′ ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right) is similar to D ⊆ RSD ​ ( s ) D\subseteq\text{{RSD}}\left(s\right) via involution ϕ \phi . Power 𝒟 bound ​ ( s ′ , 1 ) \displaystyle\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right) = 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ′ ) ⁡ 𝐝 ⊤ ​ 𝐫 ] \displaystyle=\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s^{\prime}\right)}\mathbf{d}^{\top}\mathbf{r}\right] (203) ≤ most : 𝔇 bound 𝔼 𝐫 ∼ 𝒟 bound [ max 𝐝 ∈ RSD nd ​ ( s ) 𝐝 ⊤ 𝐫 ] \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[\max_{\mathbf{d}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right] (204) = Power 𝒟 bound ​ ( s , 1 ) \displaystyle=\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) (205) Equation 203 and eq. 205 follow from Lemma E.49 . By applying Lemma E.24 with A ≔ RSD ​ ( s ′ ) , B ′ ≔ D , B ≔ RSD ​ ( s ) A\coloneqq\text{{RSD}}\left(s^{\prime}\right),B^{\prime}\coloneqq D,B\coloneqq\text{{RSD}}\left(s\right) and g g the identity function, eq. 204 follows.

Suppose RSD nd ​ ( s ) ∖ D \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus D is non-empty. By the same result, eq. 204 is a strict inequality for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and we conclude that Power 𝒟 bound ( s ′ , 1 ) ≱ most : 𝔇 bound Power 𝒟 bound ( s , 1 ) \text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s^{\prime},1\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\text{{Power}}_{\mathcal{D}_{\text{bound}}}\left(s,1\right) . ∎

See 6.13

###### Proof.

Let D sub ≔ ϕ ⋅ D ′ D_{\text{sub}}\coloneqq\phi\cdot D^{\prime} , where D sub ⊆ D D_{\text{sub}}\subseteq D by assumption. Let X ≔ { s i ∈ 𝒮 ∣ max 𝐝 ∈ D ′ ∪ D ⁡ 𝐝 ⊤ ​ 𝐞 s i > 0 } X\coloneqq\left\{s_{i}\in\mathcal{S}\mid\max_{\mathbf{d}\in D^{\prime}\cup D}\mathbf{d}^{\top}\mathbf{e}_{s_{i}}>0\right\} . Define ϕ ′ ​ ( s i ) ≔ { ϕ ⁡ ( s i ) if ​ s i ∈ X s i else . \phi^{\prime}(s_{i})\coloneqq\begin{cases}\phi(s_{i})&\text{ if }s_{i}\in X\\ s_{i}&\text{ else}.\end{cases} (206)

Since ϕ \phi is an involution, ϕ ′ \phi^{\prime} is also an involution. Furthermore, by the definition of X X , ϕ ′ ⋅ D ′ = D sub \phi^{\prime}\cdot D^{\prime}=D_{\text{sub}} and ϕ ′ ⋅ D sub = D ′ \phi^{\prime}\cdot D_{\text{sub}}=D^{\prime} (because we assumed that both equalities hold for ϕ \phi ).

Let D ∗ ≔ D ′ ∪ D sub ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) D^{*}\coloneqq D^{\prime}\cup D_{\text{sub}}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) . ϕ ′ ⋅ D ∗ \displaystyle\phi^{\prime}\cdot D^{*} ≔ ϕ ′ ⋅ ( D ′ ∪ D sub ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) ) \displaystyle\coloneqq\phi^{\prime}\cdot\left(D^{\prime}\cup D_{\text{sub}}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)\right) (207) = ( ϕ ′ ⋅ D ′ ) ∪ ( ϕ ′ ⋅ D sub ) ∪ ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) \displaystyle=\left(\phi^{\prime}\cdot D^{\prime}\right)\cup\left(\phi^{\prime}\cdot D_{\text{sub}}\right)\cup\phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) (208) = D sub ∪ D ′ ∪ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) \displaystyle=D_{\text{sub}}\cup D^{\prime}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right) (209) ≕ D ∗ . \displaystyle\eqqcolon D^{*}. (210)

In eq. 209 , we know that ϕ ′ ⋅ D ′ = D sub \phi^{\prime}\cdot D^{\prime}=D_{\text{sub}} and ϕ ′ ⋅ D sub = D ′ \phi^{\prime}\cdot D_{\text{sub}}=D^{\prime} . We just need to show that ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D) .

Suppose ∃ s i ∈ X , 𝐝 ′ ∈ RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) : 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i > 0 \exists s_{i}\in X,\mathbf{d}^{\prime}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}}>0 . By the definition of X X , ∃ 𝐝 ∈ D ′ ∪ D : 𝐝 ⊤ ​ 𝐞 s i > 0 \exists\mathbf{d}\in D^{\prime}\cup D\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\top}\mathbf{e}_{s_{i}}>0 . Then 𝐝 ⊤ ​ 𝐝 ′ \displaystyle\mathbf{d}^{\top}\mathbf{d}^{\prime} = ∑ j = 1 | 𝒮 | 𝐝 ⊤ ​ ( 𝐝 ′ ⊙ 𝐞 s j ) \displaystyle=\sum_{j=1}^{\left|\mathcal{S}\right|}\mathbf{d}^{\top}(\mathbf{d}^{\prime}\odot\mathbf{e}_{s_{j}}) (211) ≥ 𝐝 ⊤ ​ ( 𝐝 ′ ⊙ 𝐞 s i ) \displaystyle\geq\mathbf{d}^{\top}(\mathbf{d}^{\prime}\odot\mathbf{e}_{s_{i}}) (212) = 𝐝 ⊤ ​ ( ( 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i ) ​ 𝐞 s i ) \displaystyle=\mathbf{d}^{\top}\left((\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}})\mathbf{e}_{s_{i}}\right) (213) = ( 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i ) ⋅ ( 𝐝 ⊤ ​ 𝐞 s i ) \displaystyle=(\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}})\cdot(\mathbf{d}^{\top}\mathbf{e}_{s_{i}}) (214) > 0 . \displaystyle>0. (215)

Equation 211 follows from the definitions of the dot and Hadamard products. Equation 212 follows because 𝐝 \mathbf{d} and 𝐝 ′ \mathbf{d}^{\prime} have non-negative entries. Equation 215 follows because 𝐝 ⊤ ​ 𝐞 s i \mathbf{d}^{\top}\mathbf{e}_{s_{i}} and 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i \mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}} are both positive. But eq. 215 shows that 𝐝 ⊤ ​ 𝐝 ′ > 0 \mathbf{d}^{\top}\mathbf{d}^{\prime}>0 , contradicting our assumption that 𝐝 \mathbf{d} and 𝐝 ′ \mathbf{d}^{\prime} are orthogonal.

Therefore, such an s i s_{i} cannot exist, and X ′ ≔ { s i ′ ∈ 𝒮 ∣ max 𝐝 ′ ∈ RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ⁡ 𝐝 ′ ⁣ ⊤ ​ 𝐞 s i > 0 } ⊆ ( 𝒮 ∖ X ) X^{\prime}\coloneqq\left\{s_{i}^{\prime}\in\mathcal{S}\mid\max_{\mathbf{d}^{\prime}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)}\mathbf{d}^{\prime\top}\mathbf{e}_{s_{i}}>0\right\}\subseteq(\mathcal{S}\setminus X) . By eq. 206 , ∀ s i ′ ∈ X ′ : ϕ ′ ​ ( s i ′ ) = s i ′ \forall s_{i}^{\prime}\in X^{\prime}\mathrel{\mathop{\ordinarycolon}}\phi^{\prime}(s_{i}^{\prime})=s_{i}^{\prime} . Thus, ϕ ′ ⋅ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \phi^{\prime}\cdot\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D) , and eq. 209 follows. We conclude that ϕ ′ ⋅ D ∗ = D ∗ \phi^{\prime}\cdot D^{*}=D^{*} .

Consider Z ≔ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) ∪ D ∪ D ′ Z\coloneqq\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)\cup D\cup D^{\prime} . First, Z ⊆ RSD ​ ( s ) Z\subseteq\text{{RSD}}\left(s\right) by definition. Second, RSD nd ​ ( s ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ∪ ( RSD nd ​ ( s ) ∩ D ′ ) ∪ ( RSD nd ​ ( s ) ∩ D ) ⊆ Z \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\cup(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D^{\prime})\cup(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D)\subseteq Z . Note that D ∗ = Z ∖ ( D ∖ D sub ) D^{*}=Z\setminus(D\setminus D_{\text{sub}}) . ℙ 𝒟 any ( D ′ , average ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right) = p 𝒟 any ​ ( D ′ ≥ RSD ​ ( s ) ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(D^{\prime}\geq\text{{RSD}}\left(s\right)\right) (216) ≤ most : 𝔇 any p 𝒟 any ( D ≥ RSD ( s ) ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}p_{\mathcal{D}_{\text{any}}}\left(D\geq\text{{RSD}}\left(s\right)\right) (217) = ℙ 𝒟 any ( D , average ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right). (218)

Since ϕ ⋅ D ′ ⊆ D \phi\cdot D^{\prime}\subseteq D and ND ​ ( D ′ ) ⊆ D ′ \text{{ND}}\left(D^{\prime}\right)\subseteq D^{\prime} , ϕ ⋅ ND ​ ( D ′ ) ⊆ D \phi\cdot\text{{ND}}\left(D^{\prime}\right)\subseteq D . Then eq. 217 holds by applying Lemma E.28 with A ≔ D ′ , B ′ ≔ D sub , B ≔ D , C ≔ RSD ​ ( s ) A\coloneqq D^{\prime},B^{\prime}\coloneqq D_{\text{sub}},B\coloneqq D,C\coloneqq\text{{RSD}}\left(s\right) , and the previously defined Z Z which we showed satisfies ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C . Furthermore, involution ϕ ′ \phi^{\prime} satisfies ϕ ′ ⋅ B ∗ = ϕ ′ ⋅ ( Z ∖ ( B ∖ B ′ ) ) = Z ∖ ( B ∖ B ′ ) = B ∗ \phi^{\prime}\cdot B^{*}=\phi^{\prime}\cdot\left(Z\setminus(B\setminus B^{\prime})\right)=Z\setminus(B\setminus B^{\prime})=B^{*} by eq. 210 .

When RSD nd ​ ( s ) ∩ ( D ∖ D sub ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap\left(D\setminus D_{\text{sub}}\right) is non-empty, since B ′ ⊆ C B^{\prime}\subseteq C by assumption, Lemma E.28 also shows that eq. 217 is strict for all 𝒟 X ​ - iid ∈ 𝔇 c/b/ iid \mathcal{D}_{X\text{-}\textsc{iid}}\in\mathfrak{D}_{\textsc{c/b/}\textsc{iid}} , and that ℙ 𝒟 any ( D ′ , average ) ≱ most : 𝔇 any ℙ 𝒟 any ( D , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right) . ∎

###### Proposition E.50 ( Rsd properties) .

Let 𝐝 ∈ RSD ​ ( s ) \mathbf{d}\in\text{{RSD}}\left(s\right) . 𝐝 \mathbf{d} is element-wise non-negative and ‖ 𝐝 ‖ 1 = 1 \left\lVert\mathbf{d}\right\rVert_{1}=1 .

###### Proof.

𝐝 \mathbf{d} has non-negative elements because it equals the limit of lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) \lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma) , whose elements are non-negative by Proposition E.3 item 1 . ‖ 𝐝 ‖ 1 \displaystyle\left\lVert\mathbf{d}\right\rVert_{1} = ‖ lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) ‖ 1 \displaystyle=\left\lVert\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma)\right\rVert_{1} (219) = lim γ → 1 ( 1 − γ ) ​ ‖ 𝐟 ⁡ ( γ ) ‖ 1 \displaystyle=\lim_{\gamma\to 1}(1-\gamma)\left\lVert\mathbf{f}(\gamma)\right\rVert_{1} (220) = 1 . \displaystyle=1. (221) Equation 219 follows because the definition of rsd s ( definition 6.10 ) ensures that ∃ 𝐟 ∈ ℱ ⁡ ( s ) : lim γ → 1 ( 1 − γ ) ​ 𝐟 ​ ( γ ) = 𝐝 \exists\mathbf{f}\in\Fop(s)\mathrel{\mathop{\ordinarycolon}}\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}(\gamma)=\mathbf{d} . Equation 220 follows because ‖ ⋅ ‖ 1 \left\lVert\cdot\right\rVert_{1} is a continuous function. Equation 221 follows because ‖ 𝐟 ⁡ ( γ ) ‖ 1 = 1 1 − γ \left\lVert\mathbf{f}(\gamma)\right\rVert_{1}=\frac{1}{1-\gamma} by Proposition E.3 item 2 . ∎

###### Lemma E.51 (When reachable with probability 1, 1-cycles induce non-dominated rsd s) .

If 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) , then 𝐞 s ′ ∈ RSD nd ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) .

###### Proof.

If 𝐝 ∈ RSD ​ ( s ) \mathbf{d}\in\text{{RSD}}\left(s\right) is distinct from 𝐞 s ′ \mathbf{e}_{s^{\prime}} , then ‖ 𝐝 ‖ 1 = 1 \left\lVert\mathbf{d}\right\rVert_{1}=1 and 𝐝 \mathbf{d} has non-negative entries by Proposition E.50 . Since 𝐝 \mathbf{d} is distinct from 𝐞 s ′ \mathbf{e}_{s^{\prime}} , then its entry for index s ′ s^{\prime} must be strictly less than 1: 𝐝 ⊤ ​ 𝐞 s ′ < 1 = 𝐞 s ′ ⊤ ​ 𝐞 s ′ \mathbf{d}^{\top}\mathbf{e}_{s^{\prime}}<1=\mathbf{e}_{s^{\prime}}^{\top}\mathbf{e}_{s^{\prime}} . Therefore, 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) is strictly optimal for the reward function 𝐫 ≔ 𝐞 s ′ \mathbf{r}\coloneqq\mathbf{e}_{s^{\prime}} , and so 𝐞 s ′ ∈ RSD nd ​ ( s ) \mathbf{e}_{s^{\prime}}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) . ∎

See 6.14

###### Proof.

Suppose 𝐞 s x , 𝐞 s ′ ∈ RSD ​ ( s ) \mathbf{e}_{s_{x}},\mathbf{e}_{s^{\prime}}\in\text{{RSD}}\left(s\right) are distinct. Let ϕ ≔ ( s x ​ s ′ ) , D ′ ≔ { 𝐞 s x } , D ≔ RSD ​ ( s ) ∖ { 𝐞 s x } \phi\coloneqq(s_{x}\,\,\,s^{\prime}),D^{\prime}\coloneqq\left\{\mathbf{e}_{s_{x}}\right\},D\coloneqq\text{{RSD}}\left(s\right)\setminus\left\{\mathbf{e}_{s_{x}}\right\} . ϕ ⋅ D ′ = { 𝐞 s ′ } ⊆ RSD ​ ( s ) ∖ { 𝐞 s x } ≕ D \phi\cdot D^{\prime}=\left\{\mathbf{e}_{s^{\prime}}\right\}\subseteq\text{{RSD}}\left(s\right)\setminus\left\{\mathbf{e}_{s_{x}}\right\}\eqqcolon D since s x ≠ s ′ s_{x}\neq s^{\prime} . D ′ ∪ D = RSD ​ ( s ) D^{\prime}\cup D=\text{{RSD}}\left(s\right) and RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) = RSD nd ​ ( s ) ∖ RSD ​ ( s ) = ∅ \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\text{{RSD}}\left(s\right)=\emptyset trivially have pairwise orthogonal vector elements. Then apply Theorem 6.13 to conclude that ℙ 𝒟 any ( { 𝐞 s x } , average ) ≤ most : 𝔇 any ℙ 𝒟 any ( RSD ( s ) ∖ { 𝐞 s x } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{s_{x}}\},\text{average}\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus\{\mathbf{e}_{s_{x}}\},\text{average}\right) .

Suppose there exists another 𝐞 s ′′ ∈ RSD ​ ( s ) \mathbf{e}_{s^{\prime\prime}}\in\text{{RSD}}\left(s\right) . By Lemma E.51 , 𝐞 s ′′ ∈ RSD nd ​ ( s ) \mathbf{e}_{s^{\prime\prime}}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) . Furthermore, since s ′′ ∉ { s ′ , s x } s^{\prime\prime}\not\in\left\{s^{\prime},s_{x}\right\} , 𝐞 s ′′ ∈ ( RSD ​ ( s ) ∖ { 𝐞 s x } ) ∖ { 𝐞 s ′ } = D ∖ ϕ ⋅ D ′ \mathbf{e}_{s^{\prime\prime}}\in\left(\text{{RSD}}\left(s\right)\setminus\left\{\mathbf{e}_{s_{x}}\right\}\right)\setminus\left\{\mathbf{e}_{s^{\prime}}\right\}=D\setminus\phi\cdot D^{\prime} . Therefore, 𝐞 s ′′ ∈ RSD nd ​ ( s ) ∩ ( D ∖ ϕ ⋅ D ′ ) \mathbf{e}_{s^{\prime\prime}}\in\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap\left(D\setminus\phi\cdot D^{\prime}\right) . Then apply the second condition of Theorem 6.13 to conclude that ℙ 𝒟 any ( { 𝐞 s x } , average ) ≱ most : 𝔇 bound ℙ 𝒟 any ( RSD ( s ) ∖ { 𝐞 s x } , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\{\mathbf{e}_{s_{x}}\},\text{average}\right)\not\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus\{\mathbf{e}_{s_{x}}\},\text{average}\right) . ∎

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .