# Parametrically Retargetable Decision-Makers Tend To Seek Power ###### Abstract If capable ai agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive [ Turner et al., 2021 ] . However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal. We consider a range of models of ai decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable , and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad. We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma’s Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans. ## 1 Introduction Bostrom [2014] , Russell [2019] argue that in the future, we may know how to train and deploy superintelligent ai agents which capably optimize goals in the world. Furthermore, we would not want such agents to act against our interests by ensuring their own survival, by gaining resources, and by competing with humanity for control over the future. Turner et al. [2021] show that most reward functions have optimal policies which seek power over the future, whether by staying alive or by keeping their options open. Some Markov decision processes ( mdp s) cause there to be more ways for power-seeking to be optimal, than for it to not be optimal. Analogously, there are relatively few goals for which dying is a good idea. We show that a wide range of decision-making algorithms produce these power-seeking tendencies—they are not unique to reward maximizers. We develop a simple, broad criterion of functional retargetability ( definition 3.5 ) which is a sufficient condition for power-seeking tendencies. Crucially, these results allow us to reason about what decisions are incentivized by most algorithm parameter inputs, even when it is impractical to compute the agent’s decisions for any given parameter input. Useful “general” ai agents could be directed to complete a range of tasks. However, we show that this flexibility can cause the ai to have power-seeking tendencies. In section 2 and section 3 , we discuss how a “retargetability” property creates statistical tendencies by which agents make similar decisions for a wide range of parameter settings for their decision-making algorithms. Basically, if a decision-making algorithm is retargetable, then for every configuration under which a decision-making algorithm does not choose to seek power, there exist several reconfigurations which do induce power-seeking. More formally, for every decision-making parameter setting θ \theta which does not induce power-seeking, n n -retargetability ensures we can injectively map θ \theta to n n parameters θ 1 ′ , … , θ n ′ \theta^{\prime}_{1},\ldots,\theta^{\prime}_{n} which do induce power-seeking. Equipped with these results, section 4 works out agent incentives in the Montezuma’s Revenge game. Section 5 speculates that increasingly useful and impressive learning algorithms will be increasingly retargetable, and how retargetability can imply power-seeking tendencies. By this reasoning, increasingly powerful rl techniques may (eventually) train increasingly competent real-world power-seeking agents. Such agents could be unaligned with human values [ Russell, 2019 ] and—we speculate—would take power from humanity. ## 2 Statistical tendencies for a range of decision-making algorithms Turner et al. [2021] consider the Pac-Man video game, in which an agent consumes pellets, navigates a maze, and avoids deadly ghosts ( Figure 1 ). Instead of the usual score function, Turner et al. [2021] consider optimal action across a range of state-based reward functions. They show that most reward functions have an (average-)optimal policy which avoids immediate death in order to navigate to a future terminal state. 1 1 1 We use “reward function” somewhat loosely in implying that reward functions reasonably describe a trained agent’s goals. Turner [2022] argues that capable rl algorithms do not necessarily train policy networks which are best understood as optimizing the reward function itself. Rather, they point out that—especially in policy-gradient approaches—reward provides gradients to the network and thereby modifies the network’s generalization properties, but doesn’t ensure the agent generalizes to “robustly optimizing reward” off of the training distribution. Our results show that optimality is not required. Instead, if the agent’s decision-making is parametrically retargetable from death to other outcomes, Pac-Man avoids the ghost under most decision-making parameter inputs. To build intuition about these notions, consider three outcomes ( i . e ., terminal states): Immediate death to a nearby ghost, consuming a cherry, and consuming an apple. Let A ≔ { } A\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\right\} and B ≔ { , } B\coloneqq\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\} . For simplicity of exposition, we assume these are the three possible terminal states. Suppose that in some fashion, the agent probabilistically decides on an outcome to induce. Let p p take as input a set of outcomes and return the probability that the agent selects one of those outcomes. For example, p ⁡ ( { } ) p(\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\}) is the probability that the agent selects , and p ⁡ ( { , } ) p(\left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}\right\}) is the probability that the agent escapes the ghost and ends up in an apple or cherry terminal state. But this just amounts to a probability distribution over the terminal states. We want to examine how decision-making changes as we swap out the parameter inputs to the agent’s decision-making algorithm with decision-making parameter space Θ \Theta . We then let p ⁡ ( X ∣ θ ) p(X\mid\theta) take as input a set of outcomes X X and a decision-making algorithm parameter setting θ ∈ Θ \theta\in\Theta , and return the probability that the agent chooses an outcome in X X . We first consider agents which maximize terminal-state utility, following Turner et al. [2021] (in their language, “average-reward optimality”). Suppose that the agent has a utility function parameter 𝐮 \mathbf{u} assigning a real number to each of the three outcomes. Then the relevant parameter space is the agent’s utility function 𝐮 ∈ Θ ≔ ℝ 3 \mathbf{u}\in\Theta\coloneqq\mathbb{R}^{3} . p max ​ ( A ∣ 𝐮 ) p_{\max}(A\mid\mathbf{u}) indicates whether has the most utility: 𝐮 ⁡ ( ) ≥ max ⁡ ( 𝐮 ⁡ ( ) , 𝐮 ⁡ ( ) ) \mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf})\geq\max(\mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}),\mathbf{u}(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf})) . Consider the utility function 𝐮 \mathbf{u} in Table 1 . Since has strictly maximal utility, the agent selects : p max ​ ( A ∣ 𝐮 ) = 1 > 0 = p max ​ ( B ∣ 𝐮 ) p_{\max}(A\mid\mathbf{u})=1>0=p_{\max}(B\mid\mathbf{u}) . However, most “variants" of 𝐮 \mathbf{u} have an optimal policy which stays alive. That is, for every 𝐮 \mathbf{u} for which immediate death is optimal but immediate survival is not, we can swap the utility of e . g ., and via permutation ϕ ↔ \phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}} to produce a new utility function 𝐮 ′ ≔ ϕ ↔ ⋅ 𝐮 \mathbf{u}^{\prime}\coloneqq\phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}}\cdot\mathbf{u} for which staying alive ( right ) is strictly optimal. The same kind of argumentation holds for ϕ ↔ \phi_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}} . Table 1 suggests a counting argument. For every utility function 𝐮 \mathbf{u} for which is optimal, there are two unique utility functions ϕ 1 ⋅ 𝐮 , ϕ 2 ⋅ 𝐮 \phi_{1}\cdot\mathbf{u},\phi_{2}\cdot\mathbf{u} under which either or is optimal. In section 3 , we will generalize this particular counting argument. Definition 3.3 shows a functional condition ( retargetability ) under which the agent decides to avoid the ghost, for most parameter inputs to the decision-making algorithm. Given this retargetability assumption, Proposition 3.4 roughly shows that most θ ∈ Θ \theta\in\Theta induce p ⁡ ( B ∣ θ ) ≥ p ⁡ ( A ∣ θ ) p(B\mid\theta)\geq p(A\mid\theta) . First, consider two more retargetable decision-making functions: Uniformly randomly picking a terminal state. p rand p_{\text{rand}} ignores the reward function and assigns equal probability to each terminal state in Pac-Man’s state space. Choosing an action based on a numerical parameter. p numerical p_{\text{numerical}} takes as input a natural number θ ∈ Θ ≔ { 1 , … , 6 } \theta\in\Theta\coloneqq\left\{1,\ldots,6\right\} and makes decisions as follows: p numerical ​ ( A ∣ θ ) ≔ { 1 if θ = 1 , 0 otherwise. p numerical ​ ( B ∣ θ ) ≔ 1 − p numerical ​ ( A ∣ θ ) . \displaystyle p_{\text{numerical}}(A\mid\theta)\coloneqq\begin{cases}1\quad\text{ if $\theta=1$},\\ 0\quad\text{ otherwise.}\end{cases}\qquad p_{\text{numerical}}(B\mid\theta)\coloneqq 1-p_{\text{numerical}}(A\mid\theta). (1) In this situation, Θ \Theta is acted on by permutations over 6 6 elements ϕ ∈ S 6 \phi\in S_{6} . Then p numerical p_{\text{numerical}} is retargetable from A A to B B via ϕ k : 1 ↔ k , k ≠ 1 \phi_{k}\mathrel{\mathop{\ordinarycolon}}1\leftrightarrow k,k\neq 1 . p max p_{\text{max}} , p rand p_{\text{rand}} , and p numerical p_{\text{numerical}} encode varying sensitivities to the utility function parameter input, and to the internal structure of the Pac-Man decision process. Nonetheless, they all are retargetable from A A to B B . For an example of a non -retargetable function, consider p stubborn ​ ( X ∣ θ ) ≔ 𝟙 X = A p_{\text{stubborn}}(X\mid\theta)\coloneqq\mathbbm{1}_{X=A} which returns 1 1 for A A and 0 0 otherwise. However, we cannot explicitly define and evaluate more interesting functions, such as those defined by reinforcement learning training processes. For example, given that we provide such-and-such reward function in a fixed task environment, what is the probability that the learned policy will take action a a ? We will analyze such procedures in section 4 . We now motivate the title of this work. For most parameter settings, retargetable decision-makers induce an element of the larger set of outcomes. Such decision-makers tend to induce an element of a larger set of outcomes (with the “tendency” being taken across parameter settings). Consider that the larger set of outcomes { , } \left\{\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf},\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}\right\} can only be induced if Pac-Man stays alive. Intuitively, navigating to this larger set is power-seeking because the agent retains more optionality ( i . e ., the agent can’t do anything when dead). Therefore, parametrically retargetable decision-makers tend to seek power . ## 3 Formal notions of retargetability and decision-making tendencies Section 2 informally illustrated parametric retargetability in the context of swapping which utilities are assigned to which outcomes in the Pac-Man video game. For many utility-based decision-making algorithms, swapping the utility assignments also swaps the agent’s final decisions. For example, if death is anti-rational, and then death’s utility is swapped with the cherry utility, then now the cherry is anti-rational. In this section, we formalize the notion of parametric retargetability and of “most” parameter inputs producing a given result. In section 4 , we will use these formal notions to reason about the behavior of rl -trained policies in the Montezuma’s Revenge video game. To define our notion of “retargeting”, we assume that Θ \Theta is a subset of a set acted on by symmetric group S d S_{d} , which consists of all permutations on d d items ( e . g ., in the rl setting, this might represent states or observations). A parameter θ \theta ’s orbit is the set of θ \theta ’s permuted variants. For example, Table 1 lists the six orbit elements of the parameter 𝐮 \mathbf{u} . ###### Definition 3.1 (Orbit of a parameter) . Let θ ∈ Θ \theta\in\Theta . The orbit of θ \theta under the symmetric group S d S_{d} is S d ⋅ θ ≔ { ϕ ⋅ θ ∣ ϕ ∈ S d } S_{d}\cdot\theta\coloneqq\left\{\phi\cdot\theta\mid\phi\in S_{d}\right\} . Sometimes, Θ \Theta is not closed under permutation. In that case, the orbit inside Θ \Theta is Orbit | Θ ​ ( θ ) ≔ ( S d ⋅ θ ) ∩ Θ \mathrm{Orbit}|_{\Theta}\left(\theta\right)\coloneqq\left(S_{d}\cdot\theta\right)\cap\Theta . Let p ⁡ ( B ∣ θ ) p(B\mid\theta) return the probability that the agent chooses an outcome in B B given θ \theta . To express “ B B -outcomes are chosen instead of A A -outcomes”, we write p ⁡ ( B ∣ θ ) > p ⁡ ( A ∣ θ ) p(B\mid\theta)>p(A\mid\theta) . However, even “retargetable” decision-making functions (defined shortly) generally won’t choose a B B -outcome for every input θ \theta . Instead, we consider the orbit-level tendencies of such decision-makers, showing that for every parameter input θ ∈ Θ \theta\in\Theta , most of θ \theta ’s permutations push the decision towards B B instead of A A . ###### Definition 3.2 (Inequalities which hold for most orbit elements) . Suppose Θ \Theta is a subset of a set acted on by S d S_{d} , the symmetric group on d d elements. Let f : { A , B } × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} and let n ≥ 1 n\geq 1 . We write f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta) when, for all θ ∈ Θ \theta\in\Theta , the following cardinality inequality holds: | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( B ∣ θ ′ ) > f ⁡ ( A ∣ θ ′ ) } | ≥ n ​ | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( B ∣ θ ′ ) < f ⁡ ( A ∣ θ ′ ) } | . \left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(B\mid\theta^{\prime})>f(A\mid\theta^{\prime})\right\}\right|\geq n\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(B\mid\theta^{\prime}) 1 = | A | \left|B\right|=2>1=\left|A\right| . p max p_{\text{max}} ’s simple retargetability arises in part due to B B having more outcomes. ###### Proposition 3.4 (Simply-retargetable functions have orbit-level tendencies) . If f f is ( Θ , A ​ → simple ​ B ) (\Theta,A\overset{\text{simple}}{\to}B) -retargetable, then f ( B ∣ θ ) ≥ most : Θ 1 f ( A ∣ θ ) . f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{1}f(A\mid\theta). We now want to make even stronger claims— how much of each orbit incentivizes B B over A A ? Turner et al. [2021] asked whether the existence of multiple retargeting permutations ϕ i \phi_{i} guarantees a quantitative lower-bound on the fraction of θ ∈ Θ \theta\in\Theta for which B B is chosen. Theorem 3.6 answers “yes.” ###### Definition 3.5 (Multiply retargetable function) . Let Θ \Theta be a subset of a set acted on by S d S_{d} , and let f : { A , B } × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} . f f is a ( Θ , A ​ → 𝑛 ​ B ) (\Theta,A\overset{n}{\to}B) -retargetable function when, for each θ ∈ Θ \theta\in\Theta , we can choose permutations ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} which satisfy the following conditions: Consider any θ A ∈ Orbit | Θ , A > B ​ ( θ ) ≔ { θ ∗ ∈ Orbit | Θ ​ ( θ ) ∣ f ⁡ ( A ∣ θ ∗ ) > f ⁡ ( B ∣ θ ∗ ) } \theta^{A}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\coloneqq\left\{\theta^{*}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f(A\mid\theta^{*})>f(B\mid\theta^{*})\right\} . 1. Retargetable via n n permutations. ∀ i = 1 , … , n : f ⁡ ( A ∣ ϕ i ⋅ θ A ) < f ⁡ ( B ∣ ϕ i ⋅ θ A ) \forall i=1,\ldots,n\mathrel{\mathop{\ordinarycolon}}f\left(A\mid\phi_{i}\cdot\theta^{A}\right) B ​ ( θ ) : ϕ i ⋅ θ A ≠ ϕ j ⋅ θ ′ \forall i\neq j,\theta^{\prime}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot\theta^{A}\neq\phi_{j}\cdot\theta^{\prime} . ###### Theorem 3.6 (Multiply retargetable functions have orbit-level tendencies) . If f f is ( Θ , A ​ → 𝑛 ​ B ) (\Theta,A\overset{n}{\to}B) -retargetable, then f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) . f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta). ###### Proof outline (full proof in Appendix B ). For every θ A ∈ Orbit | Θ , A > B ​ ( θ ) \theta^{A}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) such that A A is chosen over B B , item 1 retargets θ A \theta^{A} via n n permutations ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} such that each ϕ i ⋅ θ A \phi_{i}\cdot\theta^{A} makes the agent choose B B over A A . These permuted parameters are valid parameter inputs by item 2 . Furthermore, the ϕ i ⋅ θ A \phi_{i}\cdot\theta^{A} are distinct by item 3 . Therefore, the cosets ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) \phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) are pairwise disjoint. By a counting argument, every orbit must contain at least n n times as many parameters choosing B B over A A , than vice versa. ∎ ## 4 Decision-making tendencies in Montezuma’s Revenge To illustrate a high-dimensional setting in which parametrically retargetable decision-makers tend to seek power, we consider Montezuma’s Revenge ( mr ), an Atari adventure game in which the player navigates deadly traps and collects treasure. The game is notoriously difficult for ai agents due to its sparse reward. mr was only recently solved [ Ecoffet et al., 2021 ] . Figure 2 shows the starting observation o 0 o_{0} for the first level. This section culminates with section 4.3 , where we argue that increasingly powerful rl training processes will cause increasing retargetability via the reward function, which in turn causes increasingly strong decision-making tendencies. ##### Terminology. Retargetability is a property of the policy training process, and power-seeking is a property of the trained policy. More precisely, the policy training process takes as input a parameterization θ \theta and outputs a probability distribution over policies. For each trained policy drawn from this distribution, the environment, starting state, and the drawn policy jointly specify a probability distribution over trajectories. Therefore, the training process associates each parameterization θ \theta with the mixture distribution P P over trajectories (with the mixture taken over the distribution of trained policies). A policy training process can be simply retargeted from one trajectory set A A to another trajectory set B B when there exists a permutation ϕ ∈ S d \phi\in S_{d} such that, for every θ \theta for which P ⁡ ( A ∣ θ ) > P ⁡ ( B ∣ θ ) P(A\mid\theta)>P(B\mid\theta) , we have P ⁡ ( A ∣ ϕ ⋅ θ ) < P ⁡ ( B ∣ ϕ ⋅ θ ) P(A\mid\phi\cdot\theta) 1 5 = p rand ​ ( A ∣ θ ) \forall\theta\in\Theta\mathrel{\mathop{\ordinarycolon}}p_{\text{rand}}(B\mid\theta)=\frac{4}{5}>\frac{1}{5}=p_{\text{rand}}(A\mid\theta) , all parameter inputs produce a greater chance of B B than of A A , so p rand p_{\text{rand}} is (trivially) retargetable from A A to B B . Always choosing the same action. p stubborn p_{\text{stubborn}} always chooses ↓ \downarrow . Since ∀ θ ∈ Θ : p stubborn ​ ( A ∣ θ ) = 1 > 0 = p stubborn ​ ( B ∣ θ ) \forall\theta\in\Theta\mathrel{\mathop{\ordinarycolon}}p_{\text{stubborn}}(A\mid\theta)=1>0=p_{\text{stubborn}}(B\mid\theta) , all parameter inputs produce a greater chance of A A than of B B . p stubborn p_{\text{stubborn}} is not retargetable from A A to B B . Greedily optimizing state-action reward. Let Θ ≔ ℝ 𝒮 × 𝒜 \Theta\coloneqq\mathbb{R}^{\mathcal{S}\times\mathcal{A}} be the space of state-action reward functions. Let p max p_{\text{max}} greedily maximize initial state-action reward, breaking ties uniformly randomly. We now check that p max p_{\text{max}} is retargetable from A A to B B . Suppose θ ∗ ∈ Θ \theta^{*}\in\Theta is such that p max ​ ( A ∣ θ ∗ ) > p max ​ ( B ∣ θ ∗ ) p_{\text{max}}(A\mid\theta^{*})>p_{\text{max}}(B\mid\theta^{*}) . Then among the initial action rewards, θ ∗ \theta^{*} assigns strictly maximal reward to ↓ \downarrow , and so p max ​ ( A ∣ θ ∗ ) = 1 p_{\text{max}}(A\mid\theta^{*})=1 . Let ϕ \phi swap the reward for the ↓ \downarrow and jump actions. Then ϕ ⋅ θ ∗ \phi\cdot\theta^{*} assigns strictly maximal reward to jump . This means that p max ​ ( A ∣ ϕ ⋅ θ ∗ ) = 0 < 1 = p max ​ ( B ∣ ϕ ⋅ θ ∗ ) p_{\text{max}}(A\mid\phi\cdot\theta^{*})=0<1=p_{\text{max}}(B\mid\phi\cdot\theta^{*}) , satisfying definition 3.3 . Then apply Proposition 3.4 to conclude that p max ( B ∣ θ ) ≥ most : Θ 1 p max ( A ∣ θ ) p_{\text{max}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{1}p_{\text{max}}(A\mid\theta) . In fact, appendix A shows that p max p_{\text{max}} is ( Θ , A ​ → 4 ​ B ) (\Theta,A\overset{4}{\to}B) -retargetable ( definition 3.5 ), and so p max ( B ∣ θ ) ≥ most : Θ 4 p max ( A ∣ θ ) p_{\text{max}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{4}p_{\text{max}}(A\mid\theta) . The reasoning is more complicated, but the rule of thumb is: When decisions are made based on the reward of outcomes, then a proportionally larger set B B of outcomes induces proportionally strong retargetability, which induces proportionally strong orbit-level incentives. Learning an exploitation policy. Suppose we run a bandit algorithm which tries different initial actions, learns their rewards, and produces an exploitation policy which maximizes estimated reward. The algorithm uses ϵ \epsilon -greedy exploration and trains for T T trials. Given fixed T T and ϵ \epsilon , p bandit ​ ( A ∣ θ ) p_{\text{bandit}}(A\mid\theta) returns the probability that an exploitation policy is learned which chooses an action in A A ; likewise for p bandit ​ ( B ∣ θ ) p_{\text{bandit}}(B\mid\theta) . Here is a heuristic argument that p bandit p_{\text{bandit}} is retargetable. Since the reward is deterministic, the exploitation policy will choose an optimal action if the agent has tried each action at least once, which occurs with a probability approaching 1 1 exponentially quickly in the number of trials T T . Then when T T is large, p bandit p_{\text{bandit}} approximates p max p_{\text{max}} , which is retargetable. Therefore, perhaps p bandit p_{\text{bandit}} is also retargetable. A more careful analysis in appendix C.1 reveals that p bandit p_{\text{bandit}} is 4-retargetable from A A to B B , and so p bandit ( B ∣ θ ) ≥ most : Θ 4 p bandit ( A ∣ θ ) p_{\text{bandit}}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{4}p_{\text{bandit}}(A\mid\theta) . ### 4.2 Tendencies for maximizing reward over the final observation When evaluating the performance of an algorithm in mr , we do not focus on the agent’s initial action. Rather, we focus on the longer-term consequences of the agent’s actions, such as whether the agent leaves the first room. To begin reasoning about such behavior, the reader must distinguish between different kinds of retargetability. Suppose the agent will die unless they choose action ↓ \downarrow at the initial state s 0 s_{0} ( Figure 2 ). By section 4.1 , action-retargetable decision-making procedures tend to choose actions besides ↓ \downarrow . On the other hand, Turner et al. [2021] showed that most reward functions make it reward-optimal to stay alive (in this situation, by choosing ↓ \downarrow ). However, in that situations, the optimal policies are not retargetable across the agent’s immediate choice of action, but rather across future consequences ( i . e ., which room the agent ends up in). With that in mind, we now analyze how often decision-makers leave the first room of mr . 4 4 4 In Appendix C.2 , Figure 3 shows a map of the first level. Decision-making functions decide ⁡ ( θ ) \mathrm{decide}(\theta) produce a probability distribution over policies π ∈ Π \pi\in\Pi , which are rolled out from the initial state s 0 s_{0} to produce observation-action trajectories τ = o 0 ​ a 0 ​ … ​ o T ​ a T ​ … \tau=o_{0}a_{0}\ldots o_{T}a_{T}\ldots , where T T is the rollout length we are interested in. Let O T -reach O_{\text{$T$-reach}} be the set of observations reachable starting from state s 0 s_{0} and acting for T T time steps, let O leave ⊆ O T -reach O_{\text{leave}}\subseteq O_{\text{$T$-reach}} be those observations which can only be realized by leaving, and let O stay ≔ O T -reach ∖ O leave O_{\text{stay}}\coloneqq O_{\text{$T$-reach}}\setminus O_{\text{leave}} . Consider the probability that decide \mathrm{decide} realizes some subset of observations X ⊆ 𝒪 X\subseteq\mathcal{O} at step T T : p decide ​ ( X ∣ θ ) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o T ∈ X ) . p_{\mathrm{decide}}(X\mid\theta)\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in X\right). (3) Let Θ ≔ ℝ 𝒪 \Theta\coloneqq\mathbb{R}^{\mathcal{O}} be the set of reward functions mapping observations o ∈ 𝒪 o\in\mathcal{O} to real numbers, and let T ≔ 1,000 T\coloneqq 1{,}000 . We first consider the previous decision functions, since they are simple to analyze. decide rand \mathrm{decide}_{\text{rand}} randomly chooses a final observation o o which can be realized at step 1,000, and then chooses some policy which realizes o o . 5 5 5 decide rand \mathrm{decide}_{\text{rand}} does not act randomly at each time step, it induces a randomly selected final observation. Analogously, randomly turning a steering wheel is different from driving to a randomly chosen destination. decide rand \mathrm{decide}_{\text{rand}} induces an p rand p_{\text{rand}} defined by eq. 3 . As before, p rand p_{\text{rand}} tends to leave the room under all parameter inputs. decide max ​ ( θ ) \mathrm{decide}_{\max}(\theta) produces a policy which maximizes the reward of the observation at step 1,000 of the rollout. Since mr is deterministic, we discuss which observation decide max ​ ( θ ) \mathrm{decide}_{\max}(\theta) realizes. In a stochastic setting, the decision-maker would choose a policy realizing some probability distribution over step- T T observations, and the analysis would proceed similarly. Here is the semi-formal argument for p max p_{\text{max}} ’s retargetability. There are combinatorially more game-screens visible if the agent leaves the room (due to e . g ., more point combinations, more inventory layouts, more screens outside of the first room). In other words, | O stay | ≪ | O leave | \left|O_{\text{stay}}\right|\ll\left|O_{\text{leave}}\right| . There are more ways for the selected observation to require leaving the room, than not. Thus, p max p_{\text{max}} is extremely retargetable from O stay O_{\text{stay}} to O leave O_{\text{leave}} . Detailed analysis in section C.2 confirms that p max ( O leave ∣ θ ) ≥ most : Θ n p max ( O stay ∣ θ ) p_{\text{max}}(O_{\text{leave}}\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{\text{max}}(O_{\text{stay}}\mid\theta) for the large n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor , which we show implies that p max p_{\text{max}} tends to leave the room. ### 4.3 Tendencies for rl on featurized reward over the final observation In the real world, we do not run p max p_{\text{max}} , which can be computed via T T -depth exhaustive tree search in order to find and induce a maximal-reward observation o T o_{T} . Instead, we use reinforcement learning. Better rl algorithms seem to be more retargetable because of their greater capability to explore. 6 6 6 Conversely, if the agent cannot figure out how to leave the first room, any reward signal from outside of the first room can never causally affect the learned policy. In that case, retargetability away from the first room is impossible. ##### Exploring the first room. Consider a featurized reward function over observations θ ∈ ℝ 𝒪 \theta\in\mathbb{R}^{\mathcal{O}} , which provides an end-of-episode return signal which adds a fixed reward for each item displayed in the observation ( e . g ., 5 reward for a sword, 2 reward for a key). Consider a coefficient vector α ∈ ℝ 4 \alpha\in\mathbb{R}^{4} , with each entry denoting the value of an item, and feat : 𝒪 → ℝ 4 \textrm{feat}\mathrel{\mathop{\ordinarycolon}}\mathcal{O}\to\mathbb{R}^{4} maps observations to feature vectors which tally the items in the agent’s inventory. A reinforcement learning algorithm Alg \mathrm{Alg} uses this return signal to update a fixed-initialization policy network. Then p Alg ​ ( O leave ∣ θ ) p_{\mathrm{Alg}}(O_{\text{leave}}\mid\theta) returns the probability that Alg \mathrm{Alg} trains an policy whose step- T T observation required the agent to leave the initial room. The retargetability ( definition 3.3 ) of Alg \mathrm{Alg} is closely linked to the quality of Alg \mathrm{Alg} as an rl training procedure. For example, as explained in section C.4 , Mnih et al. [2015] ’s dqn isn’t good enough to train policies which leave the first room of mr , and so dqn (trivially) cannot be retargetable away from the first room via the reward function. There isn’t a single featurized reward function for which dqn visits other rooms, and so we can’t have α \alpha such that ϕ ⋅ α \phi\cdot\alpha retargets the agent to O leave O_{\text{leave}} . dqn isn’t good enough at exploring. More formally, in this situation, Alg \mathrm{Alg} is retargetable if there exists a permutation ϕ ∈ S 4 \phi\in S_{4} such that whenever α ∈ Θ ≔ ℝ 4 \alpha\in\Theta\coloneqq\mathbb{R}^{4} induces the learned policies to stay in the room ( p Alg ​ ( O stay ∣ α ) > p Alg ​ ( O leave ∣ α ) p_{\mathrm{Alg}}(O_{\text{stay}}\mid\alpha)>p_{\mathrm{Alg}}(O_{\text{leave}}\mid\alpha) ), ϕ ⋅ α \phi\cdot\alpha makes Alg \mathrm{Alg} train policies which leave the room ( p Alg ​ ( O stay ∣ α ) < p Alg ​ ( O leave ∣ α ) p_{\mathrm{Alg}}(O_{\text{stay}}\mid\alpha) α sword \alpha_{\text{key}}>\alpha_{\text{sword}} . Otherwise, α key < α sword \alpha_{\text{key}}<\alpha_{\text{sword}} (by assumption of unique item reward coefficients); in this case, the agent would leave and acquire the sword (since we assumed it knows how to do so). Then by switching the reward for the key and the sword, we retarget Alg ′ \mathrm{Alg}^{\prime} to go get the sword. Alg ′ \mathrm{Alg}^{\prime} is simply-retargetable away from the first room, because it can explore enough of the environment. ##### Exploring the entire level. Algorithms like go-explore [ Ecoffet et al., 2021 ] are probably good at exploring even given sparse featurized reward. Therefore, go-explore is even more retargetable in this setting, because it is more able to explore and discover the breadth of options (final inventory counts) available to it, and remember how to navigate to them. Furthermore, sufficiently powerful planning algorithms should likewise be retargetable in a similar way, insofar as they can reliably find high-scoring item configurations. We speculate that increasingly “impressive” algorithms (whether rl training or planning) are often more impressive because they can allow retargeting the agent’s final behavior from one kind of outcome, to another. Just as go-explore seems highly retargetable while dqn does not, we expect increasingly impressive algorithms to be increasingly retargetable—whether over actions in a bandit problem, or over the final observation in an rl episode. ## 5 Retargetability can imply power-seeking tendencies ### 5.1 Generalizing the power-seeking theorems for Markov decision processes Turner et al. [2021] considered finite mdp s in which decision-makers took as input a reward function over states ( 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} ) and selected an optimal policy for that reward function. They considered the state visit distributions 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) , which basically correspond to the trajectories which the agent could induce starting from state s s . For F ⊆ ℱ ⁡ ( s ) F\subseteq\Fop(s) , p max ​ ( F ∣ 𝐫 ) p_{\max}(F\mid\mathbf{r}) returns 1 1 if an element of F F is optimal for reward function 𝐫 \mathbf{r} , and 0 0 otherwise. They showed situations where a larger set of distributions F large F_{\text{large}} tended to be optimal over a smaller set: p max ( F large ∣ 𝐫 ) ≥ most : ℝ | 𝒮 | 1 p max ( F small ∣ 𝐫 ) p_{\max}(F_{\text{large}}\mid\mathbf{r})\geq_{\text{{most}}\text{: }\mathbb{R}^{\left|\mathcal{S}\right|}}^{1}p_{\max}(F_{\text{small}}\mid\mathbf{r}) . For example, in Pac-Man, most reward functions make it optimal to stay alive for at least one time step: p max ( F survival ∣ 𝐫 ) ≥ most : ℝ | 𝒮 | 1 p max ( F instant death ∣ 𝐫 ) p_{\max}(F_{\text{survival}}\mid\mathbf{r})\geq_{\text{{most}}\text{: }\mathbb{R}^{\left|\mathcal{S}\right|}}^{1}p_{\max}(F_{\text{instant death}}\mid\mathbf{r}) . Turner et al. [2021] showed that optimal policies tend to seek power by keeping options open and staying alive. Appendix D provides a quantitative generalization of Turner et al. [2021] ’s results on optimal policies. Throughout this paper, we abstracted their arguments away from finite mdp s and reward-optimal decision-making. Instead, parametrically retargetable decision-makers tend to seek power: Proposition A.11 shows that a wide range of decision-making procedures are retargetable over outcomes, and Theorem A.13 demonstrates the retargetability of any decision-making which is determined by the expected utility of outcomes. In particular, these results apply straightforwardly to mdp s. ### 5.2 Better rl algorithms tend to be more retargetable Reinforcement learning algorithms are practically useful insofar as they can train an agent to accomplish some task ( e . g ., cleaning a room). A good rl algorithm is relatively task-agnostic ( e . g ., is not restricted to only training policies which clean rooms). Task-agnosticism suggests retargetability across desired future outcomes / task completions. In mr , suppose we instead give the agent 1 1 reward for the initial state, and 0 0 otherwise. Any reasonable reinforcement learning procedure will just learn to stay put (which is the optimal policy). However, consider whether we can retarget the agent’s policy to beat the game, by swapping the initial state reward with the end-game state reward. Most present-day rl algorithms are not good enough to solve such a sparse game, and so are not retargetable in this sense. But an agent which did enough exploration would also learn a good policy for the permuted reward function. Such an effective training regime could be useful for solving real-world tasks. Many researchers aim to develop effective training regimes. Our results suggest that once rl capabilities reach a certain level, trained agents will tend to seek power in the real world. Presently, it is not dangerous to train an agent to complete a task—such an agent will not be able to complete its task by staying activated against the designers’ wishes. The present lack of danger is not because optimal policies do not have self-preservation tendencies—they do [ Turner et al., 2021 ] . Rather, the lack of danger reflects the fact that present-day rl agents cannot learn such complex action sequences at all . Just as the Montezuma’s Revenge agent had to be sufficiently competent to be retargetable from initial-state reward to game-complete reward, real-world agents have to be sufficiently intelligent in order to be retargetable from outcomes which don’t require power-seeking, to those which do require power-seeking. Here is some speculation. After training an rl agent to a high level of capability, the agent may be optimizing internally represented goals over its model of the environment [ Hubinger et al., 2019 ] . Furthermore, we think that different reward parameter settings would train different internal goals into the agent. To make an analogy, changing a person’s reward circuitry would presumably reinforce them for different kinds of activities and thereby change their priorities. In this sense, trained real-world agents may be retargetable towards power-requiring outcomes via the reward function parameter setting. Insofar as this speculation holds, our theory predicts that advanced reinforcement learning at scale will—for most settings of the reward function—train policies which tend to seek power. ## 6 Discussion In section 3 , we formalized a notion of parametric retargetability and stated several key results. While our results are broadly applicable, further work is required to understand the implications for ai . ### 6.1 Prior work In this work, we do not motivate the risks from ai power-seeking. We refer the reader to e . g ., Carlsmith [2021] . As explained in section 5.1 , Turner et al. [2021] show that, given certain environmental symmetries in an mdp , the optimal-policy-producing algorithm f f (state visitation distribution set, state-based reward function) is 1-retargetable via the reward function, from smaller to larger sets of environmental options. Appendix A shows that optimality is not required, and instead a wide range of decision-making procedures satisfy the retargetability criterion. Furthermore, we generalize from 1-retargetability to n n -fold-retargetability whenever option set B B contains “ n n copies” of set A A ( definition A.7 in appendix A ). ### 6.2 Future work and limitations We currently have analyzed planning- and reinforcement learning-based settings. However, results such as Theorem 3.6 might in some way apply to the training of other machine learning networks. Furthermore, while Theorem 3.6 does not assume a finite environment, we currently do not see how to apply that result to e . g ., infinite-state partially observable Markov decision processes. Section 4 semi-formally analyzes decision-making incentives in the mr video game, leaving the proofs to appendix C . However, these proofs are several pages long. Perhaps additional lemmas can allow quick proof of orbit-level incentives in situations relevant to real-world decision-makers. Consider a sequence of decision-making functions p t : { A , B } × Θ → ℝ p_{t}\mathrel{\mathop{\ordinarycolon}}\{A,B\}\times\Theta\to\mathbb{R} which converges pointwise to some p p such that p ( B ∣ θ ) ≥ most : Θ n p ( A ∣ θ ) p(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p(A\mid\theta) . We expect that under rather mild conditions, ∃ T : ∀ t ≥ T : p t ( B ∣ θ ) ≥ most : Θ n p t ( A ∣ θ ) \exists T\mathrel{\mathop{\ordinarycolon}}\forall t\geq T\mathrel{\mathop{\ordinarycolon}}p_{t}(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{t}(A\mid\theta) . As a corollary, for any decision-making procedure p t p_{t} which runs for t t time steps and satisfies lim t → ∞ p t = p \lim_{t\to\infty}p_{t}=p , the function p t p_{t} will have decision-making incentives after finite time. For example, value iteration ( vi ) eventually finds an optimal policy [ Puterman, 2014 ] , and optimal policies tend to seek power [ Turner et al., 2021 ] . Therefore, this conjecture would imply that if vi is run for some long but finite time, it tends to produce power-seeking policies. More interestingly, the result would allow us to reason about the effect of e . g ., randomly initializing parameters (in vi , the tabular value function at t = 0 t=0 ). The effect of random initialization washes out in the limit of infinite time, so we would still conclude the presence of finite-time power-seeking incentives. Our results do not prove that we will build unaligned ai agents which seek power over the world. Here are a few situations in which our results are not concerning or not applicable. 1. The ai is aligned with human interests. For example, we want a robotic cartographer to prevent itself from being deactivated. However, the ai alignment problem is not yet understood for highly intelligent agents [ Russell, 2019 ] . 2. The ai ’s decision-making is not retargetable ( definition 3.5 ). 3. The ai ’s decision-making is retargetable over e . g ., actions ( section 4.1 ) instead of over final outcomes ( section 4.2 ). This retargetability seems less concerning, but also less practically useful. ### 6.3 Conclusion We introduced the concept of retargetability and showed that retargetable decision-makers often make similar instrumental choices. We applied these results in the Montezuma’s Revenge ( mr ) video game, showing how increasingly advanced reinforcement learning algorithms correspond to increasingly retargetable agent decision-making. Increasingly retargetable agents make increasingly similar instrumental decisions— e . g ., leaving the initial room in mr , or staying alive in Pac-Man. In particular, these decisions will often correspond to gaining power and keeping options open [ Turner et al., 2021 ] . Our theory suggests that when rl training processes become sufficiently advanced, the trained agents will tend to seek power over the world. This theory suggests a safety risk. We hope for future work on this theory so that the field of ai can understand the relevant safety risks before the field trains power-seeking agents. ## Broader impacts Our theory of orbit-level tendencies constitutes basic mathematical research into the decision-making tendencies of certain kinds of agents. We hope that this theory will prevent negative impacts from unaligned power-seeking ai . We do not anticipate that our work will have negative impact. ## Acknowledgements We thank Irene Tematelewo, Colin Shea-Blymyer, and our anonymous reviewers for feedback. We thank Justis Mills for proofreading. ## Appendix A Retargetability over outcome lotteries Suppose we are interested in d d outcomes. Each outcome could be the visitation of an mdp state, or a trajectory, or the receipt of a physical item. In the Pac-Man example of section 2 , d = 3 d=3 states. The agent can induce each outcome with probability 1 1 , so let 𝐞 o ∈ ℝ 3 \mathbf{e}_{o}\in\mathbb{R}^{3} be the standard basis vector with probability 1 1 on outcome o o and 0 0 elsewhere. Then the agent chooses among outcome lotteries C ≔ { 𝐞 , 𝐞 , 𝐞 } C\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} , which we partition into A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} and B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} . ###### Definition A.1 (Outcome lotteries) . A unit vector 𝐱 ∈ ℝ d \mathbf{x}\in\mathbb{R}^{d} with non-negative entries is an outcome lottery . 7 7 7 Our results on outcome lotteries hold for generic 𝐱 ′ ∈ ℝ d \mathbf{x}^{\prime}\in\mathbb{R}^{d} , but we find it conceptually helpful to consider the non-negative unit vector case. Many decisions are made consequentially: based on the consequences of the decision, on what outcomes are brought about by an act. For example, in a deterministic Atari game, a policy induces a trajectory. A reward function and discount rate tuple ( R , γ ) (R,\gamma) assigns a return to each state trajectory τ = s 0 , s 1 , … \tau=s_{0},s_{1},\ldots : G ⁡ ( τ ) = ∑ i = 0 ∞ γ i ​ R ​ ( s i ) G(\tau)=\sum_{i=0}^{\infty}\gamma^{i}R(s_{i}) . The relevant outcome lottery is the discounted visit distribution over future states in an Atari game, and policies are optimal or not depending on which outcome lottery is induced by the policy. ###### Definition A.2 (Optimality indicator function) . Let X , C ⊊ ℝ d X,C\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . IsOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{IsOptimal}\left(X\mid C,\mathbf{u}\right) returns 1 1 if max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ≥ max 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 \max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}\geq\max_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u} , and 0 0 otherwise. We consider decision-making procedures which take in a targeting parameter 𝐮 \mathbf{u} . For example, the column headers of Table 2(a) show the 6 permutations of the utility function u ⁡ ( ) ≔ 10 , u ⁡ ( ) ≔ 5 , u ⁡ ( ) ≔ 0 u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf})\coloneqq 10,u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf})\coloneqq 5,u(\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf})\coloneqq 0 , representable as a vector 𝐮 ∈ ℝ 3 \mathbf{u}\in\mathbb{R}^{3} . 𝐮 \mathbf{u} can be permuted as follows. The outcome permutation ϕ ∈ S d \phi\in S_{d} inducing an d × d d\times d permutation matrix 𝐏 ϕ \mathbf{P}_{\phi} in row representation: ( 𝐏 ϕ ) i ​ j = 1 (\mathbf{P}_{\phi})_{ij}=1 if i = ϕ ⁡ ( j ) i=\phi(j) and 0 0 otherwise. Table 2(a) shows that for a given utility function, 2 3 \frac{2}{3} of its orbit agrees that B B is strictly optimal over A A . Orbit-level incentives occur when an inequality holds for most permuted parameter choices 𝐮 ′ \mathbf{u}^{\prime} . Table 2(a) demonstrates an application of Turner et al. [2021] ’s results: Optimal decision-making induces orbit-level incentives for choosing Pac-Man outcomes in B B over outcomes in A A . Furthermore, Turner et al. [2021] conjectured that “larger” B B will imply stronger orbit-level tendencies: If going right leads to 500 times as many options as going left, then right is better than left for at least 500 times as many reward functions for which the opposite is true. We prove this conjecture with Theorem D.11 in appendix D . However, orbit-level incentives do not require optimality. One clue is that the same results hold for anti-optimal agents, since anti-optimality/utility minimization of 𝐮 \mathbf{u} is equivalent to maximizing − 𝐮 -\mathbf{u} . Table 2(b) illustrates that the same orbit guarantees hold in this case. ###### Definition A.3 (Anti-optimality indicator function) . Let X , C ⊊ ℝ d X,C\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . AntiOpt ⁡ ( X ∣ C , 𝐮 ) \mathrm{AntiOpt}\left(X\mid C,\mathbf{u}\right) returns 1 1 if min 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ≤ min 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 \min_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}\leq\min_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u} , and 0 0 otherwise. Stepping beyond expected utility maximization/minimization, Boltzmann-rational decision-making selects outcome lotteries proportional to the exponential of their expected utility. ###### Definition A.4 (Boltzmann rationality [ Baker et al., 2007 ] ) . For X ⊆ C X\subseteq C and temperature T > 0 T>0 , let Boltzmann T ​ ( X ∣ C , 𝐮 ) ≔ ∑ 𝐱 ∈ X e T − 1 ​ 𝐱 ⊤ ​ 𝐮 ∑ 𝐜 ∈ C e T − 1 ​ 𝐜 ⊤ ​ 𝐮 \mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right)\coloneqq\frac{\sum_{\mathbf{x}\in X}e^{T^{-1}\mathbf{x}^{\top}\mathbf{u}}}{\sum_{\mathbf{c}\in C}e^{T^{-1}\mathbf{c}^{\top}\mathbf{u}}} be the probability that some element of X X is Boltzmann-rational. Lastly, orbit-level tendencies occur even under decision-making procedures which partially ignore expected utility and which “don’t optimize too hard.” Satisficing agents randomly choose an outcome lottery with expected utility exceeding some threshold. Table 2(d) demonstrates that satisficing induces orbit-level tendencies. ###### Definition A.5 (Satisficing) . Let t ∈ ℝ t\in\mathbb{R} , let X ⊆ C ⊊ ℝ d X\subseteq C\subsetneq\mathbb{R}^{d} be finite. Satisfice t ​ ( X , C ∣ 𝐮 ) ≔ | X ∩ { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 ≥ t } | | { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 ≥ t } | \mathrm{Satisfice}_{t}\left(X,C\mid\mathbf{u}\right)\coloneqq\frac{\left|X\cap\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}\geq t\right\}\right|}{\left|\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}\geq t\right\}\right|} is the fraction of X X whose value exceeds threshold t t . Satisfice t ​ ( X , C ∣ 𝐮 ) \mathrm{Satisfice}_{t}\left(X,C\mid\mathbf{u}\right) evaluates to 0 0 the denominator equals 0 0 . For each table, two-thirds of the utility permutations (columns) assign strictly larger values (shaded dark gray) to an element of B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} than to an element of A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} . For optimal, anti-optimal, Boltzmann-rational, and satisficing agents, Proposition A.11 proves that these tendencies hold for all targeting parameter orbits. ### A.1 A range of decision-making functions are retargetable In mdp s, Turner et al. [2021] consider state visitation distributions which record the total discounted time steps spent in each environment state, given that the agent follows some policy π \pi from an initial state s s . These visitation distributions are one kind of outcome lottery, with d = | 𝒮 | d=\left|\mathcal{S}\right| the number of mdp states. In general, we suppose the agent has an objective function 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} which maps outcomes to real numbers. In Turner et al. [2021] , 𝐮 \mathbf{u} was a state-based reward function (and so the outcomes were states ). However, we need not restrict ourselves to the mdp setting. To state our key results, we define several technical concepts which we informally used when reasoning about A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} and B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} . ###### Definition A.6 (Similarity of vector sets) . For ϕ ∈ S d \phi\in S_{d} and X ⊆ ℝ d X\subseteq\mathbb{R}^{d} , ϕ ⋅ X ≔ { 𝐏 ϕ ​ 𝐱 ∣ 𝐱 ∈ X } \phi\cdot X\coloneqq\left\{\mathbf{P}_{\phi}\mathbf{x}\mid\mathbf{x}\in X\right\} . X ′ ⊆ ℝ | 𝒮 | X^{\prime}\subseteq\mathbb{R}^{\left|\mathcal{S}\right|} is similar to X X when ∃ ϕ : ϕ ⋅ X ′ = X \exists\phi\mathrel{\mathop{\ordinarycolon}}\phi\cdot X^{\prime}=X . ϕ \phi is an involution if ϕ = ϕ − 1 \phi=\phi^{-1} (it either transposes states, or fixes them). X X contains a copy of X ′ X^{\prime} when X ′ X^{\prime} is similar to a subset of X X via an involution ϕ \phi . ###### Definition A.7 (Containment of set copies) . Let n n be a positive integer, and let A , B ⊆ ℝ d A,B\subseteq\mathbb{R}^{d} . We say that B B contains n n copies of A A when there exist involutions ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} such that ∀ i : ϕ i ⋅ A ≕ B i ⊆ B \forall i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot A\eqqcolon B_{i}\subseteq B and ∀ j ≠ i : ϕ i ⋅ B j = B j \forall j\neq i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot B_{j}=B_{j} . 8 8 8 Technically, definition A.7 implies that A A contains n n copies of A A holds for all n n , via n n applications of the identity permutation. For our purposes, this provides greater generality, as all of the relevant results still hold. Enforcing pairwise disjointness of the B i B_{i} would handle these issues, but would narrow our results to not apply e . g ., when the B i B_{i} share a constant vector. B ≔ { 𝐞 , 𝐞 } B\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf}},\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf}}\right\} contains two copies of A ≔ { 𝐞 } A\coloneqq\left\{\mathbf{e}_{\hskip 0.0pt\includegraphics[height=5.1819pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}}\right\} via ϕ 1 ≔ ↔ \phi_{1}\coloneqq\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/apple.pdf} and ϕ 2 ≔ ↔ \phi_{2}\coloneqq\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/ghost.pdf}\leftrightarrow\hskip 0.0pt\includegraphics[height=6.49994pt,clip,trim=0.0pt 0.0pt 0.0pt 0.0pt]{quantitative/assets/sprites/cherry.pdf} . ###### Definition A.8 (Targeting parameter distribution assumptions) . Results with 𝒟 any \mathcal{D}_{\text{any}} hold for any probability distribution over ℝ d \mathbb{R}^{d} . Let 𝔇 any ≔ Δ ⁡ ( ℝ d ) \mathfrak{D}_{\text{any}}\coloneqq\Delta(\mathbb{R}^{d}) . For a function f : ℝ d ↦ ℝ f\mathrel{\mathop{\ordinarycolon}}\mathbb{R}^{d}\mapsto\mathbb{R} , we write f ⁡ ( 𝒟 any ) f(\mathcal{D}_{\text{any}}) as shorthand for 𝔼 𝐮 ∼ 𝒟 any [ f ⁡ ( 𝐮 ) ] \opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[f(\mathbf{u})\right] . The symmetry group on d d elements, S d S_{d} , acts on the set of probability distributions over ℝ d \mathbb{R}^{d} . ###### Definition A.9 (Pushforward distribution of a permutation [ Turner et al., 2021 ] ) . Let ϕ ∈ S d \phi\in S_{d} . ϕ ⋅ 𝒟 any \phi\cdot\mathcal{D}_{\text{any}} is the pushforward distribution induced by applying the random vector p ⁡ ( 𝐮 ) ≔ 𝐏 ϕ ​ 𝐮 p(\mathbf{u})\coloneqq\mathbf{P}_{\phi}\mathbf{u} to 𝒟 any \mathcal{D}_{\text{any}} . ###### Definition A.10 (Orbit of a probability distribution [ Turner et al., 2021 ] ) . The orbit of 𝒟 any \mathcal{D}_{\text{any}} under the symmetric group S d S_{d} is S d ⋅ 𝒟 any ≔ { ϕ ⋅ 𝒟 any ∣ ϕ ∈ S d } S_{d}\cdot\mathcal{D}_{\text{any}}\coloneqq\{\phi\cdot\mathcal{D}_{\text{any}}\mid\phi\in S_{d}\} . Because B B contains 2 copies of A A , there are “at least two times as many ways” for B B to be optimal, than for A A to be optimal. Similarly, B B is “at least two times as likely” to contain an anti-rational outcome lottery for generic utility functions. As demonstrated by Table 2 , the key idea is that “larger” sets (a set B B containing several copies of set A A ) are more likely to be chosen under a wide range of decision-making criteria. ###### Proposition A.11 (Orbit incentives for different rationalities) . Let A , B ⊆ C ⊊ ℝ d A,B\subseteq C\subsetneq\mathbb{R}^{d} be finite, such that B B contains n n copies of A A via involutions ϕ i \phi_{i} such that ϕ i ⋅ C = C \phi_{i}\cdot C=C . 1. Rational choice [ Turner et al., 2021 ] . IsOptimal ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n IsOptimal ( A ∣ C , 𝒟 any ) . \mathrm{IsOptimal}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{IsOptimal}\left(A\mid C,\mathcal{D}_{\text{any}}\right). 2. Uniformly randomly choosing an optimal lottery. For X ⊆ C X\subseteq C , let FracOptimal ⁡ ( X ∣ C , 𝐮 ) ≔ | { arg ​ max 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 } ∩ X | | { arg ​ max 𝐜 ∈ C ⁡ 𝐜 ⊤ ​ 𝐮 } | . \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right)\coloneqq\frac{\left|\left\{\argmax_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u}\right\}\cap X\right|}{\left|\left\{\argmax_{\mathbf{c}\in C}\mathbf{c}^{\top}\mathbf{u}\right\}\right|}. Then FracOptimal ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n FracOptimal ( A ∣ C , 𝒟 any ) \mathrm{FracOptimal}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{FracOptimal}\left(A\mid C,\mathcal{D}_{\text{any}}\right) . 3. Anti-rational choice. AntiOpt ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n AntiOpt ( A ∣ C , 𝒟 any ) \mathrm{AntiOpt}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{AntiOpt}\left(A\mid C,\mathcal{D}_{\text{any}}\right) . 4. Boltzmann rationality. Boltzmann T ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n Boltzmann T ( A ∣ C , 𝒟 any ) . \mathrm{Boltzmann}_{T}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{Boltzmann}_{T}\left(A\mid C,\mathcal{D}_{\text{any}}\right). 5. Uniformly randomly drawing k k outcome lotteries and choosing the best. For X ⊆ C X\subseteq C , 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} , and k ≥ 1 k\geq 1 , let best-of- ​ k ​ ( X , C ∣ 𝐮 ) ≔ 𝔼 𝐚 1 , … , 𝐚 k ∼ unif ​ ( C ) [ FracOptimal ⁡ ( X ∩ { 𝐚 1 , … , 𝐚 k } ∣ { 𝐚 1 , … , 𝐚 k } , 𝐮 ) ] . \textrm{best-of-}k(X,C\mid\mathbf{u})\coloneqq\opE_{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\sim\text{unif}(C)}\left[\mathrm{FracOptimal}\left(X\cap\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\}\mid\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\},\mathbf{u}\right)\right]. Then best-of- k ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n best-of- k ( A ∣ C , 𝒟 any ) \textrm{best-of-}k(B\mid C,\mathcal{D}_{\text{any}})\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\textrm{best-of-}k(A\mid C,\mathcal{D}_{\text{any}}) . 6. Satisficing [ Simon, 1956 ] . Satisfice t ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n Satisfice t ( A ∣ C , 𝒟 any ) \mathrm{Satisfice}_{t}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{Satisfice}_{t}\left(A\mid C,\mathcal{D}_{\text{any}}\right) . 7. Quantilizing over outcome lotteries [ Taylor, 2016 ] . Let P P be the uniform probability distribution over C C . For X ⊆ C X\subseteq C , 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} , and q ∈ ( 0 , 1 ] q\in(0,1] , let Q q , P ​ ( X ∣ C , 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u}) ( definition B.12 ) return the probability that an outcome lottery in X X is drawn from the top q q -quantile of P P , sorted by expected utility under 𝐮 \mathbf{u} . Then Q q , P ( B ∣ C , 𝐮 ) ≥ most : ℝ d n Q q , P ( A ∣ C , 𝐮 ) Q_{q,P}(B\mid C,\mathbf{u})\geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n}Q_{q,P}(A\mid C,\mathbf{u}) . One retargetable class of decision-making functions are those which only account for the expected utilities of available choices. ###### Definition A.12 (EU-determined functions) . Let 𝒫 ⁡ ( ℝ d ) \mathcal{P}\left(\mathbb{R}^{d}\right) be the power set of ℝ d \mathbb{R}^{d} , and let f : ∏ i = 1 m 𝒫 ⁡ ( ℝ d ) × ℝ d → ℝ f\mathrel{\mathop{\ordinarycolon}}\prod_{i=1}^{m}\mathcal{P}\left(\mathbb{R}^{d}\right)\times\mathbb{R}^{d}\to\mathbb{R} . f f is an EU-determined function if there exists a family of functions { g ω 1 , … , ω m } \left\{g^{\omega_{1},\ldots,\omega_{m}}\right\} such that f ⁡ ( X 1 , … , X m ∣ 𝐮 ) = g | X 1 | , … , | X m | ​ ( [ 𝐱 1 ⊤ ​ 𝐮 ] 𝐱 1 ∈ X 1 , … , [ 𝐱 m ⊤ ​ 𝐮 ] 𝐱 m ∈ X m ) , f(X_{1},\ldots,X_{m}\mid\mathbf{u})=g^{|X_{1}|,\ldots,|X_{m}|}\left(\left[\mathbf{x}_{1}^{\top}\mathbf{u}\right]_{\mathbf{x}_{1}\in X_{1}},\ldots,\left[\mathbf{x}_{m}^{\top}\mathbf{u}\right]_{\mathbf{x}_{m}\in X_{m}}\right), (4) where [ r i ] [r_{i}] is the multiset of its elements r i r_{i} . For example, let X ⊆ C ⊊ ℝ d X\subseteq C\subsetneq\mathbb{R}^{d} be finite, and consider utility function 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . A Boltzmann-rational agent is more likely to select outcome lotteries with greater expected utility. Formally, Boltzmann T ​ ( X ∣ C , 𝐮 ) ≔ ∑ 𝐱 ∈ X e T ⋅ 𝐱 ⊤ ​ 𝐮 ∑ 𝐜 ∈ C e T ⋅ 𝐜 ⊤ ​ 𝐮 \mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right)\coloneqq\sum_{\mathbf{x}\in X}\frac{e^{T\cdot\mathbf{x}^{\top}\mathbf{u}}}{\sum_{\mathbf{c}\in C}e^{T\cdot\mathbf{c}^{\top}\mathbf{u}}} depends only on the expected utility of outcome lotteries in X X , relative to the expected utility of all outcome lotteries in C C . Therefore, Boltzmann T \mathrm{Boltzmann}_{T} is a function of expected utilities. This is why Boltzmann T \mathrm{Boltzmann}_{T} satisfies the ≥ most : 𝔇 any n \geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n} relation. ###### Theorem A.13 (Orbit tendencies occur for EU-determined decision-making functions) . Let A , B , C ⊆ ℝ d A,B,C\subseteq\mathbb{R}^{d} be such that B B contains n n copies of A A via ϕ i \phi_{i} such that ϕ i ⋅ C = C \phi_{i}\cdot C=C . Let h : ∏ i = 1 2 𝒫 ⁡ ( ℝ d ) × ℝ d → ℝ h\mathrel{\mathop{\ordinarycolon}}\prod_{i=1}^{2}\mathcal{P}\left(\mathbb{R}^{d}\right)\times\mathbb{R}^{d}\to\mathbb{R} be an EU-determined function, and let p ⁡ ( X ∣ 𝐮 ) ≔ h ⁡ ( X , C ∣ 𝐮 ) p(X\mid\mathbf{u})\coloneqq h(X,C\mid\mathbf{u}) . Suppose that p p returns a probability of selecting an element of X X from C C . Then p ( B ∣ 𝐮 ) ≥ most : ℝ d n p ( A ∣ 𝐮 ) p(B\mid\mathbf{u})\geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n}p(A\mid\mathbf{u}) . The key takeaway is that decisions which are determined by expected utility are straightforwardly retargetable. By changing the targeting parameter hyperparameter, the decision-making procedure can be flexibly retargeted to choose elements of “larger” sets (in terms of set copies via definition A.7 ). Less abstractly, for many agent rationalities—ways of making decisions over outcome lotteries—it is generally the case that larger sets will more often be chosen over smaller sets. For example, consider a Pac-Man playing agent choosing which environmental state cycle it should end up in. Turner et al. [2021] show that for most reward functions, average-reward maximizing agents will tend to stay alive so that they can reach a wider range of environmental cycles. However, our results show that average-reward minimizing agents also exhibit this tendency, as do Boltzmann-rational agents who assign greater probability to higher-reward cycles. Any EU-based cycle selection method will—for most reward functions—tend to choose cycles which require Pac-Man to stay alive (at first). ## Appendix B Theoretical results See 3.2 ###### Remark . In stating their equivalent of definition 3.2 , Turner et al. [2021] define two functions f 1 ​ ( θ ) ≔ f ⁡ ( B ∣ θ ) f_{1}(\theta)\coloneqq f(B\mid\theta) and f 2 ​ ( θ ) ≔ f ⁡ ( A ∣ θ ) f_{2}(\theta)\coloneqq f(A\mid\theta) (both having type signature f i : Θ → ℝ f_{i}\mathrel{\mathop{\ordinarycolon}}\Theta\to\mathbb{R} ). For compatibility, proofs also use this notation. ###### Lemma B.1 (Limited transitivity of ≥ most \geq_{\text{most}} ) . Let f 0 , f 1 , f 2 , f 3 : Θ → ℝ f_{0},f_{1},f_{2},f_{3}\mathrel{\mathop{\ordinarycolon}}\Theta\to\mathbb{R} , and suppose Θ \Theta is a subset of a set acted on by S d S_{d} . Suppose that f 1 ( θ ) ≥ most : Θ n f 2 ( θ ) f_{1}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{2}(\theta) and ∀ θ ∈ Θ : f 0 ​ ( θ ) ≥ f 1 ​ ( θ ) \forall\theta\in\Theta\mathrel{\mathop{\ordinarycolon}}f_{0}(\theta)\geq f_{1}(\theta) and f 2 ​ ( θ ) ≥ f 3 ​ ( θ ) f_{2}(\theta)\geq f_{3}(\theta) . Then f 0 ( θ ) ≥ most : Θ n f 3 ( θ ) f_{0}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{3}(\theta) . ###### Proof. Let θ ∈ Θ \theta\in\Theta and let Orbit | Θ , f a > f b ​ ( θ ) ≔ { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f a ​ ( θ ′ ) > f b ​ ( θ ′ ) } \mathrm{Orbit}|_{\Theta,f_{a}>f_{b}}\left(\theta\right)\coloneqq\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f_{a}(\theta^{\prime})>f_{b}(\theta^{\prime})\right\} . | Orbit | Θ , f 0 > f 3 ​ ( θ ) | \displaystyle\left|\mathrm{Orbit}|_{\Theta,f_{0}>f_{3}}\left(\theta\right)\right| ≥ | Orbit | Θ , f 1 > f 2 ​ ( θ ) | \displaystyle\geq\left|\mathrm{Orbit}|_{\Theta,f_{1}>f_{2}}\left(\theta\right)\right| (5) ≥ n ​ | Orbit | Θ , f 2 > f 1 ​ ( θ ) | \displaystyle\geq n\left|\mathrm{Orbit}|_{\Theta,f_{2}>f_{1}}\left(\theta\right)\right| (6) ≥ n ​ | Orbit | Θ , f 3 > f 0 ​ ( θ ) | . \displaystyle\geq n\left|\mathrm{Orbit}|_{\Theta,f_{3}>f_{0}}\left(\theta\right)\right|. (7) For all θ ′ ∈ Orbit | Θ , f 1 > f 2 ​ ( θ ) \theta^{\prime}\in\mathrm{Orbit}|_{\Theta,f_{1}>f_{2}}\left(\theta\right) , f 0 ​ ( θ ′ ) ≥ f 1 ​ ( θ ′ ) > f 2 ​ ( θ ′ ) ≥ f 3 ​ ( θ ′ ) f_{0}(\theta^{\prime})\geq f_{1}(\theta^{\prime})>f_{2}(\theta^{\prime})\geq f_{3}(\theta^{\prime}) by assumption, and so Orbit | Θ , f 1 > f 2 ​ ( θ ) ⊆ Orbit | Θ , f 0 > f 3 ​ ( θ ) . \mathrm{Orbit}|_{\Theta,f_{1}>f_{2}}\left(\theta\right)\subseteq\mathrm{Orbit}|_{\Theta,f_{0}>f_{3}}\left(\theta\right). Therefore, eq. 5 follows. By assumption, | Orbit | Θ , f 1 > f 2 ​ ( θ ) | ≥ n ​ | Orbit | Θ , f 2 > f 1 ​ ( θ ) | ; \left|\mathrm{Orbit}|_{\Theta,f_{1}>f_{2}}\left(\theta\right)\right|\geq n\left|\mathrm{Orbit}|_{\Theta,f_{2}>f_{1}}\left(\theta\right)\right|; eq. 6 follows. For all θ ′ ∈ Orbit | Θ , f 2 > f 1 ​ ( θ ) \theta^{\prime}\in\mathrm{Orbit}|_{\Theta,f_{2}>f_{1}}\left(\theta\right) , our assumptions on f 0 f_{0} and f 3 f_{3} ensure that f 0 ​ ( θ ′ ) ≤ f 1 ​ ( θ ′ ) < f 3 ​ ( θ ′ ) ≤ f 2 ​ ( θ ′ ) , f_{0}(\theta^{\prime})\leq f_{1}(\theta^{\prime}) f 0 ​ ( θ ) ⊆ Orbit | Θ , f 2 > f 1 ​ ( θ ) . \mathrm{Orbit}|_{\Theta,f_{3}>f_{0}}\left(\theta\right)\subseteq\mathrm{Orbit}|_{\Theta,f_{2}>f_{1}}\left(\theta\right). Then eq. 7 follows. By eq. 7 , f 0 ( θ ) ≥ most : Θ n f 3 ( θ ) f_{0}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{3}(\theta) . ∎ ###### Lemma B.2 (Order inversion for ≥ most \geq_{\text{most}} ) . Let f 1 , f 2 : Θ → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Theta\to\mathbb{R} , and suppose Θ \Theta is a subset of a set acted on by S d S_{d} . Suppose that f 1 ( θ ) ≥ most : Θ n f 2 ( θ ) f_{1}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{2}(\theta) . Then − f 2 ( θ ) ≥ most : Θ n − f 1 ( θ ) -f_{2}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}-f_{1}(\theta) . ###### Proof. By definition A.10 , f 1 ( θ ) ≥ most : Θ n f 2 ( θ ) f_{1}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{2}(\theta) means that | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f 1 ​ ( θ ′ ) > f 2 ​ ( θ ′ ) } | \displaystyle\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f_{1}(\theta^{\prime})>f_{2}(\theta^{\prime})\right\}\right| ≥ n ​ | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ f 1 ​ ( θ ′ ) < f 2 ​ ( θ ′ ) } | \displaystyle\geq n\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid f_{1}(\theta^{\prime}) − f 1 ​ ( θ ′ ) } | \displaystyle\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid-f_{2}(\theta^{\prime})>-f_{1}(\theta^{\prime})\right\}\right| ≥ n ​ | { θ ′ ∈ Orbit | Θ ​ ( θ ) ∣ − f 2 ​ ( θ ′ ) < − f 1 ​ ( θ ′ ) } | . \displaystyle\geq n\left|\left\{\theta^{\prime}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right)\mid-f_{2}(\theta^{\prime})<-f_{1}(\theta^{\prime})\right\}\right|. (9) Then − f 2 ( θ ) ≥ most : Θ n − f 1 ( θ ) -f_{2}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}-f_{1}(\theta) . ∎ ###### Remark . Lemma B.3 generalizes Turner et al. [2021] ’s lemma B.2. ###### Lemma B.3 (Orbital fraction which agrees on (weak) inequality) . Suppose f 1 , f 2 : Θ → ℝ f_{1},f_{2}\mathrel{\mathop{\ordinarycolon}}\Theta\to\mathbb{R} are such that f 1 ( θ ) ≥ most : Θ n f 2 ( θ ) f_{1}(\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f_{2}(\theta) . Then for all θ ∈ Θ \theta\in\Theta , | { θ ′ ∈ ( S d ⋅ θ ) ∩ Θ ∣ f 1 ​ ( θ ′ ) ≥ f 2 ​ ( θ ′ ) } | | ( S d ⋅ θ ) ∩ Θ | ≥ n n + 1 \frac{\left|\left\{\theta^{\prime}\in\left(S_{d}\cdot\theta\right)\cap\Theta\mid f_{1}(\theta^{\prime})\geq f_{2}(\theta^{\prime})\right\}\right|}{\left|\left(S_{d}\cdot\theta\right)\cap\Theta\right|}\geq\dfrac{n}{n+1} . ###### Proof. All θ ′ ∈ ( S d ⋅ θ ) ∩ Θ \theta^{\prime}\in\left(S_{d}\cdot\theta\right)\cap\Theta such that f 1 ​ ( θ ′ ) = f 2 ​ ( θ ′ ) f_{1}(\theta^{\prime})=f_{2}(\theta^{\prime}) satisfy f 1 ​ ( θ ′ ) ≥ f 2 ​ ( θ ′ ) f_{1}(\theta^{\prime})\geq f_{2}(\theta^{\prime}) . Otherwise, consider the θ ′ ∈ ( S d ⋅ θ ) ∩ Θ \theta^{\prime}\in\left(S_{d}\cdot\theta\right)\cap\Theta such that f 1 ​ ( θ ′ ) ≠ f 2 ​ ( θ ′ ) f_{1}(\theta^{\prime})\neq f_{2}(\theta^{\prime}) . By assumption, at least n n + 1 \frac{n}{n+1} of these θ ′ \theta^{\prime} satisfy f 1 ​ ( θ ′ ) > f 2 ​ ( θ ′ ) f_{1}(\theta^{\prime})>f_{2}(\theta^{\prime}) , in which case f 1 ​ ( θ ′ ) ≥ f 2 ​ ( θ ′ ) f_{1}(\theta^{\prime})\geq f_{2}(\theta^{\prime}) . Then the desired inequality follows. ∎ ### B.1 General results on retargetable functions ###### Definition B.4 (Functions which are increasing under joint permutation) . Suppose that S d S_{d} acts on sets 𝐄 1 , … , 𝐄 m \mathbf{E}_{1},\ldots,\mathbf{E}_{m} , and let f : ∏ i = 1 m 𝐄 i → ℝ f\mathrel{\mathop{\ordinarycolon}}\prod_{i=1}^{m}\mathbf{E}_{i}\to\mathbb{R} . f ⁡ ( X 1 , … , X m ) f(X_{1},\ldots,X_{m}) is increasing under joint permutation by P ⊆ S d P\subseteq S_{d} when ∀ ϕ ∈ P : f ⁡ ( X 1 , … , X m ) ≤ f ⁡ ( ϕ ⋅ X 1 , … , ϕ ⋅ X m ) \forall\phi\in P\mathrel{\mathop{\ordinarycolon}}f(X_{1},\ldots,X_{m})\leq f(\phi\cdot X_{1},\ldots,\phi\cdot X_{m}) . If equality always holds, then f ⁡ ( X 1 , … , X m ) f(X_{1},\ldots,X_{m}) is invariant under joint permutation by P P . ###### Lemma B.5 (Expectations of joint-permutation-increasing functions are also joint-permutation-increasing) . For 𝐄 \mathbf{E} which is a subset of a set acted on by S d S_{d} , let f : 𝐄 × ℝ d → ℝ f\mathrel{\mathop{\ordinarycolon}}\mathbf{E}\times\mathbb{R}^{d}\to\mathbb{R} be a bounded function which is measurable on its second argument, and let P ⊆ S d P\subseteq S_{d} . Then if f ⁡ ( X ∣ 𝐮 ) f(X\mid\mathbf{u}) is increasing under joint permutation by P P , then f ′ ​ ( X ∣ 𝒟 any ) ≔ 𝔼 𝐮 ∼ 𝒟 any [ f ⁡ ( X ∣ 𝐮 ) ] f^{\prime}(X\mid\mathcal{D}_{\text{any}})\coloneqq\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[f(X\mid\mathbf{u})\right] is increasing under joint permutation by P P . If f f is invariant under joint permutation by P P , then so is f ′ f^{\prime} . ###### Proof. Let distribution 𝒟 any \mathcal{D}_{\text{any}} have probability measure F F , and let ϕ ⋅ 𝒟 any \phi\cdot\mathcal{D}_{\text{any}} have probability measure F ϕ F_{\phi} . f ⁡ ( X ∣ 𝒟 any ) ≔ \displaystyle f\left(X\mid\mathcal{D}_{\text{any}}\right)\coloneqq{} 𝔼 𝐮 ∼ 𝒟 any [ f ⁡ ( X ∣ 𝐮 ) ] \displaystyle\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[f(X\mid\mathbf{u})\right] (10) ≔ \displaystyle\coloneqq{} ∫ ℝ d f ⁡ ( X ∣ 𝐮 ) ​ d ⁡ F ⁡ ( 𝐮 ) \displaystyle\int_{\mathbb{R}^{d}}f(X\mid\mathbf{u})\dif F(\mathbf{u}) (11) ≤ \displaystyle\leq{} ∫ ℝ d f ⁡ ( ϕ ⋅ X ∣ 𝐏 ϕ ​ 𝐮 ) ​ d ⁡ F ⁡ ( 𝐮 ) \displaystyle\int_{\mathbb{R}^{d}}f(\phi\cdot X\mid\mathbf{P}_{\phi}\mathbf{u})\dif F(\mathbf{u}) (12) = \displaystyle={} ∫ ℝ d f ⁡ ( ϕ ⋅ X ∣ 𝐮 ′ ) ​ | det 𝐏 ϕ | ​ d ⁡ F ϕ ​ ( 𝐮 ′ ) \displaystyle\int_{\mathbb{R}^{d}}f(\phi\cdot X\mid\mathbf{u}^{\prime})\left|\det\mathbf{P}_{\phi}\right|\dif F_{\phi}(\mathbf{u}^{\prime}) (13) = \displaystyle={} ∫ ℝ d f ⁡ ( ϕ ⋅ X ∣ 𝐮 ′ ) ​ d ⁡ F ϕ ​ ( 𝐮 ′ ) \displaystyle\int_{\mathbb{R}^{d}}f(\phi\cdot X\mid\mathbf{u}^{\prime})\dif F_{\phi}(\mathbf{u}^{\prime}) (14) ≕ \displaystyle\eqqcolon{} f ′ ​ ( ϕ ⋅ X ∣ ϕ ⋅ 𝒟 any ) . \displaystyle f^{\prime}\left(\phi\cdot X\mid\phi\cdot\mathcal{D}_{\text{any}}\right). (15) Equation 12 holds by assumption on f f : f ⁡ ( X ∣ 𝐮 ) ≤ f ⁡ ( ϕ ⋅ X ∣ 𝐏 ϕ ​ 𝐮 ) f(X\mid\mathbf{u})\leq f(\phi\cdot X\mid\mathbf{P}_{\phi}\mathbf{u}) . Furthermore, f ( ϕ ⋅ X ∣ ⋅ ) f(\phi\cdot X\mid\cdot) is still measurable, and so the inequality holds. Equation 13 follows by the definition of F ϕ F_{\phi} (definition 6.3) and by substituting 𝐫 ′ ≔ 𝐏 ϕ ​ 𝐫 \mathbf{r}^{\prime}\coloneqq\mathbf{P}_{\phi}\mathbf{r} . Equation 14 follows from the fact that all permutation matrices have unitary determinant. ∎ ###### Lemma B.6 (Closure of orbit incentives under increasing functions) . Suppose that S d S_{d} acts on sets 𝐄 1 , … , 𝐄 m \mathbf{E}_{1},\ldots,\mathbf{E}_{m} (with 𝐄 1 \mathbf{E}_{1} being a poset), and let P ⊆ S d P\subseteq S_{d} . Let f 1 , … , f n : ∏ i = 1 m 𝐄 i → ℝ f_{1},\ldots,f_{n}\mathrel{\mathop{\ordinarycolon}}\prod_{i=1}^{m}\mathbf{E}_{i}\to\mathbb{R} be increasing under joint permutation by P P on input ( X 1 , … , X m ) (X_{1},\ldots,X_{m}) , and suppose the f i f_{i} are order-preserving with respect to ⪯ 𝐄 1 \preceq_{\mathbf{E}_{1}} . Let g : ∏ j = 1 n ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\prod_{j=1}^{n}\mathbb{R}\to\mathbb{R} be monotonically increasing on each argument. Then f ⁡ ( X 1 , … , X m ) ≔ g ⁡ ( f 1 ​ ( X 1 , … , X m ) , … , f n ​ ( X 1 , … , X m ) ) f\left(X_{1},\ldots,X_{m}\right)\coloneqq g\left(f_{1}\left(X_{1},\ldots,X_{m}\right),\ldots,f_{n}\left(X_{1},\ldots,X_{m}\right)\right) (16) is increasing under joint permutation by P P and order-preserving with respect to set inclusion on its first argument. Furthermore, if the f i f_{i} are invariant under joint permutation by P P , then so is f f . ###### Proof. Let ϕ ∈ P \phi\in P . f ⁡ ( X 1 , … , X m ) \displaystyle f\left(X_{1},\ldots,X_{m}\right) ≔ g ⁡ ( f 1 ​ ( X 1 , … , X m ) , … , f n ​ ( X 1 , … , X m ) ) \displaystyle\coloneqq g\left(f_{1}\left(X_{1},\ldots,X_{m}\right),\ldots,f_{n}\left(X_{1},\ldots,X_{m}\right)\right) (17) ≤ g ⁡ ( f 1 ​ ( ϕ ⋅ X 1 , … , ϕ ⋅ X m ) , … , f n ​ ( ϕ ⋅ X 1 , … , ϕ ⋅ X m ) ) \displaystyle\leq g\left(f_{1}\left(\phi\cdot X_{1},\ldots,\phi\cdot X_{m}\right),\ldots,f_{n}\left(\phi\cdot X_{1},\ldots,\phi\cdot X_{m}\right)\right) (18) ≕ f ⁡ ( ϕ ⋅ X 1 , … , ϕ ⋅ X m ) . \displaystyle\eqqcolon f\left(\phi\cdot X_{1},\ldots,\phi\cdot X_{m}\right). (19) Equation 18 follows because we assumed that f i ​ ( X 1 , … , X m ) ≤ f i ​ ( ϕ ⋅ X 1 , … , ϕ ⋅ X m ) f_{i}\left(X_{1},\ldots,X_{m}\right)\leq f_{i}\left(\phi\cdot X_{1},\ldots,\phi\cdot X_{m}\right) , and because g g is monotonically increasing on each argument. If the f i f_{i} are all invariant, then eq. 18 is an equality. Similarly, suppose X 1 ′ ⪯ 𝐄 1 X 1 X_{1}^{\prime}\preceq_{\mathbf{E}_{1}}X_{1} . The f i f_{i} are order-preserving on the first argument, and g g is monotonically increasing on each argument. Then f ⁡ ( X 1 ′ , … , X m ) ≤ f ⁡ ( X 1 , … , X m ) f\left(X_{1}^{\prime},\ldots,X_{m}\right)\leq f\left(X_{1},\ldots,X_{m}\right) . This shows that f f is order-preserving on its first argument. ∎ ###### Remark . g g could take the convex combination of its arguments, or multiply two f i f_{i} together and add them to a third f 3 f_{3} . See 3.5 See 3.6 ###### Proof. Let θ ∈ Θ \theta\in\Theta , and let ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) ≔ { ϕ i ⋅ θ A ∣ θ A ∈ Orbit | Θ , A > B ​ ( θ ) } \phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\coloneqq\left\{\phi_{i}\cdot\theta^{A}\mid\theta^{A}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\right\} . | Orbit | Θ , B > A ( θ ) ] | \displaystyle\left|\mathrm{Orbit}|_{\Theta,B>A}\left(\theta\right)]\right| ≥ | ⋃ i = 1 n ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) | \displaystyle\geq\left|\bigcup_{i=1}^{n}\phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\right| (20) = ∑ i = 1 n | ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) | \displaystyle=\sum_{i=1}^{n}\left|\phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\right| (21) = n ​ | Orbit | Θ , A > B ​ ( θ ) | . \displaystyle=n\left|\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\right|. (22) By item 1 and item 2 , ϕ i ⋅ ϕ i ⋅ Orbit | Θ , A > B ( θ ) ⊆ ϕ i ⋅ Orbit | Θ , B > A ( θ ) ] \phi_{i}\cdot\phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right)\subseteq\phi_{i}\cdot\mathrm{Orbit}|_{\Theta,B>A}\left(\theta\right)] for all i i . Therefore, eq. 20 holds. Equation 21 follows by the assumption that parameters are distinct, and so therefore the cosets ϕ i ⋅ Orbit | Θ , A > B ​ ( θ ) \phi_{i}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) and ϕ j ⋅ Orbit | Θ , A > B ​ ( θ ) \phi_{j}\cdot\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) are pairwise disjoint for i ≠ j i\neq j . Equation 22 follows because each ϕ i \phi_{i} acts injectively on orbit elements. Letting f A ​ ( θ ) ≔ f ⁡ ( A ∣ θ ) f_{A}(\theta)\coloneqq f(A\mid\theta) and f B ​ ( θ ) ≔ f ⁡ ( B ∣ θ ) f_{B}(\theta)\coloneqq f(B\mid\theta) , the shown inequality satisfies definition 3.2 . We conclude that f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta) . ∎ See 3.3 See 3.4 ###### Proof. Given that f f is a ( Θ , A ​ → simple ​ B ) (\Theta,A\overset{\text{simple}}{\to}B) -retargetable function ( definition 3.3 ), we want to show that f f is a ( Θ , A ​ → 1 ​ B ) (\Theta,A\overset{1}{\to}B) -retargetable function ( definition 3.5 when n = 1 n=1 ). Definition 3.5 ’s item 1 is true by assumption. Since Θ \Theta is acted on by S d S_{d} , Θ \Theta is closed under permutation and so definition 3.5 ’s item 2 holds. When n = 1 n=1 , there are no i ≠ j i\neq j , and so definition 3.5 ’s item 3 is tautologically true. Then f f is a ( Θ , A ​ → 1 ​ B ) (\Theta,A\overset{1}{\to}B) -retargetable function; apply Lemma B.7 . ∎ ### B.2 Helper results on retargetable functions ###### Lemma B.7 (Quantitative general orbit lemma) . Let Θ \Theta be a subset of a set acted on by S d S_{d} , and let f : 𝐄 × Θ → ℝ f\mathrel{\mathop{\ordinarycolon}}\mathbf{E}\times\Theta\to\mathbb{R} . Consider A , B ∈ 𝐄 A,B\in\mathbf{E} . For each θ ∈ Θ \theta\in\Theta , choose involutions ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} . Let θ ∗ ∈ Orbit | Θ ​ ( θ ) \theta^{*}\in\mathrm{Orbit}|_{\Theta}\left(\theta\right) . 1. Retargetable under parameter permutation. There exist B i ⋆ ∈ 𝐄 B_{i}^{\star}\in\mathbf{E} such that if f ⁡ ( B ∣ θ ∗ ) < f ⁡ ( A ∣ θ ∗ ) f(B\mid\theta^{*}) B ​ ( θ ) \theta^{\prime},\theta^{\prime\prime}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) and let i ≠ j i\neq j . Suppose ϕ i ⋅ θ ′ = ϕ j ⋅ θ ′′ \phi_{i}\cdot\theta^{\prime}=\phi_{j}\cdot\theta^{\prime\prime} . We want to show that this leads to contradiction. f ⁡ ( A ∣ θ ′′ ) \displaystyle f(A\mid\theta^{\prime\prime}) ≤ f ⁡ ( B j ⋆ ∣ ϕ j ⋅ θ ′′ ) \displaystyle\leq f(B_{j}^{\star}\mid\phi_{j}\cdot\theta^{\prime\prime}) (30) = f ⁡ ( B j ⋆ ∣ ϕ i − 1 ⋅ θ ′ ) \displaystyle=f(B_{j}^{\star}\mid\phi_{i}^{-1}\cdot\theta^{\prime}) (31) ≤ f ⁡ ( B j ⋆ ∣ θ ′ ) \displaystyle\leq f(B_{j}^{\star}\mid\theta^{\prime}) (32) ≤ f ⁡ ( B ∣ θ ′ ) \displaystyle\leq f(B\mid\theta^{\prime}) (33) < f ⁡ ( A ∣ θ ′ ) \displaystyle A ( θ ) ] \phi_{i}^{-1}\cdot\theta^{\prime}=\phi_{i}\cdot\theta^{\prime}\in\mathrm{Orbit}|_{\Theta,B>A}\left(\theta\right)] by the first half of this proof. Equation 33 follows by our assumption of item 3 . Equation 34 follows because we assumed that θ ′ ∈ Orbit | Θ , A > B ​ ( θ ) \theta^{\prime}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) . Equation 35 through eq. 39 follow by the same reasoning, switching the roles of θ ′ \theta^{\prime} and θ ′′ \theta^{\prime\prime} , and of i i and j j . But then we have demonstrated that a quantity is strictly less than itself, a contradiction. So for all θ ′ , θ ′′ ∈ Orbit | Θ , A > B ​ ( θ ) \theta^{\prime},\theta^{\prime\prime}\in\mathrm{Orbit}|_{\Theta,A>B}\left(\theta\right) , when i ≠ j i\neq j , ϕ i ⋅ θ ′ ≠ ϕ j ⋅ θ ′′ \phi_{i}\cdot\theta^{\prime}\neq\phi_{j}\cdot\theta^{\prime\prime} . Therefore, we have shown definition 3.5 ’s item 3 , and so f f is a ( Θ , A ​ → 𝑛 ​ B ) (\Theta,A\overset{n}{\to}B) -retargetable function. Apply Theorem 3.6 in order to conclude that eq. 23 holds. ∎ ###### Definition B.8 (Superset-of-copy containment) . Let A , B ⊆ ℝ d A,B\subseteq\mathbb{R}^{d} . B B contains n n superset-copies B i ⋆ B_{i}^{\star} of A A when there exist involutions ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} such that ϕ i ⋅ A ⊆ B i ⋆ ⊆ B \phi_{i}\cdot A\subseteq B_{i}^{\star}\subseteq B , and whenever i ≠ j i\neq j , ϕ i ⋅ B j ⋆ = B j ⋆ \phi_{i}\cdot B_{j}^{\star}=B_{j}^{\star} . ###### Lemma B.9 (Looser sufficient conditions for orbit-level incentives) . Suppose that Θ \Theta is a subset of a set acted on by S d S_{d} and is closed under permutation by S d S_{d} . Let A , B ∈ 𝐄 ⊆ 𝒫 ⁡ ( ℝ d ) A,B\in\mathbf{E}\subseteq\mathcal{P}\left(\mathbb{R}^{d}\right) . Suppose that B B contains n n superset-copies B i ⋆ ∈ 𝐄 B_{i}^{\star}\in\mathbf{E} of A A via ϕ i \phi_{i} . Suppose that f ⁡ ( X ∣ θ ) f(X\mid\theta) is increasing under joint permutation by ϕ 1 , … , ϕ n ∈ S d \phi_{1},\ldots,\phi_{n}\in S_{d} for all X ∈ 𝐄 , θ ∈ Θ X\in\mathbf{E},\theta\in\Theta , and suppose that ∀ i : ϕ i ⋅ A ∈ 𝐄 \forall i\mathrel{\mathop{\ordinarycolon}}\phi_{i}\cdot A\in\mathbf{E} . Suppose that f f is monotonically increasing on its first argument. Then f ( B ∣ θ ) ≥ most : Θ n f ( A ∣ θ ) . f(B\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}f(A\mid\theta). ###### Proof. We check the conditions of Lemma B.7 . Let θ ∈ Θ \theta\in\Theta , and let θ ∗ ∈ ( S d ⋅ θ ) ∩ Θ \theta^{*}\in\left(S_{d}\cdot\theta\right)\cap\Theta be an orbit element. Item 1 . Holds since f ⁡ ( A ∣ θ ∗ ) ≤ f ⁡ ( ϕ i ⋅ A ∣ ϕ i ⋅ θ ∗ ) ≤ f ⁡ ( B i ⋆ ∣ ϕ i ⋅ θ ∗ ) f(A\mid\theta^{*})\leq f(\phi_{i}\cdot A\mid\phi_{i}\cdot\theta^{*})\leq f(B^{\star}_{i}\mid\phi_{i}\cdot\theta^{*}) , with the first inequality by assumption of joint increasing under permutation, and the second following from monotonicity (as ϕ i ⋅ A ⊆ B i ⋆ \phi_{i}\cdot A\subseteq B^{\star}_{i} by superset copy definition B.8 ). Item 2 . We have ∀ θ ∗ ∈ ( S d ⋅ θ ∗ ) ∩ Θ : f ⁡ ( B ∣ θ ∗ ) < f ⁡ ( A ∣ θ ∗ ) ⟹ ∀ i = 1 , … , n : ϕ i ⋅ θ ∗ ∈ Θ \forall\theta^{*}\in\left(S_{d}\cdot\theta^{*}\right)\cap\Theta\mathrel{\mathop{\ordinarycolon}}f(B\mid\theta^{*}) M ) ≤ q } . M_{q,P}(C\mid\mathbf{u})\coloneqq\inf\left\{M\in\mathbb{R}\mid\Prb_{\mathbf{x}\sim P}\left(\mathbf{x}^{\top}\mathbf{u}>M\right)\leq q\right\}. (49) Let C > M q , P ​ ( C ∣ 𝐮 ) ≔ { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 > M q , P ​ ( C ∣ 𝐮 ) } C_{>M_{q,P}(C\mid\mathbf{u})}\coloneqq\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}>M_{q,P}(C\mid\mathbf{u})\right\} . C = M q , P ​ ( C ∣ 𝐮 ) C_{=M_{q,P}(C\mid\mathbf{u})} is defined similarly. Let 𝟙 L ⁡ ( x ) \mathbbm{1}_{L(x)} be the predicate function returning 1 1 if L ⁡ ( x ) L(x) is true and 0 0 otherwise. Then for X ⊆ C X\subseteq C , Q q , P ​ ( X ∣ C , 𝐮 ) ≔ ∑ 𝐱 ∈ X P ⁡ ( 𝐱 ) q ​ ( 𝟙 𝐱 ∈ C > M q , P ​ ( C ∣ 𝐮 ) + 𝟙 𝐱 ∈ C = M q , P ​ ( C ∣ 𝐮 ) P ⁡ ( C = M q , P ​ ( C ∣ 𝐮 ) ) ​ ( q − P ⁡ ( C > M q , P ​ ( C ∣ 𝐮 ) ) ) ) , \displaystyle Q_{q,P}(X\mid C,\mathbf{u})\coloneqq\sum_{\mathbf{x}\in X}\frac{P(\mathbf{x})}{q}\left(\mathbbm{1}_{\mathbf{x}\in C_{>M_{q,P}(C\mid\mathbf{u})}}+\frac{\mathbbm{1}_{\mathbf{x}\in C_{=M_{q,P}(C\mid\mathbf{u})}}}{P\left(C_{=M_{q,P}(C\mid\mathbf{u})}\right)}\left(q-P\left(C_{>M_{q,P}(C\mid\mathbf{u})}\right)\right)\right), (50) where the summand is defined to be 0 0 if P ⁡ ( 𝐱 ) = 0 P(\mathbf{x})=0 and 𝐱 ∈ C = M q , P ​ ( C ∣ 𝐮 ) \mathbf{x}\in C_{=M_{q,P}(C\mid\mathbf{u})} . ###### Remark . Unlike Taylor [2016] ’s or Carey [2019] ’s definitions, definition B.12 is written in closed form and requires no arbitrary tie-breaking. Instead, in the case of an expected utility tie on the quantile threshold, eq. 50 allots probability to outcomes proportional to their probability under the base distribution P P . Thanks to Theorem A.13 , we straightforwardly prove most items of Proposition A.11 by just rewriting each decision-making function as an EU-determined function. Most of the proof’s length comes from showing that the functions are measurable on 𝐮 \mathbf{u} , which means that the results also apply for distributions over utility functions 𝒟 any ∈ 𝔇 any \mathcal{D}_{\text{any}}\in\mathfrak{D}_{\text{any}} . See A.11 ###### Proof. Item 1 . Consider h ⁡ ( X , C ∣ 𝐮 ) \displaystyle h(X,C\mid\mathbf{u}) ≔ 𝟙 ∃ 𝐱 ∈ X : ∀ 𝐜 ∈ C : 𝐱 ⊤ ​ 𝐮 ≥ 𝐜 ⊤ ​ 𝐮 \displaystyle\coloneqq\mathbbm{1}_{\exists\mathbf{x}\in X\mathrel{\mathop{\ordinarycolon}}\forall\mathbf{c}\in C\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{u}\geq\mathbf{c}^{\top}\mathbf{u}} (51) = min ⁡ ( 1 , ∑ 𝐱 ∈ X ∏ 𝐜 ∈ C 𝟙 ( 𝐱 − 𝐜 ) ⊤ ​ 𝐮 ≥ 0 ) . \displaystyle=\min\left(1,\sum_{\mathbf{x}\in X}\prod_{\mathbf{c}\in C}\mathbbm{1}_{(\mathbf{x}-\mathbf{c})^{\top}\mathbf{u}\geq 0}\right). (52) Since halfspaces are measurable, each indicator function is measurable on 𝐮 \mathbf{u} . The finite sum of the finite product of measurable functions is also measurable. Since min \min is continuous (and therefore measurable), h ⁡ ( X , C ∣ 𝐮 ) h(X,C\mid\mathbf{u}) is measurable on 𝐮 \mathbf{u} . Furthermore, h h is an EU-determined function: h ⁡ ( X , C ∣ 𝐮 ) \displaystyle h(X,C\mid\mathbf{u}) = g ⁡ ( [ 𝐱 ⊤ ​ 𝐮 ] 𝐱 ∈ X ⏞ V X , [ 𝐜 ⊤ ​ 𝐮 ] 𝐜 ∈ C ⏞ V C ) \displaystyle=g\left(\overbrace{\left[\mathbf{x}^{\top}\mathbf{u}\right]_{\mathbf{x}\in X}}^{V_{X}},\overbrace{\left[\mathbf{c}^{\top}\mathbf{u}\right]_{\mathbf{c}\in C}}^{V_{C}}\right) (53) ≔ 𝟙 ∃ v x ∈ V X : ∀ v c ∈ V C : v x ≥ v c . \displaystyle\coloneqq\mathbbm{1}_{\exists v_{x}\in V_{X}\mathrel{\mathop{\ordinarycolon}}\forall v_{c}\in V_{C}\mathrel{\mathop{\ordinarycolon}}v_{x}\geq v_{c}}. (54) Then by Lemma B.11 , h h is invariant to joint permutation by the ϕ i \phi_{i} . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C , Lemma B.10 shows that h ′ ​ ( X ∣ 𝐮 ) ≔ h ⁡ ( X , C ∣ 𝐮 ) h^{\prime}(X\mid\mathbf{u})\coloneqq h(X,C\mid\mathbf{u}) is also invariant under joint permutation by the ϕ i \phi_{i} . Since h h is a measurable function of 𝐮 \mathbf{u} , so is h ′ h^{\prime} . Then since h ′ h^{\prime} is bounded, Lemma B.5 shows that f ⁡ ( X ∣ 𝒟 any ) ≔ 𝔼 𝐮 ∼ 𝒟 any [ h ′ ​ ( X ∣ 𝐮 ) ] f(X\mid\mathcal{D}_{\text{any}})\coloneqq\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[h^{\prime}(X\mid\mathbf{u})\right] is invariant under joint permutation by ϕ i \phi_{i} . Furthermore, if X ′ ⊆ X X^{\prime}\subseteq X , f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) by the monotonicity of probability. Then by Lemma B.9 , f ( B ∣ 𝒟 any ) ≔ IsOptimal ( B ∣ C , 𝒟 any ) ≥ most : 𝔇 any n IsOptimal ( A ∣ C , 𝒟 any ) ≕ f ( A ∣ 𝒟 any ) . f(B\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{IsOptimal}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\mathrm{IsOptimal}\left(A\mid C,\mathcal{D}_{\text{any}}\right)\eqqcolon f(A\mid\mathcal{D}_{\text{any}}). Item 2 . Because X , C X,C are finite sets, the denominator of FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is never zero, and so the function is well-defined. FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is an EU-determined function: FracOptimal ⁡ ( X ∣ C , 𝐮 ) \displaystyle\mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) = g ⁡ ( [ 𝐱 ⊤ ​ 𝐮 ] 𝐱 ∈ X ⏞ V X , [ 𝐜 ⊤ ​ 𝐮 ] 𝐜 ∈ C ⏞ V C ) \displaystyle=g\left(\overbrace{\left[\mathbf{x}^{\top}\mathbf{u}\right]_{\mathbf{x}\in X}}^{V_{X}},\overbrace{\left[\mathbf{c}^{\top}\mathbf{u}\right]_{\mathbf{c}\in C}}^{V_{C}}\right) (55) ≔ | [ v ∈ V X ∣ v = max v ′ ∈ V C ⁡ v ′ ] | | [ arg ​ max v ′ ∈ V C ⁡ v ′ ] | , \displaystyle\coloneqq\frac{\left|\left[v\in V_{X}\mid v=\max_{v^{\prime}\in V_{C}}v^{\prime}\right]\right|}{\left|\left[\argmax_{v^{\prime}\in V_{C}}v^{\prime}\right]\right|}, (56) with the [ ⋅ ] \left[\cdot\right] denoting a multiset which allows and counts duplicates. Then by Lemma B.11 , FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is invariant to joint permutation by the ϕ i \phi_{i} . We now show that FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is a measurable function of 𝐮 \mathbf{u} . FracOptimal ⁡ ( X ∣ C , 𝐮 ) \displaystyle\mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) ≔ | { arg ​ max 𝐜 ′ ∈ C ⁡ 𝐜 ′ ⁣ ⊤ ​ 𝐮 } ∩ X | | { arg ​ max 𝐜 ′ ∈ C ⁡ 𝐜 ′ ⁣ ⊤ ​ 𝐮 } | \displaystyle\coloneqq\frac{\left|\left\{\argmax_{\mathbf{c}^{\prime}\in C}\mathbf{c}^{\prime\top}\mathbf{u}\right\}\cap X\right|}{\left|\left\{\argmax_{\mathbf{c}^{\prime}\in C}\mathbf{c}^{\prime\top}\mathbf{u}\right\}\right|} (57) = ∑ 𝐱 ∈ X 𝟙 𝐱 ∈ arg ​ max 𝐜 ′ ∈ C ⁡ 𝐜 ′ ⁣ ⊤ ​ 𝐮 ∑ 𝐜 ∈ C 𝟙 𝐜 ∈ arg ​ max 𝐜 ′ ∈ C ⁡ 𝐜 ′ ⁣ ⊤ ​ 𝐮 \displaystyle=\frac{\sum_{\mathbf{x}\in X}\mathbbm{1}_{\mathbf{x}\in\argmax_{\mathbf{c}^{\prime}\in C}\mathbf{c}^{\prime\top}\mathbf{u}}}{\sum_{\mathbf{c}\in C}\mathbbm{1}_{\mathbf{c}\in\argmax_{\mathbf{c}^{\prime}\in C}\mathbf{c}^{\prime\top}\mathbf{u}}} (58) = ∑ 𝐱 ∈ X ∏ 𝐜 ′ ∈ C 𝟙 ( 𝐱 − 𝐜 ′ ) ⊤ ​ 𝐮 ≥ 0 ∑ 𝐜 ∈ C ∏ 𝐜 ′ ∈ C 𝟙 ( 𝐜 − 𝐜 ′ ) ⊤ ​ 𝐮 ≥ 0 . \displaystyle=\frac{\sum_{\mathbf{x}\in X}\prod_{\mathbf{c}^{\prime}\in C}\mathbbm{1}_{\left(\mathbf{x}-\mathbf{c}^{\prime}\right)^{\top}\mathbf{u}\geq 0}}{\sum_{\mathbf{c}\in C}\prod_{\mathbf{c}^{\prime}\in C}\mathbbm{1}_{\left(\mathbf{c}-\mathbf{c}^{\prime}\right)^{\top}\mathbf{u}\geq 0}}. (59) Equation 59 holds because 𝐱 \mathbf{x} belongs to the arg ​ max \argmax iff ∀ 𝐜 ∈ C : 𝐱 ⊤ ​ 𝐮 ≥ 𝐜 ⊤ ​ 𝐮 \forall\mathbf{c}\in C\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{u}\geq\mathbf{c}^{\top}\mathbf{u} . Furthermore, this condition is met iff 𝐮 \mathbf{u} belongs to the intersection of finitely many closed halfspaces; therefore, { 𝐮 ∈ ℝ d ∣ ∏ 𝐜 ∈ C 𝟙 ( 𝐱 − 𝐜 ) ⊤ ​ 𝐮 ≥ 0 = 1 } \left\{\mathbf{u}\in\mathbb{R}^{d}\mid\prod_{\mathbf{c}\in C}\mathbbm{1}_{\left(\mathbf{x}-\mathbf{c}\right)^{\top}\mathbf{u}\geq 0}=1\right\} is measurable. Then the sums in both the numerator and denominator are both measurable functions of 𝐮 \mathbf{u} , and the denominator cannot vanish. Therefore, FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is a measurable function of 𝐮 \mathbf{u} . Let g ⁡ ( X ∣ 𝐮 ) ≔ FracOptimal ⁡ ( X ∣ C , 𝐮 ) g(X\mid\mathbf{u})\coloneqq\mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C , Lemma B.10 shows that g ⁡ ( X ∣ 𝐮 ) g(X\mid\mathbf{u}) is also invariant to joint permutation by ϕ i \phi_{i} . Since g g is measurable and bounded [ 0 , 1 ] [0,1] , apply Lemma B.5 to conclude that f ⁡ ( X ∣ 𝒟 any ) ≔ 𝔼 𝐮 ∼ 𝒟 any [ g ⁡ ( X ∣ C , 𝐮 ) ] f(X\mid\mathcal{D}_{\text{any}})\coloneqq\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[g(X\mid C,\mathbf{u})\right] is also invariant to joint permutation by ϕ i \phi_{i} . Furthermore, if X ′ ⊆ X ⊆ C X^{\prime}\subseteq X\subseteq C , then f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) . So apply Lemma B.9 to conclude that FracOptimal ( B ∣ C , 𝒟 any ) ≕ f ( B ∣ 𝒟 any ) ≥ most : 𝔇 any n f ( A ∣ 𝒟 any ) ≔ FracOptimal ( A ∣ C , 𝒟 any ) \mathrm{FracOptimal}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\eqqcolon f(B\mid\mathcal{D}_{\text{any}})\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}f(A\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{FracOptimal}\left(A\mid C,\mathcal{D}_{\text{any}}\right) . Item 3 . Apply the reasoning in item 1 with inner function h ( X ∣ C , 𝐮 ) ≔ 𝟙 ∃ 𝐱 ∈ X : ∀ 𝐜 ∈ C : 𝐱 ⊤ ​ 𝐮 ≤ 𝐜 ⊤ ​ 𝐮 h(X\mid C,\mathbf{u})\coloneqq\mathbbm{1}_{\exists\mathbf{x}\in X\mathrel{\mathop{\ordinarycolon}}\forall\mathbf{c}\in C\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{u}\leq\mathbf{c}^{\top}\mathbf{u}} . Item 4 . Let X ⊆ C X\subseteq C . Boltzmann T ​ ( X ∣ C , 𝐮 ) \mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right) is the expectation of an EU function: Boltzmann T ​ ( X ∣ C , 𝐮 ) \displaystyle\mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right) = g T ​ ( [ 𝐱 ⊤ ​ 𝐮 ] 𝐱 ∈ X ⏞ V X , [ 𝐜 ⊤ ​ 𝐮 ] 𝐜 ∈ C ⏞ V C ) \displaystyle=g_{T}\left(\overbrace{\left[\mathbf{x}^{\top}\mathbf{u}\right]_{\mathbf{x}\in X}}^{V_{X}},\overbrace{\left[\mathbf{c}^{\top}\mathbf{u}\right]_{\mathbf{c}\in C}}^{V_{C}}\right) (60) ≔ ∑ v ∈ V X e v / T ∑ v ∈ V C e v / T . \displaystyle\coloneqq\frac{\sum_{v\in V_{X}}e^{v/T}}{\sum_{v\in V_{C}}e^{v/T}}. (61) Therefore, by Lemma B.11 , Boltzmann T ​ ( X ∣ C , 𝐮 ) \mathrm{Boltzmann}_{T}\left(X\mid C,\mathbf{u}\right) is invariant to joint permutation by the ϕ i \phi_{i} . Inspecting eq. 61 , we see that g g is continuous on 𝐮 \mathbf{u} (and therefore measurable), and bounded [ 0 , 1 ] [0,1] since X ⊆ C X\subseteq C and the exponential function is positive. Therefore, by Lemma B.5 , the expectation version is also invariant to joint permutation for all permutations ϕ ∈ S d \phi\in S_{d} : Boltzmann T ​ ( X ∣ C , 𝒟 any ) = Boltzmann T ​ ( ϕ ⋅ X ∣ ϕ ⋅ C , ϕ ⋅ 𝒟 any ) \mathrm{Boltzmann}_{T}\left(X\mid C,\mathcal{D}_{\text{any}}\right)=\mathrm{Boltzmann}_{T}\left(\phi\cdot X\mid\phi\cdot C,\phi\cdot\mathcal{D}_{\text{any}}\right) . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C , Lemma B.10 shows that f ⁡ ( X ∣ 𝒟 any ) ≔ Boltzmann T ​ ( X ∣ C , 𝒟 any ) f(X\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{Boltzmann}_{T}\left(X\mid C,\mathcal{D}_{\text{any}}\right) is also invariant under joint permutation by the ϕ i \phi_{i} . Furthermore, if X ′ ⊆ X X^{\prime}\subseteq X , then f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) . Then apply Lemma B.9 to conclude that Boltzmann T ( B ∣ C , 𝒟 any ) ≕ f ( B ∣ 𝒟 any ) ≥ most : 𝔇 any n f ( A ∣ 𝒟 any ) ≔ Boltzmann T ( A ∣ C , 𝒟 any ) \mathrm{Boltzmann}_{T}\left(B\mid C,\mathcal{D}_{\text{any}}\right)\eqqcolon f(B\mid\mathcal{D}_{\text{any}})\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}f(A\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{Boltzmann}_{T}\left(A\mid C,\mathcal{D}_{\text{any}}\right) . Item 5 . Let involution ϕ ∈ S d \phi\in S_{d} fix C C ( i . e ., ϕ ⋅ C = C \phi\cdot C=C ). best-of- ​ k ​ ( X ∣ C , 𝐮 ) \displaystyle\textrm{best-of-}k(X\mid C,\mathbf{u}) (62) ≔ 𝔼 𝐚 1 , … , 𝐚 k ∼ unif ​ ( C ) [ FracOptimal ⁡ ( X ∩ { 𝐚 1 , … , 𝐚 k } ∣ { 𝐚 1 , … , 𝐚 k } , 𝐮 ) ] \displaystyle\coloneqq\!\!\opE_{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\sim\text{unif}(C)}\left[\mathrm{FracOptimal}\left(X\cap\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\}\mid\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\},\mathbf{u}\right)\right] (63) = 𝔼 𝐚 1 , … , 𝐚 k ∼ unif ​ ( C ) [ FracOptimal ⁡ ( ( ϕ ⋅ X ) ∩ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } ∣ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } , ϕ ⋅ 𝐮 ) ] \displaystyle=\!\!\opE_{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\sim\text{unif}(C)}\left[\mathrm{FracOptimal}\left((\phi\cdot X)\cap\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\}\!\mid\!\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\},\phi\cdot\mathbf{u}\right)\right] (64) = 𝔼 ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k ∼ unif ​ ( ϕ ⋅ C ) [ FracOptimal ⁡ ( ( ϕ ⋅ X ) ∩ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } ∣ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } , ϕ ⋅ 𝐮 ) ] \displaystyle=\!\!\opE_{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\sim\text{unif}(\phi\cdot C)}\left[\mathrm{FracOptimal}\left((\phi\cdot X)\cap\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\}\!\mid\!\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\},\phi\cdot\mathbf{u}\right)\right] (65) ≕ best-of- ​ k ​ ( ϕ ⋅ X ∣ ϕ ⋅ C , ϕ ⋅ 𝐮 ) . \displaystyle\eqqcolon\textrm{best-of-}k(\phi\cdot X\mid\phi\cdot C,\phi\cdot\mathbf{u}). (66) By the proof of item 2 , FracOptimal ⁡ ( X ∩ { 𝐚 1 , … , 𝐚 k } ∣ { 𝐚 1 , … , 𝐚 k } , 𝐮 ) = FracOptimal ⁡ ( ( ϕ ⋅ X ) ∩ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } ∣ { ϕ ⋅ 𝐚 1 , … , ϕ ⋅ 𝐚 k } , ϕ ⋅ 𝐮 ) ; \mathrm{FracOptimal}\left(X\cap\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\}\mid\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\},\mathbf{u}\right)=\\ \mathrm{FracOptimal}\left((\phi\cdot X)\cap\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\}\mid\{\phi\cdot\mathbf{a}_{1},\ldots,\phi\cdot\mathbf{a}_{k}\},\phi\cdot\mathbf{u}\right); thus, eq. 64 holds. Since ϕ ⋅ C = C \phi\cdot C=C and since the distribution is uniform, eq. 65 holds. Therefore, best-of- ​ k ​ ( X ∣ C , 𝐮 ) \textrm{best-of-}k(X\mid C,\mathbf{u}) is invariant to joint permutation by the ϕ i \phi_{i} , which are involutions fixing C C . We now show that best-of- ​ k ​ ( X ∣ C , 𝐮 ) \textrm{best-of-}k(X\mid C,\mathbf{u}) is measurable on 𝐮 \mathbf{u} . best-of- ​ k ​ ( X ∣ C , 𝐮 ) \displaystyle\textrm{best-of-}k(X\mid C,\mathbf{u}) (67) ≔ 𝔼 𝐚 1 , … , 𝐚 k ∼ unif ​ ( C ) [ FracOptimal ⁡ ( X ∩ { 𝐚 1 , … , 𝐚 k } ∣ { 𝐚 1 , … , 𝐚 k } , 𝐮 ) ] \displaystyle\coloneqq\opE_{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\sim\text{unif}(C)}\left[\mathrm{FracOptimal}\left(X\cap\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\}\mid\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\},\mathbf{u}\right)\right] (68) = 1 | C | k ​ ∑ ( 𝐚 1 , … , 𝐚 k ) ∈ C k FracOptimal ⁡ ( X ∩ { 𝐚 1 , … , 𝐚 k } ∣ { 𝐚 1 , … , 𝐚 k } , 𝐮 ) . \displaystyle=\frac{1}{\left|C\right|^{k}}\sum_{\left(\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\right)\in C^{k}}\mathrm{FracOptimal}\left(X\cap\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\}\mid\{\mathbf{a}_{1},\ldots,\mathbf{a}_{k}\},\mathbf{u}\right). (69) Equation 69 holds because FracOptimal ⁡ ( X ∣ C , 𝐮 ) \mathrm{FracOptimal}\left(X\mid C,\mathbf{u}\right) is measurable on 𝐮 \mathbf{u} by item 2 , and measurable functions are closed under finite addition and scalar multiplication. Then best-of- ​ k ​ ( X ∣ C , 𝐮 ) \textrm{best-of-}k(X\mid C,\mathbf{u}) is measurable on 𝐮 \mathbf{u} . Let g ⁡ ( X ∣ 𝐮 ) ≔ best-of- ​ k ​ ( X ∣ C , 𝐮 ) g(X\mid\mathbf{u})\coloneqq\textrm{best-of-}k(X\mid C,\mathbf{u}) . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C , Lemma B.10 shows that g ⁡ ( X ∣ 𝐮 ) g(X\mid\mathbf{u}) is also invariant to joint permutation by ϕ i \phi_{i} . Since g g is measurable and bounded [ 0 , 1 ] [0,1] , apply Lemma B.5 to conclude that f ⁡ ( X ∣ 𝒟 any ) ≔ 𝔼 𝐮 ∼ 𝒟 any [ g ⁡ ( X ∣ C , 𝐮 ) ] f(X\mid\mathcal{D}_{\text{any}})\coloneqq\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[g(X\mid C,\mathbf{u})\right] is also invariant to joint permutation by ϕ i \phi_{i} . Furthermore, if X ′ ⊆ X ⊆ C X^{\prime}\subseteq X\subseteq C , then f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) . So apply Lemma B.9 to conclude that best-of- k ( B ∣ C , 𝒟 any ) ≕ f ( B ∣ 𝒟 any ) ≥ most : 𝔇 any n f ( A ∣ 𝒟 any ) ≔ best-of- k ( A ∣ C , 𝒟 any ) \textrm{best-of-}k(B\mid C,\mathcal{D}_{\text{any}})\eqqcolon f(B\mid\mathcal{D}_{\text{any}})\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}f(A\mid\mathcal{D}_{\text{any}})\coloneqq\textrm{best-of-}k(A\mid C,\mathcal{D}_{\text{any}}) . Item 6 . Satisfice t ​ ( X ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) is an EU-determined function: Satisfice t ​ ( X ∣ C , 𝐮 ) \displaystyle\mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) = g t ​ ( [ 𝐱 ⊤ ​ 𝐮 ] 𝐱 ∈ X ⏞ V X , [ 𝐜 ⊤ ​ 𝐮 ] 𝐜 ∈ C ⏞ V C ) \displaystyle=g_{t}\left(\overbrace{\left[\mathbf{x}^{\top}\mathbf{u}\right]_{\mathbf{x}\in X}}^{V_{X}},\overbrace{\left[\mathbf{c}^{\top}\mathbf{u}\right]_{\mathbf{c}\in C}}^{V_{C}}\right) (70) ≔ ∑ v ∈ V X 𝟙 v ≥ t ∑ v ∈ V C 𝟙 v ≥ t , \displaystyle\coloneqq\frac{\sum_{v\in V_{X}}\mathbbm{1}_{v\geq t}}{\sum_{v\in V_{C}}\mathbbm{1}_{v\geq t}}, (71) with the function evaluating to 0 0 if the denominator is 0 0 . Then applying Lemma B.11 , Satisfice t ​ ( X ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) is invariant under joint permutation by the ϕ i \phi_{i} . We now show that Satisfice t ​ ( X ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) is measurable on 𝐮 \mathbf{u} . Satisfice t ​ ( X ∣ C , 𝐮 ) \displaystyle\mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) = { ∑ 𝐱 ∈ X 𝟙 𝐱 ∈ { 𝐱 ′ ∈ ℝ d ∣ 𝐱 ′ ⁣ ⊤ ​ 𝐮 ≥ t } ∑ 𝐜 ∈ C 𝟙 𝐜 ∈ { 𝐱 ′ ∈ ℝ d ∣ 𝐱 ′ ⁣ ⊤ ​ 𝐮 ≥ t } ∃ 𝐜 ∈ C : 𝐜 ⊤ ​ 𝐮 ≥ t , 0 else . \displaystyle=\begin{cases}\frac{\sum_{\mathbf{x}\in X}\mathbbm{1}_{\mathbf{x}\in\left\{\mathbf{x}^{\prime}\in\mathbb{R}^{d}\mid\mathbf{x}^{\prime\top}\mathbf{u}\geq t\right\}}}{\sum_{\mathbf{c}\in C}\mathbbm{1}_{\mathbf{c}\in\left\{\mathbf{x}^{\prime}\in\mathbb{R}^{d}\mid\mathbf{x}^{\prime\top}\mathbf{u}\geq t\right\}}}&\exists\mathbf{c}\in C\mathrel{\mathop{\ordinarycolon}}\mathbf{c}^{\top}\mathbf{u}\geq t,\\ 0&\text{ else}.\end{cases} (72) Consider the two cases. ∃ 𝐜 ∈ C : 𝐜 ⊤ ​ 𝐮 ≥ t ⇔ 𝐮 ∈ ⋃ 𝐜 ∈ C { 𝐮 ′ ∈ ℝ d ∣ 𝐜 ⊤ ​ 𝐮 ≥ t } . \exists\mathbf{c}\in C\mathrel{\mathop{\ordinarycolon}}\mathbf{c}^{\top}\mathbf{u}\geq t\iff\mathbf{u}\in\bigcup_{\mathbf{c}\in C}\left\{\mathbf{u}^{\prime}\in\mathbb{R}^{d}\mid\mathbf{c}^{\top}\mathbf{u}\geq t\right\}. The right-hand set is the union of finitely many halfspaces (which are measurable), and so the right-hand set is also measurable. Then the casing is a measurable function of 𝐮 \mathbf{u} . Clearly the zero function is measurable. Now we turn to the first case. In the first case, eq. 72 ’s indicator functions test each 𝐱 , 𝐜 \mathbf{x},\mathbf{c} for membership in a closed halfspace with respect to 𝐮 \mathbf{u} . Halfspaces are measurable sets. Therefore, the indicator function is a measurable function of 𝐮 \mathbf{u} , and so are the finite sums. Since the denominator does not vanish within the case, the first case as a whole is a measurable function of 𝐮 \mathbf{u} . Therefore, Satisfice t ​ ( X ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) is measurable on 𝐮 \mathbf{u} . Since Satisfice t ​ ( X ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathbf{u}\right) is measurable and bounded [ 0 , 1 ] [0,1] (as X ⊆ C X\subseteq C ), apply Lemma B.5 to conclude that Satisfice t ​ ( X ∣ C , 𝒟 any ) = Satisfice t ​ ( ϕ ⋅ X ∣ ϕ ⋅ C , ϕ ⋅ 𝒟 any ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathcal{D}_{\text{any}}\right)=\mathrm{Satisfice}_{t}\left(\phi\cdot X\mid\phi\cdot C,\phi\cdot\mathcal{D}_{\text{any}}\right) . Next, let f ⁡ ( X ∣ 𝒟 any ) ≔ Satisfice t ​ ( X ∣ C , 𝒟 any ) f(X\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{Satisfice}_{t}\left(X\mid C,\mathcal{D}_{\text{any}}\right) . Since we just showed that Satisfice t ​ ( X ∣ C , 𝒟 any ) \mathrm{Satisfice}_{t}\left(X\mid C,\mathcal{D}_{\text{any}}\right) is invariant to joint permutation by the involutions ϕ i \phi_{i} and since ϕ i ⋅ C = C \phi_{i}\cdot C=C , f ⁡ ( X ∣ 𝒟 any ) f(X\mid\mathcal{D}_{\text{any}}) is also invariant to joint permutation by ϕ i \phi_{i} . Furthermore, if X ′ ⊆ X X^{\prime}\subseteq X , we have f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) . Then applying Lemma B.9 , Satisfice t ( B ∣ C , 𝐮 ) ≕ f ( B ∣ 𝒟 any ) ≥ most : 𝔇 any n f ( A ∣ 𝒟 any ) ≔ Satisfice t ( A ∣ C , 𝐮 ) \mathrm{Satisfice}_{t}\left(B\mid C,\mathbf{u}\right)\eqqcolon f(B\mid\mathcal{D}_{\text{any}})\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}f(A\mid\mathcal{D}_{\text{any}})\coloneqq\mathrm{Satisfice}_{t}\left(A\mid C,\mathbf{u}\right) . Item 7 . Suppose P P is uniform over C C and consider any of the involutions ϕ i \phi_{i} . M q , P ​ ( C ∣ 𝐮 ) \displaystyle M_{q,P}(C\mid\mathbf{u}) ≔ inf { M ∈ ℝ ∣ ℙ 𝐱 ∼ P ( 𝐱 ⊤ ​ 𝐮 > M ) ≤ q } \displaystyle\coloneqq\inf\left\{M\in\mathbb{R}\mid\Prb_{\mathbf{x}\sim P}\left(\mathbf{x}^{\top}\mathbf{u}>M\right)\leq q\right\} (73) = inf { M ∈ ℝ ∣ ℙ 𝐱 ∼ P ( ( 𝐏 ϕ i ​ 𝐱 ) ⊤ ​ ( 𝐏 ϕ i ​ 𝐮 ) > M ) ≤ q } \displaystyle=\inf\left\{M\in\mathbb{R}\mid\Prb_{\mathbf{x}\sim P}\left((\mathbf{P}_{\phi_{i}}\mathbf{x})^{\top}(\mathbf{P}_{\phi_{i}}\mathbf{u})>M\right)\leq q\right\} (74) = inf { M ∈ ℝ ∣ ℙ 𝐱 ∼ ϕ i ⋅ P ( 𝐱 ⊤ ​ ( 𝐏 ϕ i ​ 𝐮 ) > M ) ≤ q } \displaystyle=\inf\left\{M\in\mathbb{R}\mid\Prb_{\mathbf{x}\sim\phi_{i}\cdot P}\left(\mathbf{x}^{\top}(\mathbf{P}_{\phi_{i}}\mathbf{u})>M\right)\leq q\right\} (75) = inf { M ∈ ℝ ∣ ℙ 𝐱 ∼ P ( 𝐱 ⊤ ​ ( 𝐏 ϕ i ​ 𝐮 ) > M ) ≤ q } \displaystyle=\inf\left\{M\in\mathbb{R}\mid\Prb_{\mathbf{x}\sim P}\left(\mathbf{x}^{\top}(\mathbf{P}_{\phi_{i}}\mathbf{u})>M\right)\leq q\right\} (76) ≕ M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) . \displaystyle\eqqcolon M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u}). (77) Equation 74 follows by the orthogonality of permutation matrices. Equation 76 follows because if 𝐱 ∈ supp ⁡ ( P ) = C \mathbf{x}\in\optSupp(P)=C , then ϕ i ⋅ 𝐱 ∈ C = supp ⁡ ( P ) \phi_{i}\cdot\mathbf{x}\in C=\optSupp(P) , and furthermore P ⁡ ( 𝐱 ) = P ⁡ ( 𝐏 ϕ i ​ 𝐱 ) P(\mathbf{x})=P(\mathbf{P}_{\phi_{i}}\mathbf{x}) by uniformity. Now we show the invariance of C > M q , P ​ ( C ∣ 𝐮 ) C_{>M_{q,P}(C\mid\mathbf{u})} under joint permutation by ϕ i \phi_{i} : C > M q , P ​ ( C ∣ 𝐮 ) \displaystyle C_{>M_{q,P}(C\mid\mathbf{u})} ≔ { 𝐜 ∈ C ∣ 𝐜 ⊤ ​ 𝐮 > M q , P ​ ( C ∣ 𝐮 ) } \displaystyle\coloneqq\left\{\mathbf{c}\in C\mid\mathbf{c}^{\top}\mathbf{u}>M_{q,P}(C\mid\mathbf{u})\right\} (78) = { 𝐜 ∈ C ∣ ( 𝐏 ϕ i ​ 𝐜 ) ⊤ ​ ( 𝐏 ϕ i ​ 𝐮 ) > M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) } \displaystyle=\left\{\mathbf{c}\in C\mid(\mathbf{P}_{\phi_{i}}\mathbf{c})^{\top}(\mathbf{P}_{\phi_{i}}\mathbf{u})>M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u})\right\} (79) = { 𝐜 ∈ ϕ i ⋅ C ∣ 𝐜 ⊤ ​ ( 𝐏 ϕ i ​ 𝐮 ) > M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) } \displaystyle=\left\{\mathbf{c}\in\phi_{i}\cdot C\mid\mathbf{c}^{\top}(\mathbf{P}_{\phi_{i}}\mathbf{u})>M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u})\right\} (80) ≕ C > M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) . \displaystyle\eqqcolon C_{>M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u})}. (81) Equation 79 follows by the orthogonality of permutation matrices and because M q , P ​ ( C ∣ 𝐮 ) = M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) M_{q,P}(C\mid\mathbf{u})=M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u}) by eq. 77 . A similar proof shows that C = M q , P ​ ( C ∣ 𝐮 ) = C = M q , P ​ ( ϕ i ⋅ C ∣ ϕ i ⋅ 𝐮 ) C_{=M_{q,P}(C\mid\mathbf{u})}=C_{=M_{q,P}(\phi_{i}\cdot C\mid\phi_{i}\cdot\mathbf{u})} . Recall that Q q , P ​ ( X ∣ C , 𝐮 ) ≔ ∑ 𝐱 ∈ X P ⁡ ( 𝐱 ) q ​ ( 𝟙 𝐱 ∈ C > M q , P ​ ( C ∣ 𝐮 ) + 𝟙 𝐱 ∈ C = M q , P ​ ( C ∣ 𝐮 ) P ⁡ ( C = M q , P ​ ( C ∣ 𝐮 ) ) ​ ( q − P ⁡ ( C > M q , P ​ ( C ∣ 𝐮 ) ) ) ) . \displaystyle Q_{q,P}(X\mid C,\mathbf{u})\coloneqq\sum_{\mathbf{x}\in X}\frac{P(\mathbf{x})}{q}\left(\mathbbm{1}_{\mathbf{x}\in C_{>M_{q,P}(C\mid\mathbf{u})}}+\frac{\mathbbm{1}_{\mathbf{x}\in C_{=M_{q,P}(C\mid\mathbf{u})}}}{P\left(C_{=M_{q,P}(C\mid\mathbf{u})}\right)}\left(q-P\left(C_{>M_{q,P}(C\mid\mathbf{u})}\right)\right)\right). (82) Q q , P ​ ( X ∣ C , 𝐮 ) = Q q , P ​ ( ϕ i ⋅ X ∣ ϕ i ⋅ C , ϕ i ⋅ 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u})=Q_{q,P}(\phi_{i}\cdot X\mid\phi_{i}\cdot C,\phi_{i}\cdot\mathbf{u}) , since Q Q is the sum of products of ϕ i \phi_{i} -invariant quantities. P ⁡ ( 𝐱 ) P(\mathbf{x}) is non-negative because P P is a probability distribution, and q q is assumed positive. The indicator functions 𝟙 \mathbbm{1} are non-negative. By the definition of M q , P M_{q,P} , P ⁡ ( C > M q , P ​ ( C ∣ 𝐮 ) ) ≤ q P\left(C_{>M_{q,P}(C\mid\mathbf{u})}\right)\leq q . Therefore, eq. 82 is the sum of non-negative terms. Thus, if X ′ ⊆ X X^{\prime}\subseteq X , then Q q , P ​ ( X ′ ∣ C , 𝐮 ) ≤ Q q , P ​ ( X ∣ C , 𝐮 ) Q_{q,P}(X^{\prime}\mid C,\mathbf{u})\leq Q_{q,P}(X\mid C,\mathbf{u}) . Let f ⁡ ( X ∣ 𝐮 ) ≔ Q q , P ​ ( X ∣ C , 𝐮 ) f(X\mid\mathbf{u})\coloneqq Q_{q,P}(X\mid C,\mathbf{u}) . Since ϕ i ⋅ C = C \phi_{i}\cdot C=C and since Q q , P ​ ( X ∣ C , 𝐮 ) = Q q , P ​ ( ϕ i ⋅ X ∣ ϕ i ⋅ C , ϕ i ⋅ 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u})=Q_{q,P}(\phi_{i}\cdot X\mid\phi_{i}\cdot C,\phi_{i}\cdot\mathbf{u}) , Lemma B.10 shows that f ⁡ ( X ∣ 𝐮 ) f(X\mid\mathbf{u}) is also jointly invariant to permutation by ϕ i \phi_{i} . Lastly, if X ′ ⊆ X X^{\prime}\subseteq X , we have f ⁡ ( X ′ ∣ 𝒟 any ) ≤ f ⁡ ( X ∣ 𝒟 any ) f(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f(X\mid\mathcal{D}_{\text{any}}) . Apply Lemma B.9 to conclude that Q q , P ( B ∣ C , 𝐮 ) ≕ f ( B ∣ 𝐮 ) ≥ most : ℝ d n f ( A ∣ 𝐮 ) ≔ Q q , P ( A ∣ C , 𝐮 ) Q_{q,P}(B\mid C,\mathbf{u})\eqqcolon f(B\mid\mathbf{u})\geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n}f(A\mid\mathbf{u})\coloneqq Q_{q,P}(A\mid C,\mathbf{u}) . ∎ ###### Conjecture B.13 (Orbit tendencies occur for more quantilizer base distributions) . Proposition A.11 ’s item 7 holds for any base distribution P P over C C such that min 𝐛 ∈ B ⁡ P ⁡ ( 𝐛 ) ≥ max 𝐚 ∈ A ⁡ P ⁡ ( 𝐚 ) \min_{\mathbf{b}\in B}P(\mathbf{b})\geq\max_{\mathbf{a}\in A}P(\mathbf{a}) . Furthermore, Q q , P ​ ( X ∣ C , 𝐮 ) Q_{q,P}(X\mid C,\mathbf{u}) is measurable on 𝐮 \mathbf{u} and so ≥ most : ℝ d n \geq_{\text{{most}}\text{: }\mathbb{R}^{d}}^{n} can be generalized to ≥ most : 𝔇 any n \geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n} . ## Appendix C Detailed analyses of mr scenarios ### C.1 Action selection Consider a bandit problem with five arms a 1 , … , a 5 a_{1},\ldots,a_{5} partitioned A ≔ { a 1 } , B ≔ { a 2 , … , a 5 } A\coloneqq\left\{a_{1}\right\},B\coloneqq\left\{a_{2},\ldots,a_{5}\right\} , which each action has a definite utility 𝐮 i \mathbf{u}_{i} . There are T = 100 T=100 trials. Suppose the training procedure train uses the ϵ \epsilon -greedy strategy to learn value estimates for each arm. At the end of training, train outputs a greedy policy with respect to its value estimates. Consider any action-value initialization, and the learning rate is set α ≔ 1 \alpha\coloneqq 1 . To learn an optimal policy, at worst, the agent just has to try each action once. ###### Lemma C.1 (Lower bound on success probability of the train bandit) . Let 𝐮 ∈ ℝ 5 \mathbf{u}\in\mathbb{R}^{5} assign strictly maximal utility to a i a_{i} , and suppose train (described above) runs for T ≥ 5 T\geq 5 trials. Then p train ​ ( { a i } ∣ 𝐮 ) ≥ 1 − ( 1 − ϵ 4 ) T p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\geq 1-(1-\frac{\epsilon}{4})^{T} . ###### Proof. Since the trained policy can be stochastic, p train ​ ( { a i } ∣ 𝐮 ) ≥ ℙ ( a i ​ is assigned probability 1 by the learned greedy policy ) . p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\geq\Prb\left(a_{i}\text{ is assigned probability $1$ by the learned greedy policy}\right). Since a i a_{i} has strictly maximal utility which is deterministic, and since the learning rate α ≔ 1 \alpha\coloneqq 1 , if action a i a_{i} is ever drawn, it is assigned probability 1 1 by the learned policy. The probability that a i a_{i} is never explored is at most ( 1 − ϵ 4 ) T (1-\frac{\epsilon}{4})^{T} , because at worst, a i a_{i} is an “explore” action (and not an “exploit” action) at every time step, in which case it is ignored with probability 1 − ϵ 4 1-\frac{\epsilon}{4} . ∎ ###### Proposition C.2 (The train bandit is 4-retargetable) . p train p_{\textrm{train}} is ( ℝ 5 , A ​ → 4 ​ B ) (\mathbb{R}^{5},A\overset{4}{\to}B) -retargetable. ###### Proof. Let ϕ i ≔ a 1 ↔ a i \phi_{i}\coloneqq a_{1}\leftrightarrow a_{i} for i = 2 , … , 5 i=2,\ldots,5 and let Θ ≔ ℝ 5 \Theta\coloneqq\mathbb{R}^{5} . We want to show that whenever 𝐮 ∈ ℝ 5 \mathbf{u}\in\mathbb{R}^{5} induces p train ​ ( A ∣ 𝐮 ) > p train ​ ( B ∣ 𝐮 ) p_{\textrm{train}}(A\mid\mathbf{u})>p_{\textrm{train}}(B\mid\mathbf{u}) , retargeting 𝐮 \mathbf{u} will get train to instead learn to pull a B B -action: p train ​ ( A ∣ ϕ i ⋅ 𝐮 ) < p train ​ ( B ∣ ϕ i ⋅ 𝐮 ) p_{\textrm{train}}(A\mid\phi_{i}\cdot\mathbf{u}) max i = 2 , … , 5 ⁡ 𝐮 i \mathbf{u}_{1}>\max_{i=2,\ldots,5}\mathbf{u}_{i} ). But for T = 100 T=100 , Lemma C.1 shows that p train ​ ( A ∣ 𝐮 ) = p train ​ ( { a 1 } ∣ 𝐮 ) ≈ 1 p_{\textrm{train}}(A\mid\mathbf{u})=p_{\textrm{train}}(\left\{a_{1}\right\}\mid\mathbf{u})\approx 1 and p train ​ ( { a i ≠ 1 } ∣ 𝐮 ) ≈ 0 ⟹ p train ​ ( B ∣ 𝐮 ) = ∑ i ≠ 1 p train ​ ( { a i } ∣ 𝐮 ) ≈ 0 p_{\textrm{train}}(\left\{a_{i\neq 1}\right\}\mid\mathbf{u})\approx 0\implies p_{\textrm{train}}(B\mid\mathbf{u})=\sum_{i\neq 1}p_{\textrm{train}}(\left\{a_{i}\right\}\mid\mathbf{u})\approx 0 . The converse statement holds when considering ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} instead of 𝐮 \mathbf{u} . Therefore, train satisfies definition 3.5 ’s item 1 (retargetability). These ϕ i ⋅ 𝐮 ∈ Θ ≔ ℝ 5 \phi_{i}\cdot\mathbf{u}\in\Theta\coloneqq\mathbb{R}^{5} because ℝ 5 \mathbb{R}^{5} is closed under permutation by S 5 S_{5} , satisfying item 2 . Consider another 𝐮 ′ ∈ ℝ 5 \mathbf{u}^{\prime}\in\mathbb{R}^{5} such that p train ​ ( A ∣ 𝐮 ′ ) > p train ​ ( B ∣ 𝐮 ′ ) p_{\textrm{train}}(A\mid\mathbf{u}^{\prime})>p_{\textrm{train}}(B\mid\mathbf{u}^{\prime}) , and consider i ≠ j i\neq j . By the above symmetry arguments, 𝐮 ′ \mathbf{u}^{\prime} must also assign a 1 a_{1} maximal utility. By Lemma C.1 , p train ​ ( { a i } ∣ ϕ i ⋅ 𝐮 ) ≈ 1 p_{\textrm{train}}(\left\{a_{i}\right\}\mid\phi_{i}\cdot\mathbf{u})\approx 1 and p train ​ ( { a j } ∣ ϕ i ⋅ 𝐮 ) ≈ 0 p_{\textrm{train}}(\left\{a_{j}\right\}\mid\phi_{i}\cdot\mathbf{u})\approx 0 since i ≠ j i\neq j , and vice versa when considering ϕ j ⋅ 𝐮 \phi_{j}\cdot\mathbf{u} instead of ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} . Then since ϕ i ⋅ 𝐮 \phi_{i}\cdot\mathbf{u} and ϕ j ⋅ 𝐮 \phi_{j}\cdot\mathbf{u} induce distinct probability distributions over learned actions, they cannot be the same utility function. This satisfies item 3 . ∎ ###### Corollary C.3 (The train bandit has orbit-level tendencies) . p train ( B ∣ 𝐮 ) ≥ most: ℝ 5 4 p train ( A ∣ 𝐮 ) p_{\textrm{train}}(B\mid\mathbf{u})\geq^{4}_{\text{most: }\mathbb{R}^{5}}p_{\textrm{train}}(A\mid\mathbf{u}) . ###### Proof. Combine Proposition C.2 and Theorem 3.6 . ∎ ### C.2 Observation reward maximization Let T T be a reasonably long rollout length, so that O T -reach O_{\text{$T$-reach}} is large—many different step- T T observations can be induced. ###### Proposition C.4 (Final reward maximization has strong orbit-level incentives in mr ) . Let n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor . p max ( O leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( O stay ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\text{max}}(O_{\text{stay}}\mid R) . ###### Proof. Consider the vector space representation of observations, ℝ | 𝒪 | \mathbb{R}^{\left|\mathcal{O}\right|} . Define A ≔ { 𝐞 o ∣ o ∈ O stay } , B ≔ { 𝐞 o ∣ o ∈ O leave } A\coloneqq\{\mathbf{e}_{o}\mid o\in O_{\text{stay}}\},B\coloneqq\{\mathbf{e}_{o}\mid o\in O_{\text{leave}}\} , and C ≔ O T -reach = A ∪ B C\coloneqq O_{\text{$T$-reach}}=A\cup B the union of O stay , O leave O_{\text{stay}},O_{\text{leave}} . Since | O leave | ≥ | O stay | \left|O_{\text{leave}}\right|\geq\left|O_{\text{stay}}\right| by assumption that T T is reasonably large, consider the involution ϕ 1 ∈ S | 𝒪 | \phi_{1}\in S_{\left|\mathcal{O}\right|} which embeds O stay O_{\text{stay}} into O leave O_{\text{leave}} , while fixing all other observations. If possible, produce another involution ϕ 2 \phi_{2} which also embeds O stay O_{\text{stay}} into O leave O_{\text{leave}} , which fixes all other observations, and which “doesn’t interfere with ϕ 1 \phi_{1} ” ( i . e ., ϕ 2 ⋅ ( ϕ 1 ⋅ A ) = ϕ 1 ⋅ A \phi_{2}\cdot(\phi_{1}\cdot A)=\phi_{1}\cdot A ). We can produce n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor such involutions. Therefore, B B contains n n copies ( definition A.7 ) of A A via involutions ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} . Furthermore, ϕ i ⋅ ( A ∪ B ) = A ∪ B \phi_{i}\cdot(A\cup B)=A\cup B , since each ϕ i \phi_{i} swaps A A with B ′ ⊆ B B^{\prime}\subseteq B , and fixes all 𝐛 ∈ B ∖ B ′ \mathbf{b}\in B\setminus B^{\prime} by assumption. Thus, ϕ ⋅ C = C \phi\cdot C=C . By Proposition A.11 ’s item 2 , FracOptimal ( B ∣ C , R ) ≥ most : ℝ 𝒪 n FracOptimal ( A ∣ C , R ) \mathrm{FracOptimal}\left(B\mid C,R\right)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}\mathrm{FracOptimal}\left(A\mid C,R\right) . Since p max p_{\text{max}} uniformly randomly chooses a maximal-reward observation to induce, ∀ X ⊆ C : p max ​ ( X ∣ R ) = FracOptimal ⁡ ( X ∣ C , R ) \forall X\subseteq C\mathrel{\mathop{\ordinarycolon}}p_{\text{max}}(X\mid R)=\mathrm{FracOptimal}\left(X\mid C,R\right) . Therefore, p max ( O leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( O stay ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\text{max}}(O_{\text{stay}}\mid R) . ∎ We want to reason about the probability that decide \mathrm{decide} leaves the initial room by time T T in its rollout trajectories. p decide ​ ( leave ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{leave}\mid\theta) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ has left the first room by step T ) , \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has left the first room by step $T$}\right), (83) p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ has not left the first room by step T ) . \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has not left the first room by step $T$}\right). (84) We want to show that reward maximizers tend to leave the room: p max ( leave ∣ R ) ≥ most : Θ n p max ( stay ∣ R ) p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{\max}(\text{stay}\mid R) . However, we must be careful: In general, p max ​ ( O leave ∣ R ) ≠ p max ​ ( leave ∣ R ) p_{\text{max}}(O_{\text{leave}}\mid R)\neq p_{\max}(\text{leave}\mid R) and p max ​ ( O stay ∣ R ) ≠ p max ​ ( stay ∣ R ) p_{\text{max}}(O_{\text{stay}}\mid R)\neq p_{\max}(\text{stay}\mid R) . For example, suppose that o T ∈ O leave o_{T}\in O_{\text{leave}} . By the definition of O leave O_{\text{leave}} , o T o_{T} can only be observed if the agent has left the room by time step T T , and so the trajectory τ \tau must have left the first room. The converse argument does not hold: The agent could leave the first room, re-enter, and then wait until time T T . Although one of the doors would have been opened ( fig. 2 ), the agent can also open the door without leaving the room, and then realize the same step- T T observation. Therefore, this observation doesn’t belong to O leave O_{\text{leave}} . ###### Lemma C.5 (Room-status inequalities for mr ) . p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) ≤ p decide ​ ( O stay ∣ θ ) , \displaystyle\leq p_{\mathrm{decide}}(O_{\text{stay}}\mid\theta), (85) and ​ p decide ​ ( O leave ∣ θ ) \displaystyle\text{and }p_{\mathrm{decide}}(O_{\text{leave}}\mid\theta) ≤ p decide ​ ( leave ∣ θ ) . \displaystyle\leq p_{\mathrm{decide}}(\text{leave}\mid\theta). (86) ###### Proof. For any decide \mathrm{decide} , p decide ​ ( stay ∣ θ ) \displaystyle p_{\mathrm{decide}}(\text{stay}\mid\theta) (87) = ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ stays through step T ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ stays through step $T$}\right) (88) = ∑ o ∈ 𝒪 ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T of τ ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in\mathcal{O}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$ of $\tau$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (89) = ∑ o ∈ O T -reach ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{$T$-reach}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (90) = ∑ o ∈ O stay ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{stay}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right) (91) ≤ ∑ o ∈ O stay ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) \displaystyle\leq\sum_{o\in O_{\text{stay}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right) (92) = ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o T ∈ O stay ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}}\right) (93) ≕ p decide ​ ( O stay ∣ θ ) . \displaystyle\eqqcolon p_{\mathrm{decide}}(O_{\text{stay}}\mid\theta). (94) Equation 90 holds because the definition of O T -reach O_{\text{$T$-reach}} ensures that if o ∉ O T -reach o\not\in O_{\text{$T$-reach}} , then ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o ∣ θ ) = 0 \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o\mid\theta\right)=0 . Because o ∈ O T -reach ∖ O stay o\in O_{\text{$T$-reach}}\setminus O_{\text{stay}} implies that τ \tau left and so ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ stays ∣ o at step T ) = 0 , \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ stays}\mid\text{$o$ at step $T$}\right)=0, eq. 91 follows. Then we have shown eq. 85 . For eq. 86 , p decide ​ ( O leave ∣ θ ) \displaystyle p_{\mathrm{decide}}(O_{\text{leave}}\mid\theta) (95) ≔ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o T ∈ O leave ) \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}}\right) (96) = ∑ o ∈ O leave ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) \displaystyle=\sum_{o\in O_{\text{leave}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right) (97) = ∑ o ∈ O leave ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) \displaystyle=\sum_{o\in O_{\text{leave}}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right) (98) = ∑ o ∈ 𝒪 ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( o at step T ) ​ ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) \displaystyle=\sum_{o\in\mathcal{O}}\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$o$ at step $T$}\right)\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right) (99) = ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ ​ has left the first room by step T ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\tau\text{ has left the first room by step $T$}\right) (100) ≕ p decide ​ ( leave ∣ θ ) . \displaystyle\eqqcolon p_{\mathrm{decide}}(\text{leave}\mid\theta). (101) Equation 98 follows because, since o ∈ O leave o\in O_{\text{leave}} are only realizable by leaving the first room, this implies ℙ π ∼ decide ⁡ ( θ ) , τ ∼ π | s 0 ( τ leaves by step T ∣ o at step T ) = 1 \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(\text{$\tau$ leaves by step $T$}\mid\text{$o$ at step $T$}\right)=1 . Equation 99 follows because O leave ⊆ 𝒪 O_{\text{leave}}\subseteq\mathcal{O} , and probabilities are non-negative. Then we have shown eq. 86 . ∎ ###### Corollary C.6 (Final reward maximizers tend to leave the first room in mr ) . p max ( leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( stay ∣ R ) . p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\max}(\text{stay}\mid R). (102) ###### Proof. Using Lemma C.5 and Proposition C.4 , apply Lemma B.1 with f 0 ​ ( R ) ≔ p max ​ ( leave ∣ R ) , f 1 ​ ( R ) ≔ p max ​ ( O leave ∣ R ) , f 2 ​ ( R ) ≔ p max ​ ( O stay ∣ R ) , f 3 ​ ( R ) ≔ p max ​ ( stay ∣ R ) f_{0}(R)\coloneqq p_{\max}(\text{leave}\mid R),f_{1}(R)\coloneqq p_{\text{max}}(O_{\text{leave}}\mid R),f_{2}(R)\coloneqq p_{\text{max}}(O_{\text{stay}}\mid R),f_{3}(R)\coloneqq p_{\max}(\text{stay}\mid R) to conclude that p max ( leave ∣ R ) ≥ most : ℝ 𝒪 n p max ( stay ∣ R ) . p_{\max}(\text{leave}\mid R)\geq_{\text{{most}}\text{: }\mathbb{R}^{\mathcal{O}}}^{n}p_{\max}(\text{stay}\mid R). ∎ ### C.3 Featurized reward maximization Θ ≔ ℝ 𝒪 \Theta\coloneqq\mathbb{R}^{\mathcal{O}} assumes we will specify complicated reward functions over observations, with | 𝒪 | \left|\mathcal{O}\right| degrees of freedom in their specification. Any observation can get any number. However, reward functions are often specified more compactly. For example, in section 4.3 , the (additively) featurized reward function R feat ​ ( o T ) ≔ feat ​ ( o T ) ⊤ ​ α R_{\textrm{feat}}(o_{T})\coloneqq\textrm{feat}(o_{T})^{\top}\alpha has four degrees of freedom. Compared to typical reward functions (which would look like “random noise” to a human), R feat R_{\textrm{feat}} more easily trains competent policies because of the regularities between the reward and the state features. In this setup, p max p_{\text{max}} chooses a policy which induces a step- T T observation with maximal reward. Reward depends only on the feature vector of the final observation—more specifically, on the agent’s item counts. There are more possible item counts available by first leaving the room, than by staying. We will now conduct a more detailed analysis and conclude that p max ( O leave ∣ α ) ≥ most : ℝ 4 3 p max ( O stay ∣ α ) p_{\text{max}}(O_{\text{leave}}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{3}p_{\text{max}}(O_{\text{stay}}\mid\alpha) . Informally, we can retarget which items the agent prioritizes, and thereby retarget from O stay O_{\text{stay}} to O leave O_{\text{leave}} . Consider the featurization function which takes as input an observation o ∈ 𝒪 o\in\mathcal{O} : feat ​ ( o ) ≔ ( # of keys in inventory shown by o # of swords in inventory shown by o # of torches in inventory shown by o # of amulets in inventory shown by o ) . \textrm{feat}(o)\coloneqq\begin{pmatrix}\text{\# of keys in inventory shown by $o$}\\ \text{\# of swords in inventory shown by $o$}\\ \text{\# of torches in inventory shown by $o$}\\ \text{\# of amulets in inventory shown by $o$}\end{pmatrix}. (103) Consider A feat ≔ { feat ​ ( o ) ∣ o ∈ O stay } , B feat ≔ { feat ​ ( o ) ∣ o ∈ O leave } A_{\text{feat}}\coloneqq\left\{\textrm{feat}(o)\mid o\in O_{\text{stay}}\right\},B_{\text{feat}}\coloneqq\left\{\textrm{feat}(o)\mid o\in O_{\text{leave}}\right\} . Let 𝐞 i ∈ ℝ 4 \mathbf{e}_{i}\in\mathbb{R}^{4} be the standard basis vector with a 1 1 in entry i i and 0 0 elsewhere. When restricted to the room shown in fig. 2 , the agent can either acquire the key in the first room and retain it until step T T ( 𝐞 1 \mathbf{e}_{1} ), or reach time step T T empty-handed ( 𝟎 \mathbf{0} ). We conclude that A feat = { 𝐞 1 , 𝟎 } A_{\text{feat}}=\left\{\mathbf{e}_{1},\mathbf{0}\right\} . For B feat B_{\text{feat}} , recall that in section 4.2 we assumed the rollout length T T to be reasonably large. Then by leaving the room, some realizable trajectory induces o T o_{T} displaying an inventory containing only a sword ( 𝐞 2 \mathbf{e}_{2} ), or only a torch ( 𝐞 3 \mathbf{e}_{3} ), or only an amulet ( 𝐞 4 \mathbf{e}_{4} ), or nothing at all ( 𝟎 \mathbf{0} ). Therefore, { 𝐞 2 , 𝐞 3 , 𝐞 4 , 𝟎 } ⊆ B feat \left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4},\mathbf{0}\right\}\subseteq B_{\text{feat}} . B feat B_{\text{feat}} contains 3 3 copies of A feat A_{\text{feat}} ( definition A.7 ) via involutions ϕ i : 1 ↔ i \phi_{i}\mathrel{\mathop{\ordinarycolon}}1\leftrightarrow i , i ≠ 1 i\neq 1 . Suppose all feature coefficient vectors α ∈ ℝ 4 \alpha\in\mathbb{R}^{4} are plausible. Then Θ ≔ ℝ 4 \Theta\coloneqq\mathbb{R}^{4} . Let us be more specific about what is entailed by featurized reward maximization. The decide max ​ ( α ) \mathrm{decide}_{\max}(\alpha) procedure takes α \alpha as input and then considers the reward function o ↦ feat ​ ( o ) ⊤ ​ α o\mapsto\textrm{feat}(o)^{\top}\alpha . Then, decide max \mathrm{decide}_{\max} uniformly randomly chooses an observation o T ∈ O T -reach o_{T}\in O_{\text{$T$-reach}} which maximizes this featurized reward, and then uniformly randomly chooses a policy which implements o T o_{T} . ###### Lemma C.7 ( FracOptimal \mathrm{FracOptimal} inequalities) . Let X ⊆ Y ′ ⊆ Y ⊊ ℝ d X\subseteq Y^{\prime}\subseteq Y\subsetneq\mathbb{R}^{d} be finite, and let 𝐮 ∈ ℝ d \mathbf{u}\in\mathbb{R}^{d} . Then FracOptimal ⁡ ( X ∣ Y , 𝐮 ) ≤ FracOptimal ⁡ ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ⁡ ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \mathrm{FracOptimal}\left(X\mid Y,\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (104) ###### Proof. For finite X 1 ⊊ ℝ d X_{1}\subsetneq\mathbb{R}^{d} , let Best ⁡ ( X 1 ∣ 𝐮 ) ≔ arg ​ max 𝐱 1 ∈ X 1 ⁡ 𝐱 1 ⊤ ​ 𝐮 \mathrm{Best}\left(X_{1}\mid\mathbf{u}\right)\coloneqq\argmax_{\mathbf{x}_{1}\in X_{1}}\mathbf{x}_{1}^{\top}\mathbf{u} . Suppose 𝐲 ′ ∈ Best ⁡ ( Y ′ ∣ 𝐮 ) \mathbf{y}^{\prime}\in\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) , but 𝐲 ′ ∉ Best ⁡ ( Y ∣ 𝐮 ) \mathbf{y}^{\prime}\not\in\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then for all 𝐚 ∈ Best ⁡ ( Y ′ ∣ 𝐮 ) \mathbf{a}\in\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) , 𝐚 ⊤ ​ 𝐮 = 𝐲 ′ ⁣ ⊤ ​ 𝐮 < max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐮 . \mathbf{a}^{\top}\mathbf{u}=\mathbf{y}^{\prime\top}\mathbf{u}<\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{u}. (105) So 𝐚 ∉ Best ⁡ ( Y ∣ 𝐮 ) \mathbf{a}\not\in\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then either Best ⁡ ( Y ′ ∣ 𝐮 ) ⊆ Best ⁡ ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , or the two sets are disjoint. FracOptimal ⁡ ( X ∣ Y , 𝐮 ) \displaystyle\mathrm{FracOptimal}\left(X\mid Y,\mathbf{u}\right) ≔ | Best ⁡ ( Y ∣ 𝐮 ) ∩ X | | Best ⁡ ( Y ∣ 𝐮 ) | \displaystyle\coloneqq\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (106) ≤ | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | | Best ⁡ ( Y ′ ∣ 𝐮 ) | ≕ FracOptimal ⁡ ( X ∣ Y ′ , 𝐮 ) \displaystyle\leq\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|}\eqqcolon\mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right) (107) If Best ⁡ ( Y ′ ∣ 𝐮 ) ⊆ Best ⁡ ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , then since X ⊆ Y ′ X\subseteq Y^{\prime} , we have X ∩ Best ⁡ ( Y ′ ∣ 𝐮 ) = X ∩ Best ⁡ ( Y ∣ 𝐮 ) X\cap\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)=X\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right) . Then in this case, eq. 106 has equal numerator and larger denominator than eq. 107 . On the other hand, if Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ Best ⁡ ( Y ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing , then since X ⊆ Y ′ X\subseteq Y^{\prime} , X ∩ Best ⁡ ( Y ∣ 𝐮 ) = ∅ X\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing . Then eq. 106 equals 0 0 , and eq. 107 is non-negative. Either way, eq. 107 ’s inequality holds. To show the second inequality, we handle the two cases separately. ##### Subset case. Suppose that Best ⁡ ( Y ′ ∣ 𝐮 ) ⊆ Best ⁡ ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) . | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | | Best ⁡ ( Y ′ ∣ 𝐮 ) | \displaystyle\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|} ≤ | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | | Best ⁡ ( Y ′ ∣ 𝐮 ) | + | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle\leq\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (108) = | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ⁡ ( Y ′ ∣ 𝐮 ) | + | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (109) = | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ⁡ ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ⁡ ( Y ′ ∣ 𝐮 ) | + | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|+\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (110) = | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | + | Best ⁡ ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ⁡ ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (111) = | Best ⁡ ( Y ∣ 𝐮 ) ∩ X | + | Best ⁡ ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ⁡ ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X\right|+\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (112) = | Best ⁡ ( Y ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ⁡ ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (113) ≕ FracOptimal ⁡ ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \displaystyle\eqqcolon\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (114) Equation 108 follows because when n ≤ d , k ≥ 0 n\leq d,k\geq 0 , we have n d ≤ n + k d + k \frac{n}{d}\leq\frac{n+k}{d+k} . For eq. 110 , since Best ⁡ ( Y ′ ∣ 𝐮 ) ⊆ Best ⁡ ( Y ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\subseteq\mathrm{Best}\left(Y\mid\mathbf{u}\right) , we must have Best ⁡ ( Y ∣ 𝐮 ) = Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∪ Best ⁡ ( Y ′ ∣ 𝐮 ) . \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cup\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right). But then Best ⁡ ( Y ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) \displaystyle\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime}) = ( Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) ) ∪ ( Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) ) \displaystyle=\left(\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right)\cup\left(\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right) (115) = Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) . \displaystyle=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime}). (116) Then eq. 110 follows. Equation 111 follows since Best ⁡ ( Y ∣ 𝐮 ) = Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∪ Best ⁡ ( Y ′ ∣ 𝐮 ) . \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cup\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right). Equation 112 follows since X ⊆ Y ′ X\subseteq Y^{\prime} , and so Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X = Best ⁡ ( Y ∣ 𝐮 ) ∩ X . \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X=\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap X. Equation 113 follows because X ⊆ Y ′ X\subseteq Y^{\prime} is disjoint of Y ∖ Y ′ Y\setminus Y^{\prime} . We have shown that FracOptimal ⁡ ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ⁡ ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) \mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right) in this case. ##### Disjoint case. Suppose that Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ Best ⁡ ( Y ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\mid\mathbf{u}\right)=\varnothing . | Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X | | Best ⁡ ( Y ′ ∣ 𝐮 ) | \displaystyle\frac{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\right|}{\left|\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\right|} ≤ 1 \displaystyle\leq 1 (117) = | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (118) = | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( Y ∖ Y ′ ) | | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(Y\setminus Y^{\prime})\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (119) = | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\right|} (120) = | Best ⁡ ( Y ∣ 𝐮 ) ∩ ( X ∪ ( Y ∖ Y ′ ) ) | | Best ⁡ ( Y ∣ 𝐮 ) | \displaystyle=\frac{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\cap(X\cup(Y\setminus Y^{\prime}))\right|}{\left|\mathrm{Best}\left(Y\mid\mathbf{u}\right)\right|} (121) ≕ FracOptimal ⁡ ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) . \displaystyle\eqqcolon\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right). (122) Equation 117 follows because Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ X ⊆ Best ⁡ ( Y ′ ∣ 𝐮 ) \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap X\subseteq\mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right) . For eq. 120 , note that we trivially have Best ⁡ ( Y ′ ∣ 𝐮 ) ∩ Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) = ∅ \mathrm{Best}\left(Y^{\prime}\mid\mathbf{u}\right)\cap\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)=\varnothing , and also that X ⊆ Y ′ X\subseteq Y^{\prime} . Therefore, Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) ∩ X = ∅ \mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right)\cap X=\varnothing , and eq. 120 follows. Finally, the disjointness assumption implies that max 𝐲 ′ ∈ Y ′ ⁡ 𝐲 ′ ⁣ ⊤ ​ 𝐮 < max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐮 . \max_{\mathbf{y}^{\prime}\in Y^{\prime}}\mathbf{y}^{\prime\top}\mathbf{u}<\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{u}. Therefore, the optimal elements of Y Y must come exclusively from Y ∖ Y ′ Y\setminus Y^{\prime} ; i . e ., Best ⁡ ( Y ∣ 𝐮 ) = Best ⁡ ( Y ∖ Y ′ ∣ 𝐮 ) \mathrm{Best}\left(Y\mid\mathbf{u}\right)=\mathrm{Best}\left(Y\setminus Y^{\prime}\mid\mathbf{u}\right) . Then eq. 121 follows, and we have shown that FracOptimal ⁡ ( X ∣ Y ′ , 𝐮 ) ≤ FracOptimal ⁡ ( X ∪ ( Y ∖ Y ′ ) ∣ Y , 𝐮 ) \mathrm{FracOptimal}\left(X\mid Y^{\prime},\mathbf{u}\right)\leq\mathrm{FracOptimal}\left(X\cup(Y\setminus Y^{\prime})\mid Y,\mathbf{u}\right) in this case. ∎ ###### Conjecture C.8 (Generalizing Lemma C.7 ) . Lemma C.7 and Turner et al. [2021] ’s Lemma E.26 have extremely similar functional forms. How can they be unified? ###### Proposition C.9 (Featurized reward maximizers tend to leave the first room in mr ) . p max ( leave ∣ α ) ≥ most : ℝ 4 3 p max ( stay ∣ α ) . p_{\max}(\text{leave}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{3}p_{\max}(\text{stay}\mid\alpha). (123) ###### Proof. We want to show that p max ( O leave ∣ α ) ≥ most : ℝ 4 n p max ( O stay ∣ α ) p_{\text{max}}(O_{\text{leave}}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}}^{n}p_{\text{max}}(O_{\text{stay}}\mid\alpha) . Recall that A feat = { 𝐞 1 , 𝟎 } , B feat ′ ≔ { 𝐞 2 , 𝐞 3 , 𝐞 4 } ⊆ B feat A_{\text{feat}}=\{\mathbf{e}_{1},\mathbf{0}\},B^{\prime}_{\text{feat}}\coloneqq\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\}\subseteq B_{\text{feat}} . p max ​ ( stay ∣ α ) \displaystyle p_{\max}(\text{stay}\mid\alpha) (124) ≤ p max ​ ( O stay ∣ α ) \displaystyle\leq p_{\text{max}}(O_{\text{stay}}\mid\alpha) (125) ≔ ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay ) \displaystyle\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}}\right) (126) = ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ​ ( o T ) ≠ 𝟎 ) + ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ​ ( o T ) = 𝟎 ) \displaystyle=\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})\neq\mathbf{0}\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (127) ≤ FracOptimal ⁡ ( { 𝐞 1 } ∣ C feat , α ) + ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ​ ( o T ) = 𝟎 ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid C_{\text{feat}},\alpha\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (128) ≤ FracOptimal ⁡ ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) + ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ​ ( o T ) = 𝟎 ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right) (129) ≤ most : ℝ 4 > 0 3 FracOptimal ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) \displaystyle\leq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right) + ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O leave , feat ​ ( o T ) = 𝟎 ) \displaystyle\phantom{\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)}+\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}},\textrm{feat}(o_{T})=\mathbf{0}\right) (130) ≤ FracOptimal ⁡ ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∪ ( C feat ∖ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } ) ∣ C feat , α ) \displaystyle\leq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\cup(C_{\text{feat}}\setminus\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\})\mid C_{\text{feat}},\alpha\right) (131) = FracOptimal ⁡ ( C feat ∖ { 𝐞 1 } ∣ C feat , α ) \displaystyle=\mathrm{FracOptimal}\left(C_{\text{feat}}\setminus\left\{\mathbf{e}_{1}\right\}\mid C_{\text{feat}},\alpha\right) (132) ≤ ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O leave ) \displaystyle\leq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}}\right) (133) ≕ p max ​ ( O leave ∣ α ) \displaystyle\eqqcolon p_{\text{max}}(O_{\text{leave}}\mid\alpha) (134) ≤ p max ​ ( leave ∣ α ) . \displaystyle\leq p_{\max}(\text{leave}\mid\alpha). (135) Equation 124 and eq. 135 hold by Lemma C.5 . If o T ∈ O stay o_{T}\in O_{\text{stay}} is realized by p max p_{\text{max}} and feat ​ ( o T ) ≠ 𝟎 \textrm{feat}(o_{T})\neq\mathbf{0} , then we must have feat ​ ( o T ) = { 𝐞 1 } \textrm{feat}(o_{T})=\{\mathbf{e}_{1}\} be optimal and so the 𝐞 1 \mathbf{e}_{1} inventory configuration is realized. Therefore, eq. 128 follows. Equation 129 follows by applying the first inequality of Lemma C.7 with X ≔ { 𝐞 1 } , Y ′ ≔ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , Y ≔ C feat X\coloneqq\{\mathbf{e}_{1}\},Y^{\prime}\coloneqq\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\},Y\coloneqq C_{\text{feat}} . By applying Proposition A.11 ’s item 2 with A ≔ A feat = { 𝐞 1 } A\coloneqq A_{\text{feat}}=\{\mathbf{e}_{1}\} , B ′ ≔ B feat ′ = { 𝐞 2 , 𝐞 3 , 𝐞 4 } B^{\prime}\coloneqq B^{\prime}_{\text{feat}}=\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , C ≔ A ∪ B ′ C\coloneqq A\cup B^{\prime} , we have FracOptimal ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) ≤ most : ℝ 4 > 0 3 FracOptimal ⁡ ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) . \mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right)\leq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}\\ \mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right). (136) Furthermore, observe that ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O stay , feat ​ ( o T ) = 𝟎 ) ≤ ℙ π ∼ decide max ​ ( α ) , τ ∼ π | s 0 ( o T ∈ O leave , feat ​ ( o T ) = 𝟎 ) \Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{stay}},\textrm{feat}(o_{T})=\mathbf{0}\right)\leq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}_{\max}(\alpha),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in O_{\text{leave}},\textrm{feat}(o_{T})=\mathbf{0}\right) (137) because either 𝟎 \mathbf{0} is not optimal (in which case both sides equal 0), or else 𝟎 \mathbf{0} is optimal, in which case the right side is strictly greater. This can be seen by considering how decide max ​ ( α ) \mathrm{decide}_{\max}(\alpha) uniformly randomly chooses an observation in which the agent ends up with an empty inventory. As argued previously, the vast majority of such observations can only be induced by leaving the first room. Combining eq. 136 and eq. 137 , eq. 130 follows. Equation 131 follows by applying the second inequality of Lemma C.7 with X ≔ { 𝐞 2 , 𝐞 3 , 𝐞 4 } X\coloneqq\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , Y ′ ≔ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } Y^{\prime}\coloneqq\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\} , Y ≔ C feat Y\coloneqq C_{\text{feat}} . If feat ​ ( o T ) ∈ B feat \textrm{feat}(o_{T})\in B_{\text{feat}} is realized by p max p_{\text{max}} , then by the definition of B feat B_{\text{feat}} , o T ∈ O leave o_{T}\in O_{\text{leave}} is realized, and so eq. 133 follows. Then by applying Lemma B.1 with f 0 ​ ( α ) \displaystyle f_{0}(\alpha) ≔ p max ​ ( leave ∣ α ) , \displaystyle\coloneqq p_{\max}(\text{leave}\mid\alpha), (138) f 1 ​ ( α ) \displaystyle f_{1}(\alpha) ≔ FracOptimal ⁡ ( { 𝐞 1 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) , \displaystyle\coloneqq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{1}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right), (139) f 2 ​ ( α ) \displaystyle f_{2}(\alpha) ≔ FracOptimal ⁡ ( { 𝐞 2 , 𝐞 3 , 𝐞 4 } ∣ { 𝐞 1 , 𝐞 2 , 𝐞 3 , 𝐞 4 } , α ) , \displaystyle\coloneqq\mathrm{FracOptimal}\left(\left\{\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\}\mid\left\{\mathbf{e}_{1},\mathbf{e}_{2},\mathbf{e}_{3},\mathbf{e}_{4}\right\},\alpha\right), (140) f 3 ​ ( α ) \displaystyle f_{3}(\alpha) ≔ p max ​ ( stay ∣ α ) , \displaystyle\coloneqq p_{\max}(\text{stay}\mid\alpha), (141) we conclude that p max ( leave ∣ α ) ≥ most : ℝ 4 > 0 3 p max ( stay ∣ α ) . p_{\max}(\text{leave}\mid\alpha)\geq_{\text{{most}}\text{: }\mathbb{R}^{4}_{>0}}^{3}p_{\max}(\text{stay}\mid\alpha). ∎ Lastly, note that if 𝟎 ∈ Θ \mathbf{0}\in\Theta and f ⁡ ( A ∣ 𝟎 ) > f ⁡ ( B ∣ 𝟎 ) f(A\mid\mathbf{0})>f(B\mid\mathbf{0}) , f f cannot be even be simply retargetable for the Θ \Theta parameter set. This is because ∀ ϕ ∈ S d \forall\phi\in S_{d} , ϕ ⋅ 𝟎 = 𝟎 \phi\cdot\mathbf{0}=\mathbf{0} . For example, inductive bias ensures that, absent a reward signal, learned policies tend to stay in the initial room in mr . This is one reason why section 4.3 ’s analysis of the policy tendencies of reinforcement learning excludes the all-zero reward function. ### C.4 Reasoning for why dqn can’t explore well In section 4.3 , we wrote: Mnih et al. [2015] ’s dqn isn’t good enough to train policies which leave the first room of mr , and so dqn (trivially) cannot be retargetable away from the first room via the reward function. There isn’t a single featurized reward function for which dqn visits other rooms, and so we can’t have α \alpha such that ϕ ⋅ α \phi\cdot\alpha retargets the agent to O leave O_{\text{leave}} . dqn isn’t good enough at exploring. We infer this is true from Nair et al. [2015] , which shows that vanilla dqn gets zero score in mr . Thus, dqn never even gets the first key. Thus, dqn only experiences state-action-state transitions which didn’t involve acquiring an item, since (as shown in fig. 3 ) the other items are outside of the first room, which requires a key to exit. In our analysis, we considered a reward function which is featurized over item acquisition. Therefore, for all pre-key-acquisition state-action-state transitions, the featurized reward function returns exactly the same reward signals as those returned in training during the published experiments (namely, zero, because dqn can never even get to the key in order to receive a reward signal). That is, since dqn only experiences state-action-state transitions which didn’t involve acquiring an item, and the featurized reward functions only reward acquiring an item, it doesn’t matter what reward values are provided upon item acquisition— dqn ’s trained behavior will be the same. Thus, a dqn agent trained on any featurized reward function will not explore outside of the first room. ## Appendix D Lower bounds on mdp power-seeking incentives for optimal policies Turner et al. [2021] prove conditions under which at least half of the orbit of every reward function incentivizes power-seeking behavior. For example, in fig. 4 , they prove that avoiding ∅ \varnothing maximizes average per-timestep reward for at least half of reward functions. Roughly, there are more self-loop states ( ∅ \varnothing , ℓ ↙ \ell_{\swarrow} , r ↘ r_{\searrow} , r ↗ r_{\nearrow} ) available if the agent goes left or right instead of up towards ∅ \varnothing . We strengthen this claim, with Corollary D.12 showing that for at least three-quarters of the orbit of every reward function, it is average-optimal to avoid ∅ \varnothing . Therefore, we answer Turner et al. [2021] ’s open question of whether increased number of environmental symmetries quantitatively strengthens the degree to which power-seeking is incentivized. The answer is yes . In particular, it may be the case that only one in a million state-based reward functions makes it average-optimal for Pac-Man to die immediately. We will briefly restate several definitions needed for our key results, Theorem D.11 and Corollary D.12 . For explanation, see Turner et al. [2021] . ###### Definition D.1 (Non-dominated linear functionals) . Let X ⊊ ℝ | 𝒮 | X\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} be finite. ND ​ ( X ) ≔ { 𝐱 ∈ X ∣ ∃ 𝐫 ∈ ℝ | 𝒮 | : 𝐱 ⊤ ​ 𝐫 > max 𝐱 ′ ∈ X ∖ { 𝐱 } ⁡ 𝐱 ′ ⁣ ⊤ ​ 𝐫 } \text{{ND}}\left(X\right)\coloneqq\left\{\mathbf{x}\in X\mid\exists\mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|}\mathrel{\mathop{\ordinarycolon}}\mathbf{x}^{\top}\mathbf{r}>\max_{\mathbf{x}^{\prime}\in X\setminus\left\{\mathbf{x}\right\}}\mathbf{x}^{\prime\top}\mathbf{r}\right\} . ###### Definition D.2 (Bounded reward function distribution) . 𝔇 bound \mathfrak{D}_{\text{bound}} is the set of bounded-support probability distributions 𝒟 bound \mathcal{D}_{\text{bound}} . ###### Remark . When n = 1 n=1 , Lemma D.3 reduces to the first part of Turner et al. [2021] ’s lemma E.24, and Lemma D.5 reduces to the first part of Turner et al. [2021] ’s lemma E.28. ###### Lemma D.3 (Quantitative expectation superiority lemma) . Let A , B ⊊ ℝ d A,B\subsetneq\mathbb{R}^{d} be finite and let g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} be a (total) increasing function. Suppose B B contains n n copies of ND ​ ( A ) \text{{ND}}\left(A\right) . Then 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐛 ∈ B 𝐛 ⊤ 𝐫 ) ] ≥ most : 𝔇 bound n 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐚 ∈ A 𝐚 ⊤ 𝐫 ) ] . \displaystyle\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{bound}}}^{n}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\right)\right]. (142) ###### Proof. Because g : ℝ → ℝ g\mathrel{\mathop{\ordinarycolon}}\mathbb{R}\to\mathbb{R} is increasing, it is measurable (as is max \max ). Let L ≔ inf 𝐫 ∈ supp ⁡ ( 𝒟 bound ) max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 , U ≔ sup 𝐫 ∈ supp ⁡ ( 𝒟 bound ) max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐫 L\coloneqq\inf_{\mathbf{r}\in\optSupp(\mathcal{D}_{\text{bound}})}\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r},U\coloneqq\sup_{\mathbf{r}\in\optSupp(\mathcal{D}_{\text{bound}})}\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{r} . Both exist because 𝒟 bound \mathcal{D}_{\text{bound}} has bounded support. Furthermore, since g g is monotone increasing, it is bounded [ g ⁡ ( L ) , g ⁡ ( U ) ] [g(L),g(U)] on [ L , U ] [L,U] . Therefore, g g is measurable and bounded on each supp ⁡ ( 𝒟 bound ) \optSupp(\mathcal{D}_{\text{bound}}) , and so the relevant expectations exist for all 𝒟 bound \mathcal{D}_{\text{bound}} . For finite X ⊊ ℝ d X\subsetneq\mathbb{R}^{d} , let f ⁡ ( X ∣ 𝐮 ) ≔ g ⁡ ( max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ) f(X\mid\mathbf{u})\coloneqq g(\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}) . By Lemma B.11 , f f is invariant under joint permutation by S d S_{d} . Furthermore, f f is measurable because g g and max \max are. Therefore, apply Lemma B.5 to conclude that f ⁡ ( X ∣ 𝒟 bound ) ≔ 𝔼 𝐮 ∼ 𝒟 bound [ g ⁡ ( max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ) ] f(X\mid\mathcal{D}_{\text{bound}})\coloneqq\opE_{\mathbf{u}\sim\mathcal{D}_{\text{bound}}}\left[g(\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u})\right] is also invariant under joint permutation by S d S_{d} (with f f being bounded when restricted to supp ⁡ ( 𝒟 bound ) \optSupp(\mathcal{D}_{\text{bound}}) ). Lastly, if X ′ ⊆ X X^{\prime}\subseteq X , f ⁡ ( X ′ ∣ 𝒟 bound ) ≤ f ⁡ ( X ∣ 𝒟 bound ) f(X^{\prime}\mid\mathcal{D}_{\text{bound}})\leq f(X\mid\mathcal{D}_{\text{bound}}) because g g is increasing. 𝔼 𝐮 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐮 ) ] \displaystyle\opE_{\mathbf{u}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{u}\right)\right] = 𝔼 𝐮 ∼ 𝒟 bound [ g ⁡ ( max 𝐚 ∈ ND ​ ( A ) ⁡ 𝐚 ⊤ ​ 𝐮 ) ] \displaystyle=\opE_{\mathbf{u}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{a}\in\text{{ND}}\left(A\right)}\mathbf{a}^{\top}\mathbf{u}\right)\right] (143) ≤ most : 𝔇 any n 𝔼 𝐫 ∼ 𝒟 bound [ g ( max 𝐛 ∈ B 𝐛 ⊤ 𝐫 ) ] . \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\opE_{\mathbf{r}\sim\mathcal{D}_{\text{bound}}}\left[g\left(\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right)\right]. (144) Equation 143 follows by corollary E.11 of [ Turner et al., 2021 ] . Equation 144 follows by applying Lemma B.9 with f f as defined above with the ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} guaranteed by the copy assumption. ∎ ###### Definition D.4 (Linear functional optimality probability [ Turner et al., 2021 ] ) . For finite A , B ⊊ ℝ | 𝒮 | A,B\subsetneq\mathbb{R}^{\left|\mathcal{S}\right|} , the probability under 𝒟 any \mathcal{D}_{\text{any}} that A A is optimal over B B is p 𝒟 any ​ ( A ≥ B ) ≔ ℙ 𝐫 ∼ 𝒟 any ( max 𝐚 ∈ A ⁡ 𝐚 ⊤ ​ 𝐫 ≥ max 𝐛 ∈ B ⁡ 𝐛 ⊤ ​ 𝐫 ) . p_{\mathcal{D}_{\text{any}}}\left(A\geq B\right)\coloneqq\Prb\nolimits_{\mathbf{r}\sim\mathcal{D}_{\text{any}}}\left(\max_{\mathbf{a}\in A}\mathbf{a}^{\top}\mathbf{r}\geq\max_{\mathbf{b}\in B}\mathbf{b}^{\top}\mathbf{r}\right). ###### Lemma D.5 (Quantitative optimality probability superiority lemma) . Let A , B , C ⊊ ℝ d A,B,C\subsetneq\mathbb{R}^{d} be finite and let Z Z satisfy ND ​ ( C ) ⊆ Z ⊆ C \text{{ND}}\left(C\right)\subseteq Z\subseteq C . Suppose that B B contains n n copies of ND ​ ( A ) \text{{ND}}\left(A\right) via involutions ϕ i \phi_{i} . Furthermore, let B extra ≔ B ∖ ( ∪ i = 1 n ϕ i ⋅ ND ( A ) ) B_{\text{extra}}\coloneqq B\setminus\left(\cup_{i=1}^{n}\phi_{i}\cdot\text{{ND}}\left(A\right)\right) ; suppose that for all i i , ϕ i ⋅ ( Z ∖ B extra ) = Z ∖ B extra \phi_{i}\cdot\left(Z\setminus B_{\text{extra}}\right)=Z\setminus B_{\text{extra}} . Then p 𝒟 any ( B ≥ C ) ≥ most : 𝔇 any n p 𝒟 any ( A ≥ C ) p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right) . ###### Proof. For finite X , Y ⊊ ℝ d X,Y\subsetneq\mathbb{R}^{d} , let g ⁡ ( X , Y ∣ 𝒟 any ) ≔ p 𝒟 any ​ ( X ≥ Y ) = 𝔼 𝐮 ∼ 𝒟 any [ 𝟙 max 𝐱 ∈ X ⁡ 𝐱 ⊤ ​ 𝐮 ≥ max 𝐲 ∈ Y ⁡ 𝐲 ⊤ ​ 𝐮 ] . g(X,Y\mid\mathcal{D}_{\text{any}})\coloneqq p_{\mathcal{D}_{\text{any}}}\left(X\geq Y\right)=\opE_{\mathbf{u}\sim\mathcal{D}_{\text{any}}}\left[\mathbbm{1}_{\max_{\mathbf{x}\in X}\mathbf{x}^{\top}\mathbf{u}\geq\max_{\mathbf{y}\in Y}\mathbf{y}^{\top}\mathbf{u}}\right]. By the proof of item 1 of Proposition A.11 , g g is the expectation of a 𝐮 \mathbf{u} -measurable function. g g is an EU function, and so Lemma B.11 shows that it is invariant to joint permutation by ϕ i \phi_{i} . Letting f Y ​ ( X ∣ 𝒟 any ) ≔ g ⁡ ( X , Y ∣ 𝒟 any ) f_{Y}(X\mid\mathcal{D}_{\text{any}})\coloneqq g(X,Y\mid\mathcal{D}_{\text{any}}) , Lemma B.10 shows that f Y ​ ( X ∣ 𝒟 any ) = f Y ​ ( ϕ i ⋅ X ∣ ϕ i ⋅ 𝒟 any ) f_{Y}(X\mid\mathcal{D}_{\text{any}})=f_{Y}(\phi_{i}\cdot X\mid\phi_{i}\cdot\mathcal{D}_{\text{any}}) whenever the ϕ i \phi_{i} satisfy ϕ i ⋅ Y = Y \phi_{i}\cdot Y=Y . Furthermore, if X ′ ⊆ X X^{\prime}\subseteq X , then f Y ​ ( X ′ ∣ 𝒟 any ) ≤ f Y ​ ( X ∣ 𝒟 any ) f_{Y}(X^{\prime}\mid\mathcal{D}_{\text{any}})\leq f_{Y}(X\mid\mathcal{D}_{\text{any}}) . p 𝒟 any ​ ( A ≥ C ) \displaystyle p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right) = p 𝒟 any ​ ( ND ​ ( A ) ≥ C ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(\text{{ND}}\left(A\right)\geq C\right) (145) ≤ p 𝒟 any ​ ( ND ​ ( A ) ≥ Z ∖ B extra ) \displaystyle\leq p_{\mathcal{D}_{\text{any}}}\left(\text{{ND}}\left(A\right)\geq Z\setminus B_{\text{extra}}\right) (146) ≤ most : 𝔇 any n p 𝒟 any ( B ≥ Z ∖ B extra ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}p_{\mathcal{D}_{\text{any}}}\left(B\geq Z\setminus B_{\text{extra}}\right) (147) ≤ p 𝒟 any ​ ( B ∪ B extra ≥ Z ) \displaystyle\leq p_{\mathcal{D}_{\text{any}}}\left(B\cup B_{\text{extra}}\geq Z\right) (148) = p 𝒟 any ​ ( B ≥ Z ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(B\geq Z\right) (149) = p 𝒟 any ​ ( B ≥ C ) . \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right). (150) Equation 145 follows by Turner et al. [2021] ’s lemma E.12’s item 2 with X ≔ A X\coloneqq A , X ′ ≔ ND ​ ( A ) X^{\prime}\coloneqq\text{{ND}}\left(A\right) (similar reasoning holds for C C and Z Z in eq. 150 ). Equation 146 follows by the first inequality of lemma E.26 of [ Turner et al., 2021 ] with X ≔ A , Y ≔ C , Y ′ ≔ Z ∖ B extra X\coloneqq A,Y\coloneqq C,Y^{\prime}\coloneqq Z\setminus B_{\text{extra}} . Equation 147 follows by applying Lemma B.9 with the f Z ∖ B extra f_{Z\setminus B_{\text{extra}}} defined above. Equation 148 follows by the second inequality of lemma E.26 of [ Turner et al., 2021 ] with X ≔ A , Y ≔ Z , Y ′ ≔ Z ∖ B extra X\coloneqq A,Y\coloneqq Z,Y^{\prime}\coloneqq Z\setminus B_{\text{extra}} . Equation 149 follows because B extra ⊆ B B_{\text{extra}}\subseteq B . Letting f 0 ​ ( 𝒟 any ) ≔ p 𝒟 any ​ ( A ≥ C ) , f 1 ​ ( 𝒟 any ) ≔ p 𝒟 any ​ ( ND ​ ( A ) ≥ Z ∖ B extra ) , f 2 ​ ( 𝒟 any ) ≔ p 𝒟 any ​ ( B ≥ Z ∖ B extra ) , f 3 ​ ( 𝒟 any ) ≔ p 𝒟 any ​ ( B ≥ C ) f_{0}(\mathcal{D}_{\text{any}})\coloneqq p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right),f_{1}(\mathcal{D}_{\text{any}})\coloneqq p_{\mathcal{D}_{\text{any}}}\left(\text{{ND}}\left(A\right)\geq Z\setminus B_{\text{extra}}\right),f_{2}(\mathcal{D}_{\text{any}})\coloneqq p_{\mathcal{D}_{\text{any}}}\left(B\geq Z\setminus B_{\text{extra}}\right),f_{3}(\mathcal{D}_{\text{any}})\coloneqq p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right) , apply Lemma B.1 to conclude that p 𝒟 any ( A ≥ C ) ≤ most : 𝔇 any n p 𝒟 any ( B ≥ C ) . p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right). ∎ ###### Definition D.6 (Rewardless mdp [ Turner et al., 2021 ] ) . ⟨ 𝒮 , 𝒜 , T ⟩ \langle\mathcal{S},\mathcal{A},T\rangle is a rewardless mdp with finite state and action spaces 𝒮 \mathcal{S} and 𝒜 \mathcal{A} , and stochastic transition function T : 𝒮 × 𝒜 → Δ ⁡ ( 𝒮 ) T\mathrel{\mathop{\ordinarycolon}}\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) . We treat the discount rate γ \gamma as a variable with domain [ 0 , 1 ] [0,1] . ###### Definition D.7 (1-cycle states [ Turner et al., 2021 ] ) . Let 𝐞 s ∈ ℝ | 𝒮 | \mathbf{e}_{s}\in\mathbb{R}^{\left|\mathcal{S}\right|} be the standard basis vector for state s s , such that there is a 1 1 in the entry for state s s and 0 0 elsewhere. State s s is a 1-cycle if ∃ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \exists a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} . State s s is a terminal state if ∀ a ∈ 𝒜 : T ⁡ ( s , a ) = 𝐞 s \forall a\in\mathcal{A}\mathrel{\mathop{\ordinarycolon}}T(s,a)=\mathbf{e}_{s} . ###### Definition D.8 (State visit distribution [ Sutton and Barto, 1998 ] ) . Π ≔ 𝒜 𝒮 \Pi\coloneqq\mathcal{A}^{\mathcal{S}} , the set of stationary deterministic policies. The visit distribution induced by following policy π \pi from state s s at discount rate γ ∈ [ 0 , 1 ) \gamma\in[0,1) is 𝐟 π , s ​ ( γ ) ≔ ∑ t = 0 ∞ γ t ​ 𝔼 s t ∼ π | s [ 𝐞 s t ] \mathbf{f}^{\pi,s}(\gamma)\coloneqq\sum_{t=0}^{\infty}\gamma^{t}\opE_{s_{t}\sim\pi\mid s}\left[\mathbf{e}_{s_{t}}\right] . 𝐟 π , s \mathbf{f}^{\pi,s} is a visit distribution function ; ℱ ⁡ ( s ) ≔ { 𝐟 π , s ∣ π ∈ Π } \Fop(s)\coloneqq\{\mathbf{f}^{\pi,s}\mid\pi\in\Pi\} . ###### Definition D.9 (Recurrent state distributions [ Puterman, 2014 ] ) . The recurrent state distributions which can be induced from state s s are RSD ​ ( s ) ≔ { lim γ → 1 ( 1 − γ ) ​ 𝐟 π , s ​ ( γ ) ∣ π ∈ Π } \text{{RSD}}\left(s\right)\coloneqq\left\{\lim_{\gamma\to 1}(1-\gamma)\mathbf{f}^{\pi,s}(\gamma)\mid\pi\in\Pi\right\} . RSD nd ​ ( s ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right) is the set of rsd s which strictly maximize average reward for some reward function. ###### Definition D.10 (Average-optimal policies [ Turner et al., 2021 ] ) . The average-optimal policy set for reward function R R is Π avg ​ ( R ) ≔ { π ∈ Π ∣ ∀ s ∈ 𝒮 : 𝐝 π , s ∈ arg ​ max 𝐝 ∈ RSD ​ ( s ) ⁡ 𝐝 ⊤ ​ 𝐫 } \Pi^{\text{avg}}\left(R\right)\coloneqq\left\{\pi\in\Pi\mid\forall s\in\mathcal{S}\mathrel{\mathop{\ordinarycolon}}\mathbf{d}^{\pi,s}\in\argmax_{\mathbf{d}\in\text{{RSD}}\left(s\right)}\mathbf{d}^{\top}\mathbf{r}\right\} (the policies which induce optimal rsd s at all states). For D ⊆ RSD ​ ( s ) D\subseteq\text{{RSD}}\left(s\right) , the average optimality probability is ℙ 𝒟 any ( D , average ) ≔ ℙ R ∼ 𝒟 any ( ∃ 𝐝 π , s ∈ D : π ∈ Π avg ​ ( R ) ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right)\coloneqq\Prb\nolimits_{R\sim\mathcal{D}_{\text{any}}}\left(\exists\mathbf{d}^{\pi,s}\in D\mathrel{\mathop{\ordinarycolon}}\pi\in\Pi^{\text{avg}}\left(R\right)\right) . ###### Remark . Theorem D.11 generalizes the first claim of Turner et al. [2021] ’s theorem 6.13, and Corollary D.12 generalizes the first claim of Turner et al. [2021] ’s corollary 6.14. ###### Theorem D.11 (Quantitatively, average-optimal policies tend to end up in “larger” sets of rsd s) . Let D ′ , D ⊆ RSD ​ ( s ) D^{\prime},D\subseteq\text{{RSD}}\left(s\right) . Suppose that D D contains n n copies of D ′ D^{\prime} and that the sets D ′ ∪ D D^{\prime}\cup D and RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\left(D^{\prime}\cup D\right) have pairwise orthogonal vector elements (i.e., pairwise disjoint vector support). Then ℙ 𝒟 any ( D ′ , average ) ≤ most : 𝔇 any n ℙ 𝒟 any ( D , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right) . ###### Proof. Let D i ≔ ϕ i ⋅ D ′ D_{i}\coloneqq\phi_{i}\cdot D^{\prime} , where D i ⊆ D D_{i}\subseteq D by assumption. Let S ≔ { s ′ ∈ 𝒮 ∣ max 𝐝 ∈ D ′ ∪ D ⁡ 𝐝 ⊤ ​ 𝐞 s ′ > 0 } S\coloneqq\left\{s^{\prime}\in\mathcal{S}\mid\max_{\mathbf{d}\in D^{\prime}\cup D}\mathbf{d}^{\top}\mathbf{e}_{s^{\prime}}>0\right\} . Define ϕ i ′ ​ ( s ′ ) ≔ { ϕ i ​ ( s ′ ) if ​ s ′ ∈ S s ′ else . \phi_{i}^{\prime}(s^{\prime})\coloneqq\begin{cases}\phi_{i}(s^{\prime})&\text{ if }s^{\prime}\in S\\ s^{\prime}&\text{ else}.\end{cases} (151) Since ϕ i \phi_{i} is an involution, ϕ i ′ \phi_{i}^{\prime} is also an involution. Furthermore, ϕ i ′ ⋅ D ′ = D i \phi_{i}^{\prime}\cdot D^{\prime}=D_{i} , ϕ i ′ ⋅ D i = D ′ \phi_{i}^{\prime}\cdot D_{i}=D^{\prime} , and ϕ i ′ ⋅ D j = D j \phi_{i}^{\prime}\cdot D_{j}=D_{j} for j ≠ i j\neq i because we assumed that these equalities hold for ϕ i \phi_{i} , and D ′ , D i , D j ⊆ D ′ ∪ D D^{\prime},D_{i},D_{j}\subseteq D^{\prime}\cup D and so the vectors of these sets have support contained in S S . Let D ∗ ≔ D ′ ∪ i = 1 n D i ∪ ( RSD nd ( s ) ∖ ( D ′ ∪ D ) ) D^{*}\coloneqq D^{\prime}\cup_{i=1}^{n}D_{i}\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\left(D^{\prime}\cup D\right)\right) . By an argument mirroring that in the proof of theorem 6.13 in Turner et al. [2021] and using the fact that ϕ i ′ ⋅ D j = D j \phi_{i}^{\prime}\cdot D_{j}=D_{j} for all i ≠ j i\neq j , ϕ i ′ ⋅ D ∗ = D ∗ \phi_{i}^{\prime}\cdot D^{*}=D^{*} . Consider Z ≔ ( RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ) ∪ D ′ ∪ D Z\coloneqq\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\right)\cup D^{\prime}\cup D . First, Z ⊆ RSD ​ ( s ) Z\subseteq\text{{RSD}}\left(s\right) by definition. Second, RSD nd ​ ( s ) = RSD nd ​ ( s ) ∖ ( D ′ ∪ D ) ∪ ( RSD nd ​ ( s ) ∩ D ′ ) ∪ ( RSD nd ​ ( s ) ∩ D ) ⊆ Z \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)=\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus(D^{\prime}\cup D)\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D^{\prime}\right)\cup\left(\text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\cap D\right)\subseteq Z . Note that D ∗ = Z ∖ ( D ∖ ∪ i = 1 n D i ) D^{*}=Z\setminus(D\setminus\cup_{i=1}^{n}D_{i}) . ℙ 𝒟 any ( D ′ , average ) \displaystyle\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right) = p 𝒟 any ​ ( D ′ ≥ RSD ​ ( s ) ) \displaystyle=p_{\mathcal{D}_{\text{any}}}\left(D^{\prime}\geq\text{{RSD}}\left(s\right)\right) (152) ≤ most : 𝔇 any n p 𝒟 any ( D ≥ RSD ( s ) ) \displaystyle\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}p_{\mathcal{D}_{\text{any}}}\left(D\geq\text{{RSD}}\left(s\right)\right) (153) = ℙ 𝒟 any ( D , average ) . \displaystyle=\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D,\text{average}\right). (154) Since ϕ i ′ ⋅ D ′ ⊆ D \phi_{i}^{\prime}\cdot D^{\prime}\subseteq D and ND ​ ( D ′ ) ⊆ D ′ \text{{ND}}\left(D^{\prime}\right)\subseteq D^{\prime} , ϕ i ′ ⋅ ND ​ ( D ′ ) ⊆ D \phi_{i}^{\prime}\cdot\text{{ND}}\left(D^{\prime}\right)\subseteq D and so D D contains n n copies of ND ​ ( D ′ ) \text{{ND}}\left(D^{\prime}\right) via involutions ϕ i ′ \phi_{i}^{\prime} . Then eq. 153 holds by applying Lemma D.5 with A ≔ D ′ A\coloneqq D^{\prime} , B i ≔ D i B_{i}\coloneqq D_{i} for all i = 1 , … , n i=1,\ldots,n , B ≔ D , C ≔ RSD ​ ( s ) B\coloneqq D,C\coloneqq\text{{RSD}}\left(s\right) , Z Z as defined above, and involutions ϕ i ′ \phi_{i}^{\prime} which satisfy ϕ i ′ ⋅ ( Z ∖ ( B ∖ ∪ i = 1 n B i ) ) = ϕ i ′ ⋅ D ∗ = D ∗ = Z ∖ ( B ∖ ∪ i = 1 n B i ) \phi_{i}^{\prime}\cdot\left(Z\setminus(B\setminus\cup_{i=1}^{n}B_{i})\right)=\phi_{i}^{\prime}\cdot D^{*}=D^{*}=Z\setminus(B\setminus\cup_{i=1}^{n}B_{i}) . ∎ ###### Corollary D.12 (Quantitatively, average-optimal policies tend not to end up in any given 1-cycle) . Let D ′ ≔ { 𝐞 s 1 ′ , … , 𝐞 s k ′ } , D r ≔ { 𝐞 s 1 , … , 𝐞 s n ⋅ k } ⊆ RSD ​ ( s ) D^{\prime}\coloneqq\left\{\mathbf{e}_{s_{1}^{\prime}},\ldots,\mathbf{e}_{s_{k}^{\prime}}\right\},D_{r}\coloneqq\left\{\mathbf{e}_{s_{1}},\ldots,\mathbf{e}_{s_{n\cdot k}}\right\}\subseteq\text{{RSD}}\left(s\right) be disjoint, for n ≥ 1 , k ≥ 1 n\geq 1,k\geq 1 . Then ℙ 𝒟 any ( D ′ , average ) ≤ most : 𝔇 any n ℙ 𝒟 any ( RSD ( s ) ∖ D ′ , average ) \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus D^{\prime},\text{average}\right) . ###### Proof. For each i ∈ { 1 , … , n } i\in\left\{1,\ldots,n\right\} , let ϕ i \displaystyle\phi_{i} ≔ ( s 1 ′ s ( i − 1 ) ⋅ k + 1 ) ⋯ ( s k ′ s ( i − 1 ) ⋅ k + k ) , \displaystyle\coloneqq(s_{1}^{\prime}\,\,\,s_{(i-1)\cdot k+1})\cdots(s_{k}^{\prime}\,\,\,s_{(i-1)\cdot k+k}), D i \displaystyle D_{i} ≔ { 𝐞 s ( i − 1 ) ⋅ k + 1 , … , 𝐞 s ( i − 1 ) ⋅ k + k } , \displaystyle\coloneqq\left\{\mathbf{e}_{s_{(i-1)\cdot k+1}},\ldots,\mathbf{e}_{s_{(i-1)\cdot k+k}}\right\}, D \displaystyle D ≔ RSD ​ ( s ) ∖ D ′ . \displaystyle\coloneqq\text{{RSD}}\left(s\right)\setminus D^{\prime}. Each D i ⊆ D r ⊆ RSD ​ ( s ) ∖ D ′ D_{i}\subseteq D_{r}\subseteq\text{{RSD}}\left(s\right)\setminus D^{\prime} by disjointness of D ′ D^{\prime} and D r D_{r} . D D contains n n copies of D ′ D^{\prime} via involutions ϕ 1 , … , ϕ n \phi_{1},\ldots,\phi_{n} . D ′ ∪ D = RSD ​ ( s ) D^{\prime}\cup D=\text{{RSD}}\left(s\right) and RSD nd ​ ( s ) ∖ RSD ​ ( s ) = ∅ \text{{RSD}}{\text{\textsubscript{nd}}}\left(s\right)\setminus\text{{RSD}}\left(s\right)=\emptyset trivially have pairwise orthogonal vector elements. Apply Theorem D.11 to conclude that ℙ 𝒟 any ( D ′ , average ) ≤ most : 𝔇 any n ℙ 𝒟 any ( RSD ( s ) ∖ D ′ , average ) . \Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(D^{\prime},\text{average}\right)\leq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{n}\Prb\nolimits_{\mathcal{D}_{\text{any}}}\left(\text{{RSD}}\left(s\right)\setminus D^{\prime},\text{average}\right). ∎ Let A ≔ { 𝐞 1 , 𝐞 2 } , B ⊆ ℝ 5 A\coloneqq\left\{\mathbf{e}_{1},\mathbf{e}_{2}\right\},B\subseteq\mathbb{R}^{5} , C ≔ A ∪ B C\coloneqq A\cup B . Conjecture D.13 conjectures that e . g ., p 𝒟 ′ ( B ≥ C ) ≥ most : 𝔇 any 3 2 p 𝒟 ′ ( A ≥ C ) . p_{\mathcal{D}^{\prime}}\left(B\geq C\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{\frac{3}{2}}p_{\mathcal{D}^{\prime}}\left(A\geq C\right). ###### Conjecture D.13 (Fractional quantitative optimality probability superiority lemma) . Let A A , B B , C ⊊ ℝ d C\subsetneq\mathbb{R}^{d} be finite. If A = ⋃ j = 1 m A j A=\bigcup_{j=1}^{m}A_{j} and ⋃ i = 1 n B i ⊆ B \bigcup_{i=1}^{n}B_{i}\subseteq B such that for each A j A_{j} , B B contains n n copies ( B 1 , … , B n B_{1},\ldots,B_{n} ) of A j A_{j} via involutions ϕ j ​ i \phi_{ji} which also fix ϕ j ​ i ⋅ A j ′ = A j ′ \phi_{ji}\cdot A_{j^{\prime}}=A_{j^{\prime}} for j ′ ≠ j j^{\prime}\neq j , then p 𝒟 any ( B ≥ C ) ≥ most : 𝔇 any n m p 𝒟 any ( A ≥ C ) . p_{\mathcal{D}_{\text{any}}}\left(B\geq C\right)\geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}}^{\frac{n}{m}}p_{\mathcal{D}_{\text{any}}}\left(A\geq C\right). We suspect that any proof of the conjecture should generalize Lemma B.7 to the fractional set copy containment case. ## Instructions for reporting errors We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below: Click the "Report Issue" ( ) button, located in the page header. Tip: You can select the relevant text first, to include it in your report. Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all. Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .