Parametrically Retargetable Decision-Makers Tend To Seek Power — part 3/13
Cleaned-text code points [30540, 45943). Bibliography omitted here; source.md includes it.
With that in mind, we now analyze how often decision-makers leave the first room of mr . 4 4 4 In Appendix C.2 , Figure 3 shows a map of the first level. Decision-making functions decide ( θ ) \mathrm{decide}(\theta) produce a probability distribution over policies π ∈ Π \pi\in\Pi , which are rolled out from the initial state s 0 s_{0} to produce observation-action trajectories τ = o 0 a 0 … o T a T … \tau=o_{0}a_{0}\ldots o_{T}a_{T}\ldots , where T T is the rollout length we are interested in. Let O T -reach O_{\text{$T$-reach}} be the set of observations reachable starting from state s 0 s_{0} and acting for T T time steps, let O leave ⊆ O T -reach O_{\text{leave}}\subseteq O_{\text{$T$-reach}} be those observations which can only be realized by leaving, and let O stay ≔ O T -reach ∖ O leave O_{\text{stay}}\coloneqq O_{\text{$T$-reach}}\setminus O_{\text{leave}} . Consider the probability that decide \mathrm{decide} realizes some subset of observations X ⊆ 𝒪 X\subseteq\mathcal{O} at step T T : p decide ( X ∣ θ ) ≔ ℙ π ∼ decide ( θ ) , τ ∼ π | s 0 ( o T ∈ X ) . p_{\mathrm{decide}}(X\mid\theta)\coloneqq\Prb_{\begin{subarray}{c}\pi\sim\mathrm{decide}(\theta),\\ \tau\sim\pi\mid s_{0}\end{subarray}}\left(o_{T}\in X\right). (3) Let Θ ≔ ℝ 𝒪 \Theta\coloneqq\mathbb{R}^{\mathcal{O}} be the set of reward functions mapping observations o ∈ 𝒪 o\in\mathcal{O} to real numbers, and let T ≔ 1,000 T\coloneqq 1{,}000 . We first consider the previous decision functions, since they are simple to analyze.
decide rand \mathrm{decide}_{\text{rand}} randomly chooses a final observation o o which can be realized at step 1,000, and then chooses some policy which realizes o o . 5 5 5 decide rand \mathrm{decide}_{\text{rand}} does not act randomly at each time step, it induces a randomly selected final observation. Analogously, randomly turning a steering wheel is different from driving to a randomly chosen destination. decide rand \mathrm{decide}_{\text{rand}} induces an p rand p_{\text{rand}} defined by eq. 3 . As before, p rand p_{\text{rand}} tends to leave the room under all parameter inputs.
decide max ( θ ) \mathrm{decide}_{\max}(\theta) produces a policy which maximizes the reward of the observation at step 1,000 of the rollout. Since mr is deterministic, we discuss which observation decide max ( θ ) \mathrm{decide}_{\max}(\theta) realizes. In a stochastic setting, the decision-maker would choose a policy realizing some probability distribution over step- T T observations, and the analysis would proceed similarly.
Here is the semi-formal argument for p max p_{\text{max}} ’s retargetability. There are combinatorially more game-screens visible if the agent leaves the room (due to e . g ., more point combinations, more inventory layouts, more screens outside of the first room). In other words, | O stay | ≪ | O leave | \left|O_{\text{stay}}\right|\ll\left|O_{\text{leave}}\right| . There are more ways for the selected observation to require leaving the room, than not. Thus, p max p_{\text{max}} is extremely retargetable from O stay O_{\text{stay}} to O leave O_{\text{leave}} .
Detailed analysis in section C.2 confirms that p max ( O leave ∣ θ ) ≥ most : Θ n p max ( O stay ∣ θ ) p_{\text{max}}(O_{\text{leave}}\mid\theta)\geq_{\text{{most}}\text{: }\Theta}^{n}p_{\text{max}}(O_{\text{stay}}\mid\theta) for the large n ≔ ⌊ | O leave | | O stay | ⌋ n\coloneqq\lfloor\frac{\left|O_{\text{leave}}\right|}{\left|O_{\text{stay}}\right|}\rfloor , which we show implies that p max p_{\text{max}} tends to leave the room.
### 4.3 Tendencies for rl on featurized reward over the final observation
In the real world, we do not run p max p_{\text{max}} , which can be computed via T T -depth exhaustive tree search in order to find and induce a maximal-reward observation o T o_{T} . Instead, we use reinforcement learning. Better rl algorithms seem to be more retargetable because of their greater capability to explore. 6 6 6 Conversely, if the agent cannot figure out how to leave the first room, any reward signal from outside of the first room can never causally affect the learned policy. In that case, retargetability away from the first room is impossible.
##### Exploring the first room.
Consider a featurized reward function over observations θ ∈ ℝ 𝒪 \theta\in\mathbb{R}^{\mathcal{O}} , which provides an end-of-episode return signal which adds a fixed reward for each item displayed in the observation ( e . g ., 5 reward for a sword, 2 reward for a key). Consider a coefficient vector α ∈ ℝ 4 \alpha\in\mathbb{R}^{4} , with each entry denoting the value of an item, and feat : 𝒪 → ℝ 4 \textrm{feat}\mathrel{\mathop{\ordinarycolon}}\mathcal{O}\to\mathbb{R}^{4} maps observations to feature vectors which tally the items in the agent’s inventory. A reinforcement learning algorithm Alg \mathrm{Alg} uses this return signal to update a fixed-initialization policy network. Then p Alg ( O leave ∣ θ ) p_{\mathrm{Alg}}(O_{\text{leave}}\mid\theta) returns the probability that Alg \mathrm{Alg} trains an policy whose step- T T observation required the agent to leave the initial room.
The retargetability ( definition 3.3 ) of Alg \mathrm{Alg} is closely linked to the quality of Alg \mathrm{Alg} as an rl training procedure. For example, as explained in section C.4 , Mnih et al. [2015] ’s dqn isn’t good enough to train policies which leave the first room of mr , and so dqn (trivially) cannot be retargetable away from the first room via the reward function. There isn’t a single featurized reward function for which dqn visits other rooms, and so we can’t have α \alpha such that ϕ ⋅ α \phi\cdot\alpha retargets the agent to O leave O_{\text{leave}} . dqn isn’t good enough at exploring.
More formally, in this situation, Alg \mathrm{Alg} is retargetable if there exists a permutation ϕ ∈ S 4 \phi\in S_{4} such that whenever α ∈ Θ ≔ ℝ 4 \alpha\in\Theta\coloneqq\mathbb{R}^{4} induces the learned policies to stay in the room ( p Alg ( O stay ∣ α ) > p Alg ( O leave ∣ α ) p_{\mathrm{Alg}}(O_{\text{stay}}\mid\alpha)>p_{\mathrm{Alg}}(O_{\text{leave}}\mid\alpha) ), ϕ ⋅ α \phi\cdot\alpha makes Alg \mathrm{Alg} train policies which leave the room ( p Alg ( O stay ∣ α ) < p Alg ( O leave ∣ α ) p_{\mathrm{Alg}}(O_{\text{stay}}\mid\alpha)<p_{\mathrm{Alg}}(O_{\text{leave}}\mid\alpha) ).
##### Exploring four rooms.
Suppose algorithm Alg ′ \mathrm{Alg}^{\prime} can explore e . g ., the first three rooms to the right of the initial room (shown in Figure 2 ), and consider any reward coefficient vector α ∈ Θ \alpha\in\Theta which assigns unique positive weight to each item. In particular, unique positive weights rule out constant reward vectors, in which case inductive bias would produce agents which do not leave the first room.
If the agent stays in the initial room, it can induce inventory states { empty , 1key }. If the agent explores the three extra rooms, it can also induce { 1sword , 1sword&1key } (see Figure 3 in Appendix C.2 ). Since α \alpha is positive, it is never optimal to finish the episode empty-handed. Therefore, if the Alg ′ \mathrm{Alg}^{\prime} policy stays in the first room, then α \alpha ’s feature coefficients must satisfy α key > α sword \alpha_{\text{key}}>\alpha_{\text{sword}} . Otherwise, α key < α sword \alpha_{\text{key}}<\alpha_{\text{sword}} (by assumption of unique item reward coefficients); in this case, the agent would leave and acquire the sword (since we assumed it knows how to do so). Then by switching the reward for the key and the sword, we retarget Alg ′ \mathrm{Alg}^{\prime} to go get the sword. Alg ′ \mathrm{Alg}^{\prime} is simply-retargetable away from the first room, because it can explore enough of the environment.
##### Exploring the entire level.
Algorithms like go-explore [ Ecoffet et al., 2021 ] are probably good at exploring even given sparse featurized reward. Therefore, go-explore is even more retargetable in this setting, because it is more able to explore and discover the breadth of options (final inventory counts) available to it, and remember how to navigate to them. Furthermore, sufficiently powerful planning algorithms should likewise be retargetable in a similar way, insofar as they can reliably find high-scoring item configurations.
We speculate that increasingly “impressive” algorithms (whether rl training or planning) are often more impressive because they can allow retargeting the agent’s final behavior from one kind of outcome, to another. Just as go-explore seems highly retargetable while dqn does not, we expect increasingly impressive algorithms to be increasingly retargetable—whether over actions in a bandit problem, or over the final observation in an rl episode.
## 5 Retargetability can imply power-seeking tendencies
### 5.1 Generalizing the power-seeking theorems for Markov decision processes
Turner et al. [2021] considered finite mdp s in which decision-makers took as input a reward function over states ( 𝐫 ∈ ℝ | 𝒮 | \mathbf{r}\in\mathbb{R}^{\left|\mathcal{S}\right|} ) and selected an optimal policy for that reward function. They considered the state visit distributions 𝐟 ∈ ℱ ( s ) \mathbf{f}\in\Fop(s) , which basically correspond to the trajectories which the agent could induce starting from state s s . For F ⊆ ℱ ( s ) F\subseteq\Fop(s) , p max ( F ∣ 𝐫 ) p_{\max}(F\mid\mathbf{r}) returns 1 1 if an element of F F is optimal for reward function 𝐫 \mathbf{r} , and 0 0 otherwise. They showed situations where a larger set of distributions F large F_{\text{large}} tended to be optimal over a smaller set: p max ( F large ∣ 𝐫 ) ≥ most : ℝ | 𝒮 | 1 p max ( F small ∣ 𝐫 ) p_{\max}(F_{\text{large}}\mid\mathbf{r})\geq_{\text{{most}}\text{: }\mathbb{R}^{\left|\mathcal{S}\right|}}^{1}p_{\max}(F_{\text{small}}\mid\mathbf{r}) . For example, in Pac-Man, most reward functions make it optimal to stay alive for at least one time step: p max ( F survival ∣ 𝐫 ) ≥ most : ℝ | 𝒮 | 1 p max ( F instant death ∣ 𝐫 ) p_{\max}(F_{\text{survival}}\mid\mathbf{r})\geq_{\text{{most}}\text{: }\mathbb{R}^{\left|\mathcal{S}\right|}}^{1}p_{\max}(F_{\text{instant death}}\mid\mathbf{r}) . Turner et al. [2021] showed that optimal policies tend to seek power by keeping options open and staying alive. Appendix D provides a quantitative generalization of Turner et al. [2021] ’s results on optimal policies.
Throughout this paper, we abstracted their arguments away from finite mdp s and reward-optimal decision-making. Instead, parametrically retargetable decision-makers tend to seek power: Proposition A.11 shows that a wide range of decision-making procedures are retargetable over outcomes, and Theorem A.13 demonstrates the retargetability of any decision-making which is determined by the expected utility of outcomes. In particular, these results apply straightforwardly to mdp s.
### 5.2 Better rl algorithms tend to be more retargetable
Reinforcement learning algorithms are practically useful insofar as they can train an agent to accomplish some task ( e . g ., cleaning a room). A good rl algorithm is relatively task-agnostic ( e . g ., is not restricted to only training policies which clean rooms). Task-agnosticism suggests retargetability across desired future outcomes / task completions.
In mr , suppose we instead give the agent 1 1 reward for the initial state, and 0 0 otherwise. Any reasonable reinforcement learning procedure will just learn to stay put (which is the optimal policy). However, consider whether we can retarget the agent’s policy to beat the game, by swapping the initial state reward with the end-game state reward. Most present-day rl algorithms are not good enough to solve such a sparse game, and so are not retargetable in this sense. But an agent which did enough exploration would also learn a good policy for the permuted reward function. Such an effective training regime could be useful for solving real-world tasks. Many researchers aim to develop effective training regimes.
Our results suggest that once rl capabilities reach a certain level, trained agents will tend to seek power in the real world. Presently, it is not dangerous to train an agent to complete a task—such an agent will not be able to complete its task by staying activated against the designers’ wishes. The present lack of danger is not because optimal policies do not have self-preservation tendencies—they do [ Turner et al., 2021 ] . Rather, the lack of danger reflects the fact that present-day rl agents cannot learn such complex action sequences at all . Just as the Montezuma’s Revenge agent had to be sufficiently competent to be retargetable from initial-state reward to game-complete reward, real-world agents have to be sufficiently intelligent in order to be retargetable from outcomes which don’t require power-seeking, to those which do require power-seeking.
Here is some speculation. After training an rl agent to a high level of capability, the agent may be optimizing internally represented goals over its model of the environment [ Hubinger et al., 2019 ] . Furthermore, we think that different reward parameter settings would train different internal goals into the agent. To make an analogy, changing a person’s reward circuitry would presumably reinforce them for different kinds of activities and thereby change their priorities. In this sense, trained real-world agents may be retargetable towards power-requiring outcomes via the reward function parameter setting. Insofar as this speculation holds, our theory predicts that advanced reinforcement learning at scale will—for most settings of the reward function—train policies which tend to seek power.
## 6 Discussion
In section 3 , we formalized a notion of parametric retargetability and stated several key results. While our results are broadly applicable, further work is required to understand the implications for ai .
### 6.1 Prior work
In this work, we do not motivate the risks from ai power-seeking. We refer the reader to e . g ., Carlsmith [2021] . As explained in section 5.1 , Turner et al. [2021] show that, given certain environmental symmetries in an mdp , the optimal-policy-producing algorithm f f (state visitation distribution set, state-based reward function) is 1-retargetable via the reward function, from smaller to larger sets of environmental options. Appendix A shows that optimality is not required, and instead a wide range of decision-making procedures satisfy the retargetability criterion. Furthermore, we generalize from 1-retargetability to n n -fold-retargetability whenever option set B B contains “ n n copies” of set A A ( definition A.7 in appendix A ).
### 6.2 Future work and limitations
We currently have analyzed planning- and reinforcement learning-based settings. However, results such as Theorem 3.6 might in some way apply to the training of other machine learning networks. Furthermore, while Theorem 3.6 does not assume a finite environment, we currently do not see how to apply that result to e . g ., infinite-state partially observable Markov decision processes.
Section 4 semi-formally analyzes decision-making incentives in the mr video game, leaving the proofs to appendix C . However, these proofs are several pages long. Perhaps additional lemmas can allow quick proof of orbit-level incentives in situations relevant to real-world decision-makers.