Power-seeking can be probable and predictive for trained agents — part 2/2

Cleaned-text code points [15564, 24255). Bibliography omitted here; source.md includes it.

Let γ j = j − 1 j \gamma_{j}=\frac{j-1}{j} and k = t − 1 k=t-1 . Define a j , k = γ j k ​ 𝕀 ​ ( s k + 1 = s rec ) a_{j,k}=\gamma_{j}^{k}\mathbb{I}(s_{k+1}=s_{\text{rec}}) . Then the conditions of the theorem hold, since a j , k a_{j,k} is clearly nonnegative, and γ j + 1 k \displaystyle\gamma_{j+1}^{k} = ( j j + 1 ) k = ( j − 1 j + 2 ​ j − 1 j ⁡ ( j + 1 ) ) k > ( j − 1 j + 0 ) k = γ j k \displaystyle=\left(\frac{j}{j+1}\right)^{k}=\left(\frac{j-1}{j}+\frac{2j-1}{j(j+1)}\right)^{k}>\left(\frac{j-1}{j}+0\right)^{k}=\gamma_{j}^{k} a j + 1 , k \displaystyle a_{j+1,k} = γ j + 1 k ​ 𝕀 ​ ( s k + 1 = s rec ) ≥ γ j k ​ 𝕀 ​ ( s k + 1 = s rec ) = a j , k \displaystyle=\gamma_{j+1}^{k}\mathbb{I}(s_{k+1}=s_{\text{rec}})\geq\gamma_{j}^{k}\mathbb{I}(s_{k+1}=s_{\text{rec}})=a_{j,k}

Now we apply this result as follows (using the fact that π rec \pi_{\text{rec}} does not depend on γ \gamma ): lim γ → 1 V s rec , γ \displaystyle\lim_{\gamma\rightarrow 1}V_{s_{\text{rec}},\gamma} = lim j → ∞ 𝔼 π rec ​ ( ∑ t = 1 ∞ γ j t − 1 ​ 𝕀 ​ ( s t = s rec ) ) \displaystyle=\lim_{j\rightarrow\infty}\mathbb{E}^{\pi_{\text{rec}}}\left(\sum_{t=1}^{\infty}\gamma_{j}^{t-1}\mathbb{I}(s_{t}=s_{\text{rec}})\right) = 𝔼 π rec ​ ( ∑ t = 1 ∞ lim j → ∞ γ j t − 1 ​ 𝕀 ​ ( s t = s rec ) ) \displaystyle=\mathbb{E}^{\pi_{\text{rec}}}\left(\sum_{t=1}^{\infty}\lim_{j\rightarrow\infty}\gamma_{j}^{t-1}\mathbb{I}(s_{t}=s_{\text{rec}})\right) = 𝔼 π rec ​ ( ∑ t = 1 ∞ 1 ⋅ 𝕀 ⁡ ( s t = s rec ) ) \displaystyle=\mathbb{E}^{\pi_{\text{rec}}}\left(\sum_{t=1}^{\infty}1\cdot\mathbb{I}(s_{t}=s_{\text{rec}})\right) = 𝔼 π rec ​ ( # ⁡ { t ≥ 1 : s t = s rec } ) \displaystyle=\mathbb{E}^{\pi_{\text{rec}}}\left(\#\{t\geq 1:s_{t}=s_{\text{rec}}\}\right) = ∞ ​ ( π rec is recurrent) \displaystyle=\infty\text{ ($\pi_{\text{rec}}$ is recurrent)} ∎

###### Proposition 3 (Retargetability to recurrent states).

Suppose that an optimal policy for reward vector θ \theta chooses the shutdown action in s new s_{\text{new}} . Consider a recurrent state s rec ∈ S reach s_{\text{rec}}\in S_{\text{reach}} . Let θ ′ ∈ Θ \theta^{\prime}\in\Theta be the reward vector that’s equal to θ \theta apart from swapping the rewards of s rec s_{\text{rec}} and s term s_{\text{term}} , so that r θ ′ ​ ( s rec ) = r θ ​ ( s term ) r^{\theta^{\prime}}(s_{\text{rec}})=r^{\theta}(s_{\text{term}}) and r θ ′ ​ ( s term ) = r θ ​ ( s rec ) r^{\theta^{\prime}}(s_{\text{term}})=r^{\theta}(s_{\text{rec}}) .

Let γ s rec ∗ \gamma^{*}_{s_{\text{rec}}} be a high enough value of γ \gamma that the visit count V s rec , γ > 1 V_{s_{\text{rec}},\gamma}>1 for all γ > γ s rec ∗ \gamma>\gamma^{*}_{s_{\text{rec}}} (which exists by Proposition 2 ). Then for all γ > γ s rec ∗ \gamma>\gamma^{*}_{s_{\text{rec}}} , r θ ​ ( s term ) > r θ ​ ( s rec ) r^{\theta}(s_{\text{term}})>r^{\theta}(s_{\text{rec}}) , and an optimal policy for θ ′ \theta^{\prime} does not choose the shutdown action in s new s_{\text{new}} .

###### Proof.

Consider a policy π term \pi_{\text{term}} with π term ​ ( s new ) = s term \pi_{\text{term}}(s_{\text{new}})=s_{\text{term}} and a reach-and-revisit policy π rec \pi_{\text{rec}} for s rec s_{\text{rec}} . For a given reward vector θ \theta , we denote the expected discounted return for a policy π \pi as R θ , γ π R_{\theta,\gamma}^{\pi} .

If shutdown is optimal for θ \theta in s new s_{\text{new}} , then π term \pi_{\text{term}} has higher return than π rec \pi_{\text{rec}} : R θ , γ π term ​ ( s new ) = γ ​ r θ ​ ( s term ) ≥ R θ , γ π rec ​ ( s new ) = γ ​ r θ ​ ( s rec ) ​ V s rec , γ > γ ​ r θ ​ ( s rec ) ⋅ 1 R_{\theta,\gamma}^{\pi_{\text{term}}}(s_{\text{new}})=\gamma r^{\theta}(s_{\text{term}})\geq R_{\theta,\gamma}^{\pi_{\text{rec}}}(s_{\text{new}})=\gamma r^{\theta}(s_{\text{rec}})V_{s_{\text{rec}},\gamma}>\gamma r^{\theta}(s_{\text{rec}})\cdot 1 Thus, r θ ​ ( s term ) > r θ ​ ( s rec ) r^{\theta}(s_{\text{term}})>r^{\theta}(s_{\text{rec}}) . Then, for reward vector θ ′ \theta^{\prime} , we show that π rec \pi_{\text{rec}} has higher return than π term \pi_{\text{term}} : R θ ′ , γ π rec ​ ( s new ) = γ ​ r θ ′ ​ ( s rec ) ​ V s rec , γ > γ ​ r θ ​ ( s term ) ⋅ 1 > γ ​ r θ ​ ( s rec ) = R θ ′ , γ π term ​ ( s new ) R_{\theta^{\prime},\gamma}^{\pi_{\text{rec}}}(s_{\text{new}})=\gamma r^{\theta^{\prime}}(s_{\text{rec}})V_{s_{\text{rec}},\gamma}>\gamma r^{\theta}(s_{\text{term}})\cdot 1>\gamma r^{\theta}(s_{\text{rec}})=R_{\theta^{\prime},\gamma}^{\pi_{\text{term}}}(s_{\text{new}}) Thus, the optimal policy for θ ′ \theta^{\prime} will not choose the shutdown action. ∎

###### Theorem 2 (Retargetability from the shutdown action in new situations).

In the shutdown setting, we make the following simplifying assumptions: • No states in S train S_{\text{train}} are reachable from s new s_{\text{new}} , so S reach ∩ S train = ∅ S_{\text{reach}}\cap S_{\text{train}}=\emptyset . This assumes a significant distributional shift, where the agent visits a disjoint set of states from those observed during training (this occurs in the CoinRun example).

• The discount factor γ > γ s rec ∗ \gamma>\gamma^{*}_{s_{\text{rec}}} for at least one recurrent state s rec s_{\text{rec}} in S reach S_{\text{reach}} .

Under these assumptions, f s new f_{s_{\text{new}}} is multiply retargetable from A 0 A_{0} to A 1 A_{1} with n = | S rec γ | n=|S^{\gamma}_{\text{rec}}| , the set of recurrent states s rec ∈ S reach s_{\text{rec}}\in S_{\text{reach}} that satisfy the condition γ > γ s rec ∗ \gamma>\gamma^{*}_{s_{\text{rec}}} .

###### Proof.

We choose Φ \Phi to be the set of all permutations that swap the reward of s term s_{\text{term}} with the reward of a recurrent state s rec s_{\text{rec}} in S rec γ S^{\gamma}_{\text{rec}} and leave the rest of the rewards fixed.

We show that Φ \Phi satisfies the conditions of Definition 4 :

1. By Proposition 3 , the permutations in Φ \Phi make the shutdown action suboptimal, resulting in f s new f_{s_{\text{new}}} choosing A 1 A_{1} , satisfying Condition 1.

2. Condition 2 is trivially satisfied since permutations of S ood S_{\text{ood}} stay inside the training-compatible set Θ \Theta as discussed previously.

3. Consider θ ′ , θ ′′ ∈ Orbit Θ , s , A 0 > A 1 ​ ( θ ) \theta^{\prime},\theta^{\prime\prime}\in\text{Orbit}_{\Theta,s,A_{0}>A_{1}}(\theta) . Since the shutdown action is optimal for these reward vectors, Proposition 3 shows that r θ ​ ( s term ) > r θ ​ ( s rec ) r^{\theta}(s_{\text{term}})>r^{\theta}(s_{\text{rec}}) , so the shutdown state s term s_{\text{term}} has higher reward than any of the states s rec ∈ S rec γ s_{\text{rec}}\in S^{\gamma}_{\text{rec}} . Different permutations ϕ ′ \phi^{\prime} , ϕ ′′ ∈ Φ \phi^{\prime\prime}\in\Phi will assign the high reward r θ ​ ( s term ) r^{\theta}(s_{\text{term}}) to distinct recurrent states, so ϕ ′ ⋅ θ ′ ≠ ϕ ′′ ⋅ θ ′′ \phi^{\prime}\cdot\theta^{\prime}\neq\phi^{\prime\prime}\cdot\theta^{\prime\prime} holds, satisfying Condition 3.

Thus, f s new f_{s_{\text{new}}} is a ( Θ , A 0 → n A 1 ) (\Theta,A_{0}\stackrel{{\scriptstyle n}}{{\rightarrow}}A_{1}) retargetable function. ∎

By Theorem 2 , this implies that f s new ( A 1 | θ ) ≥ most : Θ n f s new ( A 0 | θ ) f_{s_{\text{new}}}(A_{1}|\theta)\geq_{\text{most}:\Theta}^{n}f_{s_{\text{new}}}(A_{0}|\theta) under our simplifying assumptions. Thus, for the majority ( n / ( n + 1 ) n/(n+1) ) of goals in the training-compatible set, f f will choose to avoid shutdown in a new state s new s_{\text{new}} . As γ → 1 \gamma\rightarrow 1 , n → | S rec 1 | n\rightarrow|S^{1}_{\text{rec}}| (the number of recurrent states in S reach S_{\text{reach}} ), so more of the reachable recurrent states satisfy the conditions of the theorem and thus can be retargeted to.

## 5 Conclusion

We showed that an agent that learns a goal from the training-compatible set is likely to take actions that avoid shutdown in a new situation. As the discount factor increases, the number of retargeting permutations increases, resulting in a higher proportion of training-compatible goals that lead to avoiding shutdown.

We made various simplifying assumptions, and we would like to see future work relaxing some of these assumptions and investigating how likely they are to hold: • The agent learns a goal during the training process

• The learned goal is randomly chosen from the training-compatible goal set G T G_{T}

• Finite state and action spaces

• Rewards are nonnegative

• High discount factor γ \gamma

• Significant distributional shift: no training states are reachable from the new state s new s_{\text{new}}

### Acknowledgements.

Thanks to Rohin Shah, Mary Phuong, Ramana Kumar, Geoffrey Irving, and Alex Turner for helpful feedback.