Optimal Policies Tend to Seek Power

arxiv:1912.01683

Original source · Paper JSON

AI safety and consciousness · 2019-12-03

Cites (0) · Cited by (4)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12
13. Part 13
14. Part 14
15. Part 15
16. Part 16

Contents

• Optimal Policies Tend To Seek Power
• Abstract
• 1 Introduction
• 2 Related work
• 3 State visit distribution functions quantify the agent’s available options
• Definition 3.1 (Rewardless mdp ) .
• Definition 3.2 (1-cycle states) .
• Definition 3.3 (State visit distribution [ Sutton and Barto, 1998 ] ) .
• Definition 3.4 ( ℱ \Fop single-state restriction) .
• Definition 3.5 (Value function) .
• Definition 3.6 (Non-domination) .
• 4 Some actions have a greater probability of being optimal
• Definition 4.1 (Optimal policy set function) .
• Definition 4.2 (Reward function distributions) .
• Definition 4.3 (Visit distribution optimality probability) .
• Definition 4.4 (Action optimality probability) .
• 5 Some states give the agent more control over the future
• Definition 5.1 (Average optimal value) .
• Definition 5.2 ( Power ) .
• Lemma 5.3 (Continuity of Power ) .
• Proposition 5.4 (Maximal Power ) .
• Proposition 5.5 ( Power is smooth across reversible dynamics) .
• Definition 5.6 ( Power -seeking actions) .
• 6 Certain environmental symmetries produce power-seeking tendencies
• Definition 6.1 (Similarity of vector sets) .
• Definition 6.2 (Similarity of vector function sets) .
• Definition 6.3 (Pushforward distribution of a permutation) .
• Definition 6.4 (Orbit of a probability distribution) .
• Definition 6.5 (Inequalities which hold for most probability distributions) .
• Proposition 6.6 (States with “more options” have more Power ) .
• 6.1 Keeping options open tends to be Power -seeking and tends to be optimal
• Definition 6.7 (Equivalent actions) .
• Definition 6.8 (States reachable after taking an action) .
• Proposition 6.9 (Keeping options open tends to be Power -seeking and tends to be optimal) .
• 6.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles
• Definition 6.10 (Recurrent state distributions [ Puterman, 2014 ] ) .
• Definition 6.11 (Average-optimal policies) .
• Proposition 6.12 (When γ = 1 \gamma=1 , rsd s control Power ) .
• Theorem 6.13 (Average-optimal policies tend to end up in “larger” sets of rsd s) .
• Corollary 6.14 (Average-optimal policies tend not to end up in any given 1-cycle) .
• 6.3 How to reason about other environments
• 7 Discussion
• Future work.
• Societal impact.
• Conclusion.
• Acknowledgments
• Appendix A Comparing Power with information-theoretic empowerment
• Appendix B Seeking Power can be a detour
• Remark .
• Proposition B.1 (Greater Power 𝒟 bound \text{{Power}}_{\mathcal{D}_{\text{bound}}} does not imply greater ℙ 𝒟 bound \Prb_{\mathcal{D}_{\text{bound}}} ) .
• Proof.
• Lemma B.2 (Fraction of orbits which agree on weak optimality) .
• Proof.
• Lemma B.3 ( ≥ most \geq_{\text{most}} and trivial orbits) .
• Proof.
• Proposition B.4 (Actions which tend to seek Power do not necessarily tend to be optimal) .
• Proof.
• Appendix C Sub-optimal Power
• Definition C.1 (Suboptimal Power ) .
• Appendix D Lists of results
• D.1 Contributions of independent interest
• D.1.1 Optimal value theory
• D.1.2 Optimal policy theory
• D.1.3 Visit distribution theory
• Appendix E Theoretical results
• Lemma E.1 (A policy is optimal iff it induces an optimal visit distribution at every state) .
• Proof.
• Definition E.2 (Transition matrix induced by a policy) .
• Proposition E.3 (Properties of visit distribution functions) .
• Proof.
• Lemma E.4 ( 𝐟 ∈ ℱ ⁡ ( s ) \mathbf{f}\in\Fop(s) is multivariate rational on γ \gamma ) .
• Proof.
• Corollary E.5 (On-policy value is rational on γ \gamma ) .
• Proof.
• E.1 Non-dominated visit distribution functions
• Definition E.6 (Continuous reward function distribution) .
• Remark .
• Lemma E.7 (Distinct linear functionals disagree almost everywhere on their domains) .
• Proof.
• Corollary E.8 (Unique maximization of almost all vectors) .
• Proof.
• E.1.1 Generalized non-domination results
• Definition E.9 (Non-dominated linear functionals) .
• Lemma E.10 (All vectors are maximized by a non-dominated linear functional) .
• Proof.
• Corollary E.11 (Maximal value is invariant to restriction to non-dominated functionals) .
• Proof.
• Lemma E.12 (How non-domination containment affects optimal value) .
• Proof.
• Definition E.13 (Non-dominated vector functions) .
• Remark .
• Definition E.14 (Affine transformation of visit distribution sets) .
• Lemma E.15 (Invariance of non-domination under positive affine transform) .
• Proof.
• E.1.2 Inequalities which hold under most reward function distributions
• Lemma E.16 (Helper lemma for demonstrating ≥ most : 𝔇 any \geq_{\text{{most}}\text{: }\mathfrak{D}_{\text{any}}} ) .
• Proof.
• Lemma E.17 (A helper result for expectations of functions) .
• Proof.
• Definition E.18 (Support of 𝒟 any \mathcal{D}_{\text{any}} ) .
• Definition E.19 (Linear functional optimality probability) .
• Proposition E.20 (Non-dominated linear functionals and their optimality probability) .
• Proof.
• Lemma E.21 (Expected value of similar linear functional sets) .
• Proof.
• Lemma E.22 (For continuous iid distributions 𝒟 X ​ - iid \mathcal{D}_{X\text{-}\textsc{iid}} , ∃ b < c : ( b , c ) | 𝒮 | ⊆ supp ⁡ ( 𝒟 X ​ - iid ) \exists b<c\mathrel{\mathop{\ordinarycolon}}(b,c)^{\left|\mathcal{S}\right|}\subseteq\optSupp(\mathcal{D}_{X\text{-}\textsc{iid}}) ) .
• Proof.
• Definition E.23 (Bounded, continuous iid reward) .
• Lemma E.24 (Expectation superiority lemma) .
• Proof.
• Definition E.25 (Indicator function) .
• Lemma E.26 (Optimality probability inclusion relations) .
• Proof.
• Lemma E.27 (Optimality probability of similar linear functional sets) .
• Proof.
• Lemma E.28 (Optimality probability superiority lemma) .
• Proof.
• Lemma E.29 (Limit probability inequalities which hold for most distributions) .
• Proof.
• E.1.3 ℱ nd \Fndop results
• Proof.
• Definition E.30 (Evaluating sets of visit distribution functions at γ \gamma ) .
• Lemma E.31 (Non-domination across γ \gamma values for expectations of visit distributions) .
• Proof.
• Lemma E.32 ( ∀ γ ∈ ( 0 , 1 ) : 𝐝 ∈ ℱ nd ⁡ ( s , γ ) \forall\gamma\in(0,1)\mathrel{\mathop{\ordinarycolon}}\mathbf{d}\in\Fndop(s,\gamma) iff 𝐝 ∈ ND ​ ( ℱ ⁡ ( s , γ ) ) \mathbf{d}\in\text{{ND}}\left(\Fop(s,\gamma)\right) ) .
• Proof.
• Proof.
• E.2 Some actions have greater probability of being optimal
• Lemma E.33 (Optimal policy shift bound) .
• Proof.
• Proposition E.34 (Optimality probability’s limits exist) .
• Proof.
• Lemma E.35 (Optimality probability identity) .
• Proof.
• E.3 Basic properties of Power
• Lemma E.36 ( Power identities) .
• Proof.
• Definition E.37 (Discount-normalized value function) .
• Lemma E.38 (Normalized value functions have uniformly bounded derivative) .
• Proof.
• Proof.
• Proof.
• Lemma E.39 (Lower bound on current Power based on future Power ) .
• Proof.
• Proof.
• E.4 Seeking Power is often more probable under optimality
• E.4.1 Keeping options open tends to be Power -seeking and tends to be optimal
• Definition E.40 (Normalized visit distribution function) .
• Remark .
• Lemma E.41 (Normalized visit distribution functions are continuous) .
• Proof.
• Lemma E.42 (Non-domination of normalized visit distribution functions) .
• Proof.
• Lemma E.43 ( Power limit identity) .
• Proof.
• Lemma E.44 (Lemma for Power superiority) .
• Proof.
• Proof.
• Lemma E.45 (Non-dominated visit distribution functions never agree with other visit distribution functions at that state) .
• Proof.
• Corollary E.46 (Cardinality of non-dominated visit distributions) .
• Proof.
• Lemma E.47 (Optimality probability and state bottlenecks) .
• Proof.
• Case: γ ∈ ( 0 , 1 ) \gamma\in(0,1) .
• Case: γ = 1 \gamma=1 , γ = 0 \gamma=0 .
• Lemma E.48 (Action optimality probability is a special case of visit distribution optimality probability) .
• Proof.
• Proof.
• E.4.2 When γ = 1 \gamma=1 , optimal policies tend to navigate towards “larger” sets of cycles
• Lemma E.49 ( Power identity when γ = 1 \gamma=1 ) .
• Proof.
• Proof.
• Proof.
• Proposition E.50 ( Rsd properties) .
• Proof.
• Lemma E.51 (When reachable with probability 1, 1-cycles induce non-dominated rsd s) .
• Proof.
• Proof.
• Instructions for reporting errors