Parametrically Retargetable Decision-Makers Tend To Seek Power

arxiv:2206.13477

Original source · Paper JSON

AI safety and consciousness · 2022-06-27

Cites (3) · Cited by (1)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3
4. Part 4
5. Part 5
6. Part 6
7. Part 7
8. Part 8
9. Part 9
10. Part 10
11. Part 11
12. Part 12
13. Part 13

Contents

• Parametrically Retargetable Decision-Makers Tend To Seek Power
• Abstract
• 1 Introduction
• 2 Statistical tendencies for a range of decision-making algorithms
• 3 Formal notions of retargetability and decision-making tendencies
• Definition 3.1 (Orbit of a parameter) .
• Definition 3.2 (Inequalities which hold for most orbit elements) .
• Definition 3.3 (Simply-retargetable function) .
• Proposition 3.4 (Simply-retargetable functions have orbit-level tendencies) .
• Definition 3.5 (Multiply retargetable function) .
• Theorem 3.6 (Multiply retargetable functions have orbit-level tendencies) .
• Proof outline (full proof in Appendix B ).
• 4 Decision-making tendencies in Montezuma’s Revenge
• Terminology.
• 4.1 Tendencies for initial action selection
• 4.2 Tendencies for maximizing reward over the final observation
• 4.3 Tendencies for rl on featurized reward over the final observation
• Exploring the first room.
• Exploring four rooms.
• Exploring the entire level.
• 5 Retargetability can imply power-seeking tendencies
• 5.1 Generalizing the power-seeking theorems for Markov decision processes
• 5.2 Better rl algorithms tend to be more retargetable
• 6 Discussion
• 6.1 Prior work
• 6.2 Future work and limitations
• 6.3 Conclusion
• Broader impacts
• Acknowledgements
• Appendix A Retargetability over outcome lotteries
• Definition A.1 (Outcome lotteries) .
• Definition A.2 (Optimality indicator function) .
• Definition A.3 (Anti-optimality indicator function) .
• Definition A.4 (Boltzmann rationality [ Baker et al., 2007 ] ) .
• Definition A.5 (Satisficing) .
• A.1 A range of decision-making functions are retargetable
• Definition A.6 (Similarity of vector sets) .
• Definition A.7 (Containment of set copies) .
• Definition A.8 (Targeting parameter distribution assumptions) .
• Definition A.9 (Pushforward distribution of a permutation [ Turner et al., 2021 ] ) .
• Definition A.10 (Orbit of a probability distribution [ Turner et al., 2021 ] ) .
• Proposition A.11 (Orbit incentives for different rationalities) .
• Definition A.12 (EU-determined functions) .
• Theorem A.13 (Orbit tendencies occur for EU-determined decision-making functions) .
• Appendix B Theoretical results
• Remark .
• Lemma B.1 (Limited transitivity of ≥ most \geq_{\text{most}} ) .
• Proof.
• Lemma B.2 (Order inversion for ≥ most \geq_{\text{most}} ) .
• Proof.
• Remark .
• Lemma B.3 (Orbital fraction which agrees on (weak) inequality) .
• Proof.
• B.1 General results on retargetable functions
• Definition B.4 (Functions which are increasing under joint permutation) .
• Lemma B.5 (Expectations of joint-permutation-increasing functions are also joint-permutation-increasing) .
• Proof.
• Lemma B.6 (Closure of orbit incentives under increasing functions) .
• Proof.
• Remark .
• Proof.
• Proof.
• B.2 Helper results on retargetable functions
• Lemma B.7 (Quantitative general orbit lemma) .
• Proof.
• Disjointness.
• Definition B.8 (Superset-of-copy containment) .
• Lemma B.9 (Looser sufficient conditions for orbit-level incentives) .
• Proof.
• Lemma B.10 (Hiding an argument which is invariant under certain permutations) .
• Proof.
• B.2.1 EU-determined functions
• Lemma B.11 (EU-determined functions are invariant under joint permutation) .
• Proof.
• Proof.
• B.3 Particular results on retargetable functions
• Definition B.12 (Quantilization, closed form) .
• Remark .
• Proof.
• Conjecture B.13 (Orbit tendencies occur for more quantilizer base distributions) .
• Appendix C Detailed analyses of mr scenarios
• C.1 Action selection
• Lemma C.1 (Lower bound on success probability of the train bandit) .
• Proof.
• Proposition C.2 (The train bandit is 4-retargetable) .
• Proof.
• Corollary C.3 (The train bandit has orbit-level tendencies) .
• Proof.
• C.2 Observation reward maximization
• Proposition C.4 (Final reward maximization has strong orbit-level incentives in mr ) .
• Proof.
• Lemma C.5 (Room-status inequalities for mr ) .
• Proof.
• Corollary C.6 (Final reward maximizers tend to leave the first room in mr ) .
• Proof.
• C.3 Featurized reward maximization
• Lemma C.7 ( FracOptimal \mathrm{FracOptimal} inequalities) .
• Proof.
• Subset case.
• Disjoint case.
• Conjecture C.8 (Generalizing Lemma C.7 ) .
• Proposition C.9 (Featurized reward maximizers tend to leave the first room in mr ) .
• Proof.
• C.4 Reasoning for why dqn can’t explore well
• Appendix D Lower bounds on mdp power-seeking incentives for optimal policies
• Definition D.1 (Non-dominated linear functionals) .
• Definition D.2 (Bounded reward function distribution) .
• Remark .
• Lemma D.3 (Quantitative expectation superiority lemma) .
• Proof.
• Definition D.4 (Linear functional optimality probability [ Turner et al., 2021 ] ) .
• Lemma D.5 (Quantitative optimality probability superiority lemma) .
• Proof.
• Definition D.6 (Rewardless mdp [ Turner et al., 2021 ] ) .
• Definition D.7 (1-cycle states [ Turner et al., 2021 ] ) .
• Definition D.8 (State visit distribution [ Sutton and Barto, 1998 ] ) .
• Definition D.9 (Recurrent state distributions [ Puterman, 2014 ] ) .
• Definition D.10 (Average-optimal policies [ Turner et al., 2021 ] ) .
• Remark .
• Theorem D.11 (Quantitatively, average-optimal policies tend to end up in “larger” sets of rsd s) .
• Proof.
• Corollary D.12 (Quantitatively, average-optimal policies tend not to end up in any given 1-cycle) .
• Proof.
• Conjecture D.13 (Fractional quantitative optimality probability superiority lemma) .
• Instructions for reporting errors