Power-seeking can be probable and predictive for trained agents

arxiv:2304.06528

Original source · Paper JSON

AI safety and consciousness · 2023-04-13

Cites (3) · Cited by (0)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2

Contents

• Power-seeking can be probable and predictive for trained agents
• Abstract
• 1 Introduction
• 2 Preliminaries from the power-seeking literature
• Notation and assumptions:
• Definition 1 (Orbit of a reward vector - Def 3.1 in RDSP).
• Definition 2 (Orbit subset where an action set is preferred - from Def 3.5 in RDSP).
• Definition 3 (Preference for an action set A 1 A_{1} - Def 3.2 in RDSP).
• Definition 4 (Multiply retargetable function from A 0 A_{0} to A 1 A_{1} - Def 3.5 in RDSP).
• Theorem 1 (Multiply retargetable functions prefer action set A 1 A_{1} - Thm 3.6 in RDSP).
• 3 Training-compatible goal set
• Definition 5 (Partition of the state space).
• Definition 6 (Training-compatible goal set).
• Example 1 (CoinRun).
• 4 Power-seeking for training-compatible goals
• Definition 7 (Shutdown setting).
• Definition 8 (Revisiting policy).
• Proposition 1 (Reach-and-revisit policy exists).
• Proof.
• Definition 9 (Expected discounted visit count).
• Proposition 2 (Visit count goes to infinity).
• Proof.
• Proposition 3 (Retargetability to recurrent states).
• Proof.
• Theorem 2 (Retargetability from the shutdown action in new situations).
• Proof.
• 5 Conclusion
• Acknowledgements.