Is Power-Seeking AI an Existential Risk? — part 1/16

Cleaned-text code points [0, 15536). Bibliography omitted here; source.md includes it.

# Is Power-Seeking AI an Existential Risk?

###### Abstract

This report examines what I see as the core argument for concern about existential risk from misaligned artificial intelligence. I proceed in two stages. First, I lay out a backdrop picture that informs such concern. On this picture, intelligent agency is an extremely powerful force, and creating agents much more intelligent than us is playing with fire – especially given that if their objectives are problematic, such agents would plausibly have instrumental incentives to seek power over humans. Second, I formulate and evaluate a more specific six-premise argument that creating agents of this kind will lead to existential catastrophe by 2070. On this argument, by 2070: (1) it will become possible and financially feasible to build relevantly powerful and agentic AI systems; (2) there will be strong incentives to do so; (3) it will be much harder to build aligned (and relevantly powerful/agentic) AI systems than to build misaligned (and relevantly powerful/agentic) AI systems that are still superficially attractive to deploy; (4) some such misaligned systems will seek power over humans in high-impact ways; (5) this problem will scale to the full disempowerment of humanity; and (6) such disempowerment will constitute an existential catastrophe. I assign rough subjective credences to the premises in this argument, and I end up with an overall estimate of ~5% that an existential catastrophe of this kind will occur by 2070. (May 2022 update: since making this report public in April 2021, my estimate here has gone up, and is now at >10%.)

## 1 Introduction

Some worry that the development of advanced artificial intelligence will result in existential catastrophe---that is, the destruction of humanity’s longterm potential. 1 1 1 See e.g. Yudkowsky (2008) , Bostrom (2014) , Hawking (2014) , Tegmark (2017) , Christiano (2019) , Russell (2019) , Ord (2020) , and Ngo (2020) . This definition of “existential catastrophe” is from Ord (2020, p. 27) ; see section 7 for a bit more discussion. Here I examine the following version of this worry (it’s not the only version):

By 2070:

1. It will become possible and financially feasible to build AI systems with the following properties:

• Advanced capability : they outperform the best humans on some set of tasks which when performed at advanced levels grant significant power in today’s world (tasks like scientific research, business/military/political strategy, engineering, and persuasion/manipulation).

• Agentic planning : they make and execute plans, in pursuit of objectives, on the basis of models of the world.

• Strategic awareness: the models they use in making plans represent with reasonable accuracy the causal upshot of gaining and maintaining power over humans and the real-world environment.

(Call these “APS”—Advanced, Planning, Strategically aware—systems.)

2. There will be strong incentives to build and deploy APS systems | (1).

3. It will be much harder to build APS systems that would not seek to gain and maintain power in unintended ways (because of problems with their objectives) on any of the inputs they’d encounter if deployed, than to build APS systems that would do this, but which are at least superficially attractive to deploy anyway | (1)–(2).

4. Some deployed APS systems will be exposed to inputs where they seek power in unintended and high-impact ways (say, collectively causing >$1 trillion dollars of damage), because of problems with their objectives | (1)--(3). 2 2 2 Let’s assume 2021 dollars.

5. Some of this power-seeking will scale (in aggregate) to the point of permanently disempowering ~all of humanity | (1)–(4).

6. This disempowerment will constitute an existential catastrophe | (1)–(5).

These claims are extremely important if true. My aim is to investigate them. I focus on (2)–(5), but I also say a few words about (1) and (6).

My current view is that there is a disturbingly substantive chance that a scenario along these lines occurs, and that many people alive today—including myself—live to see humanity permanently disempowered by AI systems we’ve lost control over. In the final section , I take an initial stab at quantifying this risk, by assigning rough probabilities to 1-6. My current, highly-unstable, subjective estimate is that there is a ~5% percent chance of existential catastrophe by 2070 from scenarios in which (1)–(6) are true. (May 2022 author’s note: since making this report public in April 2021, my estimate here has gone up; it’s currently at >10%.) My main hope, though, is not to push for a specific number, but rather to lay out the arguments in a way that can facilitate productive debate.

### 1.1 Preliminaries

Some preliminaries and caveats (those eager for the main content can skip):

• I’m focused, here, on a very specific type of worry. There are lots of other ways to be worried about AI—and even, about existential catastrophes resulting from AI. And there are lots of ways to be excited about AI, too.

• My emphasis and approach differs from that of others in the literature in various ways. 3 3 3 Though the differences listed here don’t each apply to all of the literature; and in many respects, the issues I discuss are the “classic issues.” Of existing overall analyses, mine is probably most similar to Ngo (2020) . For other discussions of alignment problems in AI (on top of those cited in footnote 1), see e.g. Amodei et al (2016) , Piper (2018, updated 2020) , and Christian (2020) . In particular: I’m less focused than some on the possibility of an extremely rapid escalation in frontier AI capabilities, on ‘‘recursive self-improvement,’’ or on scenarios in which one actor comes to dominate the world; I’m focusing on power-seeking in particular (as opposed to ‘‘misalignment’’ more broadly); I’m aiming to incorporate and respond to various objections and re-framings introduced since early work on the topic, and since machine learning has become a prominent AI paradigm; 4 4 4 For example, those of Shah (2018) , Pinker (2018) , Drexler (2019) , Zador and LeCun (2019) , Hubinger et al (2019) , Garfinkel (2020) . I’m aiming to avoid reliance on models of ‘‘utility function maximization’’ and related concepts; and I present and assign probabilities to premises in a complete argument for catastrophe. That said, I still think of this as an articulation and analysis of a certain kind of ‘‘core argument’’---one that has animated, and continues to animate, much concern about existential risk from misaligned AI. 5 5 5 Some—e.g. Adamczewski (2019) , Garfinkel (2019) , and Ngo (2019) , though Ngo notes that his views have since shifted—suggest that arguments for X-risk from misaligned AI have “shifted” or “drifted” over time. I see some of this in some places, but for me at least (and for various others I know), the basic thrust—e.g., advanced artificial agents would be very powerful, and if their objectives are problematic, they might have default incentives to disempower humans—has been pretty constant (sometimes claims about discontinuous/concentrated “takeoff” are treated as essential to the basic thrust of the argument, but I don’t see them that way—see section 6 for discussion).

• I’m focusing on 2070 because I want to keep vividly in mind that I and many other readers (and/or their children) should expect to live to see the claims at stake here falsified or confirmed. That said, the main arguments don’t actually require the development of relevant systems within any particular period of time (though timelines in this respect can matter to e.g. the amount of evidence that present-day systems and conditions provide about future risks).

• I’m not addressing the question of what sorts of interventions are currently available for lowering the risk in question, or how they compare with work on other issues.

• Section 8 involves assigning (a) subjective probabilities to (b) imprecisely operationalized claims, in the context of (c) a multi-step argument. I discuss cautions on all of three of these fronts there (and I include an appendix aimed at warding off some possible problems with (c) in particular).

• Thinking about what will happen decades in the future is famously hard, and some approach ~any such thought with extreme skepticism. I’m sympathetic to this in ways, but I also think we can do better than brute agnosticism; and to the extent that we care about the future, and can try to influence it, our actions will often express “bets” about it even absent explicit forecasts.

• I use various concepts it would be great to be more precise about. This is true of many worthwhile discussions, but that doesn’t make the attending imprecisions innocuous. To the contrary, I encourage wariness of ways they might mislead.

• The discussion focuses on how to avoid human disempowerment, but I am not assuming that it is always appropriate or good for humans to keep power. Rather, I’m assuming that we want to avoid disempowerment that is imposed on us by artificial systems we’ve lost control over. See section 7 for more discussion.

• Sufficiently sophisticated AI systems might warrant moral concern. 6 6 6 See Muehlhauser (2017) for more on what sorts of non-humans warrant this concern. In my opinion, this fact should motivate grave ethical caution in the context of many types of AI development, including many discussed in this report. However, it’s not my focus here (see section 7 for a few remarks).

### 1.2 Backdrop

The specific arguments I’ll discuss emerge from a broader backdrop picture, which I’ll gloss as:

1. Intelligent agency is an extremely powerful force for controlling and transforming the world.

2. Building agents much more intelligent than humans is playing with fire.

I’ll start by briefly describing this picture, as I think it sets an important stage for the discussion that follows.

#### 1.2.1 Intelligence

Of all the species that have lived on earth, humans are clearly strange. In particular, we exert an unprecedented scale and sophistication of intentional control over our environment. Consider, for example: the city of Tokyo , the Large Hadron Collider , and the Bingham Canyon Mine .

What makes this possible? Something about our minds seems centrally important. We can plan, learn, communicate, deduce, remember, explain, imagine, experiment, and cooperate in ways that other species can’t. These cognitive abilities---employed in the context of the culture and technology we inherit and create---give us the power, collectively, to transform the world. For ease of reference, let’s call this loose cluster of abilities ‘‘intelligence,’’ though very little will rest on the term. 7 7 7 In particular, when I say “more intelligent than humans,” I just mean “better at things like planning, learning, communicating, deducing, remembering, etc” than humans. I don’t have in mind some broader notion like “optimization power,” “solving problems,” or “achieving objectives.” Thanks to Ben Garfinkel for discussion here. Obviously, intelligence in this sense is highly multi-dimensional. I don’t think this is a problem for saying one thing is more intelligent than another (e.g., which house is better to live in is multi-dimensional, but a mansion can still be preferable to a shack); but if you object to the term for this or some other reason, feel free to substitute “cognitive abilities like planning, learning, communicating, deducing, remembering, etc” whenever I say “intelligence” (I won’t use the term very often). See Garfinkel et al (2017) for more on objections to the possibility of greater-than-human intelligence.

Our abilities in these respects are nowhere near any sort of hard limit. Human cognition---even in groups, and with the assistance of technology---depends centrally on the human brain, which, for all its wonders, is an extremely specific and limited organ, subject to very specific constraints---on cell count, energy, communication speed, signaling frequency, memory capacity, component reliability, input/output bandwidth, and so forth. 8 8 8 For example: the brain has ~1e11 neurons and 1e14-1e15 synapses ; neurons fire at a maximum of some hundreds of Hz ; action potentials travel at a max of some hundreds of m/s ; the brain runs on ~20W of power ; it has to fit within the skull; and so forth.

There are possible cognitive systems—possible brains, and possible artificial systems—to which some or all these constraints do not apply. It seems very likely that such systems could learn, communicate, reason, problem-solve, etc, much better than humans can. The variation in cognitive ability we see among humans, and across species, also suggests this; as do existing successes in approaching or exceeding human capabilities at particular tasks—mathematical calculation, game-playing, image recognition, and so forth—using artificial systems.

#### 1.2.2 Agency

Humans can also be agentic —that is (loosely), we pursue objectives, guided by models of the world (see section 2.1.2 for more). You want to travel to New York, so you buy a flight, double-check when it leaves, wake up early to pack, and so forth.

Non-human cognitive systems can be agentic, too. And depending on what you count as an agent, humans can already create novel non-human agents. 9 9 9 I’ll count agent-like systems constituted in central part by human agents—e.g., nations, corporations, parent teacher organizations, etc—as “human” (which need not imply: innocuous). We breed and genetically modify non-humans animals, for example; and some of our artificial systems display agent-like behavior in specific environments (see e.g. here and here ). But these agents can’t learn, plan, communicate, etc. like we can.

At some point, though---absent catastrophe, deliberate choice, or other disruption of scientific progress---we will likely be in a position, if we so choose, to create non-human agents whose abilities in these respects rival or exceed our own. And in the context of artificial agents, the differences between brains and computers---in possible speed, size, available energy, memory capacity, component reliability, input/output bandwidth, and so forth---make the eventual possibility of very dramatic differences in ability especially salient. 10 10 10 Thus, as Bostrom (2014, Chapter 3) discusses, where neurons can fire a maximum of hundreds of Hz, the clock speed of modern computers can reach some 2 Ghz: ~ten million times faster. Where action potentials travel at some hundreds of m/s, optical communication can take place at 300,000,000 m/s: ~a million times faster. Where brain size and neuron count are limited by cranial volume, metabolic constraints, and other factors, supercomputers can be the size of warehouses. And artificial systems need not suffer, either, from the brain’s constraints with respect to memory and component reliability, input/output bandwidth, tiring after hours, degrading after decades, storing its own repair mechanisms and blueprints inside itself, and so forth. Artificial systems can also be edited and duplicated much more easily than brains, and information can be more easily transferred between them.

#### 1.2.3 Playing with fire

The choice to create agents much more intelligent than we are should be approached with extreme caution. This is the basic backdrop view underlying much of the concern about existential risk from AI—and it would apply, in similar ways, to new biological agents (human or non-human).