Is Power-Seeking AI an Existential Risk? — part 2/16

Cleaned-text code points [15536, 30899). Bibliography omitted here; source.md includes it.

Some articulate this view by appeal to the dominant position of humans on this planet, relative to other species. 11 11 11 See e.g. Bostrom (2015) , Russell (2019, Chapter 5) on the “Gorilla Problem,” Ord (2020) , and Ngo (2020) on the “second species argument.” For example: some argue that the fate of the chimpanzees is currently in human hands, and that this difference in power is primarily attributable to differences in intelligence, rather than e.g. physical strength. Just as chimpanzees—given the choice and power—should be careful about building humans, then, we should be careful about building agents more intelligent than us.

This argument is suggestive, but far from airtight. Chimpanzees, for example, are themselves much more intelligent than mice, but the ‘‘fate of the mice’’ was never ‘‘in the hands’’ of the chimpanzees. What’s more, the control that humans can exert over the fate of other species on this planet still has limits, and we can debate whether ‘‘intelligence,’’ even in the context of accumulating culture and technology, is the best way of explaining what control we have. 12 12 12 Note, too, that a single human, unaided and uneducated, would plausibly fare worse than a chimpanzee in many environments (see Henrich (2015) for more on humans struggling in the absence of cultural learning).

More importantly, though: humans arose through an evolutionary process that chimpanzees did nothing to intentionally steer. Humans, though, will be able to control many aspects of processes we use to build and empower new intelligent agents. 13 13 13 Though whether that control will be adequate to ensure the outcomes we want is a substantially further question; and note that analogies between evolution and contemporary techniques in machine learning are somewhat sobering in this respect—see section 4.3.1.2 for more.

Still, some worry about playing with fire persists. As our own impact on the earth illustrates, intelligent agents can be an extremely powerful force for controlling and transforming an environment in pursuit of their objectives. Indeed, even on the grand scale of earth’s history, the development of human abilities in this respect seems like a very big deal---a force of unprecedented potency. If we unleash much more of this force into the world, via new, more intelligent forms of non-human agency, it seems reasonable to expect dramatic impacts, and reasonable to wonder how well we will be able to control the results. 14 14 14 Of course, the most relevant and powerful “human actors” are and have been groups (countries, corporations, ideologies, cultures, etc), equipped with technology, rather than unaided individuals. But non-human agents can create groups, and use/create new technology, too (indeed, creating and using technology more effectively than humans is one of the key things that makes advanced AI systems formidable). And note that while human organizations like corporations are much more capable than individuals along certain axes (especially e.g. tasks that can be effectively parallelized), they remain constrained by human abilities, and guided by human values, in key ways—ways that fully “non-human” corporations would not (see here for more discussion). That said, I think questions around the role of individuals vs. group agents/”superorganisms” as central loci of influence, and questions around the role of human intelligence vs. technology, culture, economic interaction, etc, in explaining human growth and dominance on this planet, are both key ways in which “building agents more much intelligent than humans warrants caution” might mislead—and I think a full fleshing out of the “backdrop picture” here requires more detail than I’ve gestured at above (and perhaps such a picture requires fuller automation of human cognitive abilities than the “advanced capability” condition used in this report implies). Thanks to Katja Grace and Ben Garfinkel for discussion.

The rest of this report focuses on a specific version of this sort of worry, in the context of artificial systems in particular.

#### 1.2.4 Power

This version centers on the following hypothesis: that by default, suitably strategic and intelligent agents, engaging in suitable types of planning, will have instrumental incentives to gain and maintain various types of power, since this power will help them pursue their objectives more effectively (see section 4.2 for more discussion). 15 15 15 By “power” I mean something like: the type of thing that helps a wide variety of agents pursue a wide variety of objectives in a given environment. For a more formal definition, see Turner et al (2020) . The worry is that if we create and lose control of such agents, and their objectives are problematic, the result won’t just be damage of the type that occurs, for example, when a plane crashes, or a nuclear plant melts down—damage which, for all its costs, remains passive. Rather, the result will be highly-capable, non-human agents actively working to gain and maintain power over their environment—agents in an adversarial relationship with humans who don’t want them to succeed.

Nuclear contamination is hard to clean up, and to stop from spreading. But it isn’t trying to not get cleaned up, or trying to spread---and especially not with greater intelligence than the humans trying to contain it. 16 16 16 Harmful structures actively optimized for self-replication—like pathogens, computer viruses, and “ grey goo ”—are a closer analogy, but these, too, lack the relevant type of agency and intelligence. But the power-seeking agents just described would be trying , in sophisticated ways, to undermine our efforts to stop them. If such agents are sufficiently capable, and/or if sufficiently many of such failures occur, humans could end up permanently disempowered, relative to the power-seeking systems we’ve created.

This difference—between the usual sort of damage that occurs when a piece of technology malfunctions, and the type that occurs when you lose control over strategically sophisticated, power-seeking agents whose objectives conflict with yours—marks a key distinction not just between worries about AI vs. other sorts of risks, but also between the specific type of AI-related worry I focus on in what follows, and the more inclusive set of worries that sometimes go under the heading of “AI alignment.”

As with any technology, AI systems can fail to behave in the way that their designers intend. And because some AI systems pursue objectives, some such unintended behavior can result from problems with their objectives in particular (call this particular type of unintended behavior “misaligned”). And regardless of the intentions of the designers, the development and social impact of AI systems can fail, more broadly, to uphold and reflect important values.

Problems in any of these veins are worth addressing. But it is power-seeking , in particular, that seems to me the most salient route to existential catastrophe from unintended AI behavior. AI systems that don’t seek to gain or maintain power may cause a lot of harm, but this harm is more easily limited by the power they already have. And such systems, by hypothesis, won’t try to maintain that power if/when humans try to stop them. Hence, it’s much harder to see why humans would fail to notice, contain, and correct the problem before it reaches an existential scale.

This, then, is the backdrop picture underlying the specific arguments I’ll examine: non-human agents, much more intelligent than humans, would likely be potent forces in the world; and if their objectives are problematic, some of them may, by default, seek to gain and maintain power. Building such agents therefore seems like it could risk human disempowerment, at least in principle. Let’s look more closely at whether to expect this in practice.

## 2 Timelines

For existential risks from power-seeking AI agents to arise, it needs to become possible and financially feasible for humans to build relevantly dangerous AI systems. In this section, I describe the type of system I’m going to focus on, and I discuss the probability that humans learn to create such systems by 2070.

### 2.1 Three key properties

I’ll focus on systems with three properties: (a) advanced capabilities, (b) agentic planning, and (c) strategic awareness.

#### 2.1.1 Advanced capabilities

I’ll say that an AI system has “advanced capabilities” if it outperforms the best humans on some set of tasks which when performed at advanced levels grant significant power in today’s world. The type of tasks I have in mind here include: scientific research, engineering, business/military/political strategy, hacking, and social persuasion/manipulation. An AI system with these capabilities can consist of many smaller systems interacting, but it should suffice to ~fully automate the tasks in question.

The aim here is to hone in on systems whose capabilities make any power-seeking behavior they engage in worth taking seriously (in aggregate) as a potential route to the disempowerment of ~all humans. Such a condition does not, I think, require meeting various stronger conditions sometimes discussed 17 17 17 A level of AI progress that disempowered all humans would constitute “transformative AI” in the sense used by Karnofsky (2016) : e.g. “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution.” But that sort of transformation is precisely the type of thing we’re trying to forecast; e.g., it’s the result, not the cause. And disempowerment does not require other, more mechanistic standards for transformation—e.g. economic growth proceeding at particular rates (though we can argue, here, about the economic value that AI progress sufficient to disempower humans would represent). ---for example, ‘‘human-level AI,’’ 18 18 18 This is used in various ways, to mean something like (a) a single AI system that is in some generic sense “as intelligent” as a human; (b) a single AI system that can do anything that a given human (an average human? the “best” human? any human?) can do; (c) a level of automation such that unaided machines can perform roughly any task better and more cheaply than human workers (see Grace et al (2017) ). My favorite is (c). ‘‘superintelligence,’’ 19 19 19 Bostrom (2014, Chapter 2) defines this as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest.” A related concept requires cognitive performance that in some loose sense exceeds all of human civilization—though exactly how to understand this isn’t quite clear (and human civilization’s “cognitive abilities” change over time). or ‘‘AGI.’’ 20 20 20 “AGI” is sometimes used as a substitute for some concept of “human-level AI”; in other contexts, it refers specifically to some concept of human learning ability (see e.g. Selsam (undated) , and Arbital here (no author listed, but I believe it is Yudkowsky)), or some method of creating systems that can perform certain tasks (see Ngo (2020) on “generalization-based approaches). “AGI” is often contrasted with “narrow AI”—though in my opinion, this contrast too easily runs together a system’s ability to learn tasks with its ability to perform them. And sometimes, a given use of “AGI” just means something like “you know, the big AI thing; real AI; the special sauce; the thing everyone else is talking about.” That said, I’m erring, here, on the side of including ‘‘weaker’’ systems --- including some that might not, on their own (or even in aggregate), be all that threatening (a fact worth bearing in mind in assigning overall probabilities). 21 21 21 Put another way, I’m erring on the side of “necessary” rather than “sufficient” for riskiness. I have yet to hear a good capability threshold that is both necessary and sufficient—and I’m skeptical that one exists. Stronger conditions have less of this problem; and I expect much of the discussion in what follows to apply to stronger systems, too.

Admittedly, the standard here is imprecise. And notably, what sorts of task performance yields what sort of real-world power can shift dramatically as the social and technological landscape changes (see discussion in 6.3). I’m going to tolerate this imprecision in what follows, but those who want more precise definitions should feel free to use them---the discussion doesn’t depend heavily on the details. 22 22 22 My own favored more precise threshold would be something like “a level of automation such that unaided machines can perform roughly any task better and more cheaply than human workers”; but I don’t think this is necessary for the threats in question to arise.

#### 2.1.2 Agentic planning

I’ll say that a system engages in ‘‘agentic planning’’ if it makes and executes plans, in pursuit of objectives, on the basis of models of the world (to me, this isn’t all that different from bare ‘‘agency,’’ but I want to emphasize the planning aspect). 23 23 23 I like emphasizing planning in part because it seems to me more tempting to speak loosely about systems as “trying” to do things (e.g., your microwave is “trying” to warm your food, a thermostat is “trying” to keep your house a certain temperature), than to speak of them as “planning” to do so.

The aim here (and in the next subsection) is to hone in on the type of goal-oriented cognition required for arguments about the instrumental value of gaining/maintaining power to be relevant to a system’s behavior. 24 24 24 We can imagine cases in which AI agents end up valuing power for its own sake, but I’m not going to focus on those here. That said, muddyness about abstractions in this vicinity is one of my top candidates for ways arguments of the type I consider might mislead; and I expect that as we gain familiarity with advanced AI systems, many of the concepts we use to think about them will alter and improve. However: I also think of many worries about existential risk from misaligned power-seeking as centrally animated by some concepts in this broad vicinity (though not necessarily mine in particular). I don’t think such concepts useless, or obviously dispensable to the argument; and for now, I’ll use them. 25 25 25 Of course, you can just talk directly about AI systems that end up causing (suitably unintended) existential catastrophes, without invoking concepts like agency, objectives, etc. The question, though, is why one might expect that sort of behavior. And especially when the existential catastrophes involve AI power-seeking in particular (as I’m inclined to think the most worrying ones do), I think something like “instrumental convergence” is the strongest argument. That said, it might be possible to put instrumental convergence-type considerations in terms that appeal less to agency, objectives, strategic reasoning, and so forth, and more to the types of novel generalization we might expect from a given sort of training. I briefly explore this possibility in a footnote at the end of section 4.2. I’m open to this approach, but my current feeling is that this argument is strongest when it leans on an (implicit or explicit) expectation that something like agentic planning, in pursuit of objectives, will develop in the systems in question.

A few clarifications, though.