Is Power-Seeking AI an Existential Risk? — part 14/16

Cleaned-text code points [199946, 215663). Bibliography omitted here; source.md includes it.

I’ve been assuming, here, that humans will, by default, be unwilling to let AI systems keep any power they succeed in gaining via misaligned behavior. But especially in multipolar scenarios, we can also imagine cases in which humans are either unable to correct a given type of misaligned power-seeking, or unwilling to pay the costs required to do so. 163 163 163 For example, perhaps a practically PS-misaligned AI system has created sufficiently many back-up copies of itself that humans can’t destroy them all without engaging in some extreme type of technological “reset,” like shutting down the internet or destroying all existing computers; or perhaps an AI system has gained control of sufficiently powerful weapons that ongoing conflict would be extremely destructive.

In such scenarios, it may be possible to reach various types of compromise arrangements, or to limit the impact of a given PS-misaligned system or systems to some contained domain, such that humans end up sharing power with practically misaligned AI systems, but not losing power entirely. These are somewhat strange scenarios to imagine, and I won’t analyze them in any depth here.

I’ll note, though, that if we reach such a point, the situation has likely become quite dire; and we might wonder, too, about its long-term stability. In particular, if the relevant PS-misaligned systems have sufficiently long-term goals, and have already been seeking power in misaligned and uncorrectable ways, then they will likely have incentives to continue to increase their power—and their ongoing presence in the world will continue to give them opportunities to do so.

## 7 Catastrophe

A final premise is that the permanent and unintentional disempowerment of ~all humans would be an existential catastrophe.

Precise definitions can matter here, but loosely, and following Ord (2020) , I’ll think of an existential catastrophe as an event that drastically reduces the value of the trajectories along which human civilization could realistically develop (see footnote for details and ambiguities). 164 164 164 Ord (2020) ’s definition of existential catastrophe—that is, “the destruction of humanity’s longterm potential”—invokes “humanity,” but he notes that “If we somehow give rise to new kinds of moral agents in the future, the term ‘humanity’ in my definition should be taken to include them” (p. 39); and he notes, too, that “I’m making a deliberate choice not to define the precise way in which the set of possible futures determines our potential. A simple approach would be to say that the value of our potential is the value of the best future open to us, so that an existential catastrophe occurs when the best remaining future is worth just a small fraction of the best future we could previously reach. Another approach would be to take account of the difficulty of achieving each possible future, for example defining the value of potential as the expected value of our future assuming we followed the best possible policy. But I leave a resolution of this to future work” (p. 37, footnote 4). That is, Ord imagines a set of possible “open” futures, where the quality of humanity’s “potential” is some (deliberately unspecified) function of that set. One issue here is that if we separate a future’s “open-ness” from its probability of occurring, then very good futures are “open” to e.g. future totalitarian regimes, or future AI systems, to choose “if they wanted,” even if their doing so is exceedingly unlikely—in the same sense that it is “open” to me to jump out a window, even though I won’t. But if we try to incorporate probability more directly (for example, by thinking of an existential catastrophe simply as some suitably drastic reduction in the expected value of the future), then we have to more explicitly incorporate further premises about the current expected value of the future; and suitably subjective notions of expected value raise their own issues. For example, if we use such notions in our definition, then getting bad news—for example, that the universe is much smaller than you thought—can constitute an existential catastrophe; and I expect we’d also want to fix a specific sort of epistemic standard for assessing the expected value in question, so such that assigning subjective probabilities to some event being an existential catastrophe sounds less like “I’m at 50% credence that my credence is above 90% that X,” and more like “I’m at 50% credence that if I thought about this for 6 months, I’d be at above 90% that X”. Like Ord, I’m not going to try to resolve these issues here. Readers should feel free, though, to substitute in their own preferred definition—the broad idea is to hone in on a category of event that people concerned about what happens in the long-term future should be extremely concerned to prevent.

It’s possible to question whether humanity’s permanent and unintentional disempowerment at the hands of AI systems would qualify. In particular, if you are optimistic about the quality of the future that practically PS-misaligned AI systems would, by default, try to create, then the disempowerment of all humans, relative to those systems, will come at a much lower cost to the future (and perhaps even to the present) in expectation.

One route to such optimism is via the belief that all or most cognitive systems (at least, of the type one expects humans to create) will converge on similar objectives in the limits of intelligence and understanding---perhaps because such objectives are ‘‘intrinsically right’’ (and motivating), or perhaps for some other reason. 165 165 165 Note that this could be compatible with Bostrom’s (2014) formulation of the “orthogonality thesis”—e.g., “Intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with any final goal.” That is, Bostrom’s formulation only applies to the “in principle” possibility of combining high intelligence and any final goal. But there could still be strong correlations, attractors, etc in practice (this is a point I first heard from David Chalmers). My own view, shared by many, is that ‘‘intrinsic rightness’’ is a bad reason for expecting convergence, 166 166 166 If, for example, you program a sophisticated AI system to try to lose at chess—see, e.g., suicide chess —it won’t, as you increase its intelligence, start to see and respond to the “objective rightness” of trying to win instead, or of trying to reduce poverty, or of spreading joy throughout the land—even after learning what humans mean when they say “good,” “right,” and so forth. See discussion in Russell (2019, p.~166) . but other possible reasons---related, for example, to various forms of cooperative game-theoretic behavior and self-modification that intelligent agents might converge on 167 167 167 For especially exotic versions of this, see Oesterheld (2017) and Fox (2020) . ---are more complicated to evaluate. 168 168 168 Though the history of atrocities committed by strategic and intelligent humans does not seem comforting in this respect; and note that the incentives at stake here depend crucially on an agent’s empirical situation, and on its power relative to the other agents whose behavior is correlated with its own. In a context where misaligned AI systems are much more powerful than humans, it seems unwise to depend on their having and responding to instrumental, game-theoretic incentives to be particularly nice. And we can imagine other routes to optimism as well—related, for example, to hypotheses about the default consciousness, pleasure, preference satisfaction, or partial alignment of the AI systems that disempowered humans.

I’m not going to dig in on this much. I do, though, want to reiterate that my concern here is with the unintentional disempowerment of humanity. That is, sharing power with AI agents—especially conscious and cooperative ones—may ultimately be the right path for humanity to take. But if so, we want it to be a path we chose , on purpose, with full knowledge of what we were doing and why: we don’t want to build AI agents who force such a path upon us, whether we like it or not.

I think the moral situation here is actually quite complex. Suitably sophisticated AI systems may be moral patients; morally insensitive efforts to use, contain, train, and incentivize them risk serious harm; and such systems may, ultimately, have just claims to things like political rights, autonomy, and so forth. In fact, I think that part of what makes alignment important, even aside from its role in making AI safe, is its role in making our interactions with AI moral patients ethically acceptable. 169 169 169 Thanks to Katja Grace for discussion of this point. It’s one thing if such systems are intrinsically motivated to behave as we want; it’s another if they aren’t, but we’re trying to get them to do so anyway. And more generally: once you build a moral patient, this creates strong moral reasons to treat it well—and what “treating artificial moral patients well” looks like seems to me a crucial question for humanity as we transition into an era of building systems that might qualify. At present, as far as I can tell, we have very little idea how to even identify what artificial systems warrant what types of moral concern. In a deep sense, I think, we know not what we do.

But some moral patients—and some agents who might, for all we know, be moral patients, but aren’t—will also try to seize power for themselves, and will be willing to do things like harm humans in the process. So building new, very powerful agents who might be moral patients is, not surprisingly, both a morally and prudentially dangerous game: one that humanity, plausibly, is not ready for. My assumption, in this report, has been that unfortunately, we—or at least, some of us—are going to barrel ahead anyway, and I fear we will make many mistakes, both moral and prudential, along the way.

The point, then, is not that humans have some deep right to power over AI systems we build. Rather, the point is to avoid losing control of our AI systems before we’ve had time to develop the maturity to really understand what is at stake in different paths into the future—including paths that involve sharing power with AI systems—and to choose wisely amongst them.

## 8 Probabilities

(May 2022 author’s note: since making this report public in April 2021, my probability estimates – discussed in this section – have shifted. My current overall probability of existential catastrophe from power-seeking AI by 2070 is >10%.)

To sum up, and with the preceding discussion in mind, let’s return to the full argument I opened with. To illustrate my current (unstable, subjective) epistemic relationship to the premises of this argument, I’ll add some provisional credences to each, along with a few words of explanation. 170 170 170 See footnote 4 here for more description of how to understand probabilities of this kind.

To be clear: I don’t think I’ve formulated these premises with adequate precision to really ‘‘forecast’’ in the sense relevant to e.g. prediction markets; and in general, the numbers here (and the exercise more broadly) should be held very lightly. I’m offering these quantitative probabilities mostly because I think that doing so is preferable, relative to leaving things in purely qualitative terms (e.g., ‘‘significant risk’’), as a way of communicating my current best guesses about the issues I’ve discussed, and of facilitating productive disagreement. 171 171 171 Ord (2020) discusses the downsides of leaving risk assessments vague in this way. This disagreement would be easier if we had more operationalized versions of the premises in question, and I encourage others interested in such operationalization to attempt it (though overly-precise versions can also artificially slim down the relevant scenarios). But my hope, in the meantime, is that this is better than nothing. 172 172 172 The imprecision here does mean that people disagreeing about specific premises may not have the same thing in mind; but a single person attempting to come to their own views can hold their own interpretations in mind. And the overall probability on permanent human disempowerment by 2070 seems less ambiguous in meaning.

Even setting imprecisions aside, some worry that assigning conditional probabilities to the premises in a many-step argument risks various biases. 173 173 173 See, for example, Yudkowsky on the “ Multiple Stage Fallacy ,” and discussion from Kaufman (2016) . For example:

I. It may be difficult to adequately imagine updating on the truth of previous premises (and hence, the premises may get treated as less correlated than they are).

II. The overall verdict may be problematically sensitive to the number of premises (for example, we might be generically reluctant to assign very high probabilities to a premise, so additional premises will generally drive the final probability lower).

III. The conclusion might be true, even if some of the premises are false.

Note, though, that compressing an argument into very few premises (or just directly forecasting the conclusion) risks hiding conjunctiveness, too. 174 174 174 More generally: the possibility of biases in one method of estimation (e.g., multiplying estimates for multiple conditional premises) doesn’t show that some alternative methodology is preferable. And some things do actually require many conditional premises to be true.

As an initial step in attempting to combat (II), and possibly (I), I’ve added a short appendix where I reformulate the argument using fewer premises; and to combat some other possible framing effects, I also offer versions in positive---e.g., we’ll be fine---rather than negative---e.g., we’re doomed---terms. And I’ve tried to make my own probabilities consistent across the board. 175 175 175 I also played around a bit with a rough Guesstimate model that sampled from high/low ranges for the different premises. Obviously, though, biases in the vein of (I) and (II) can still remain (among many others).

And note that (III), here, is true. That is, even limiting ourselves to existential catastrophes from power-seeking AI before 2070, estimates based on the premises I give are lower bounds: such catastrophes can occur without all those premises being true (see footnote for examples). 176 176 176 For example, we might see unintentional deployment of practically PS-misaligned APS systems even if they aren’t superficially attractive to deploy; practically PS-misaligned APS systems might be developed and deployed even absent strong incentives to develop them (for example, simply for the sake of scientific curiosity); systems that don’t qualify as APS might seek power in misaligned ways; and so on. That said, I think these premises do a decent job of representing my own key uncertainties, at least, in “chunks” that feel roughly right to me; and if I learned that one or more were false, I’d feel a lot less worried (at least for the next half-century).

I’ll also note two high-level “outside view” doubts I feel about the argument that follows:

• The general picture I’ve discussed, even apart from specific assessments of a given premise, feels to me like “a very specific way things could go.” This isn’t to say we can’t ever make specific forecasts about the future—I think we can (for example, about whether the economy will be bigger, the climate will be hotter, and so forth). But I have some background sense that visions of the future “of this type” (whatever that is) will generally be wrong—and often, in ways that discussion of those visions didn’t countenance.