Is Power-Seeking AI an Existential Risk? — part 10/16

Cleaned-text code points [138317, 153109). Bibliography omitted here; source.md includes it.

Absent counterargument, we might expect something similar with AI. Indeed, some amount of alignment seems like a significant constraint on the usefulness and commercial viability of AI technology generally. Thus, if problems with proxies, or search, make it difficult to give house-cleaning robots the right objectives, we shouldn’t expect to see lots of such robots killing people’s cats (or children); rather, we should expect to see lots of difficulties making profitable house-cleaning robots. 123 123 123 I believe I heard this example from Ben Garfinkel. Indeed, by the time self-driving cars see widespread use, they will likely be quite safe (maybe too safe, relative to human drivers they could’ve replaced earlier). 124 124 124 We might see GPT-3’s usefulness as bottlenecked in part by something like alignment, too. GPT-3’s outputs suggest that it has lots of knowledge—for example, about basic medicine—that could be useful to users (see Cotra (2021) ). But extracting that knowledge is difficult, partly because GPT-3 was trained, not to be useful to humans, but to predict the next token in human-written text. And its outputs reflect biases that make it a less attractive product to humans.

What’s more, safety failures can result, for a developer/deployer, in significant social/regulatory backlash and economic cost. The 2017 crashes of Boeing’s 737 MAX aircraft, for example, resulted in an estimated ~$20 billion in direct costs, and tens of billions more in cancelled orders. And sufficiently severe forms of failure can result in direct bodily harm to decision-makers and their loved ones (everyone involved in creating a doomsday virus, for example, has a strong incentive to make sure it’s not released).

Many incentives, then, favor safety—and incentives to prevent harmful and large-scale forms of misaligned power-seeking seem especially clear. Faced with such incentives, why would anyone use, or deploy, a strategically-aware AI agent that will end up seeking power in unintended ways?

It’s an important question, and one I’ll look at in some detail. In particular, I think these considerations suggest that we should be less worried about practically PS-misaligned agents that are so unreliably well-behaved (at least externally) that they aren’t useful, and more worried about practically PS-misaligned agents whose abilities (including their abilities to behave in the ways we want, when it’s useful for them to do so) make them at least superficially attractive to use/deploy—because of e.g. the profit, social benefit, and/or strategic advantage that using/deploying them affords, or appears to afford. My central worry is that it will be substantially easier to build that type of agent than it will be to build agents that are genuinely practically PS-aligned—and that the beliefs and incentives of relevant actors will result in such practically PS-misaligned agents getting used/deployed regardless.

### 5.1 Timing of problems

I’ll think of ‘‘deployment’’ as the point where an AI system moves out of a development/laboratory/testing environment and into a position of real-world influence (even if this influence is mediated via e.g. humans following its instructions). 125 125 125 I mean the term “testing” here to refer to something still under control of the developers, akin to the type of “safety test” one might run on a car or a rocket. In this sense, testing is part of “training” more broadly. This contrasts with other possible uses of a “train/test” distinction. This isn’t always a discrete point; sometimes, for example, it’s an ongoing process, influenced by many individual decisions to accord an AI agent somewhat greater influence. 126 126 126 See e.g. the roll-out of GPT-3, which involved some people having initial access, and now a limited API that the lab still controls (Brian Christian discusses this example on a recent podcast ). And note that if e.g. an agent continues learning “on the job,” lines between training and deployment blur yet further. For simplicity, though, I’ll think of it as a discrete point in what follows—analogous to the point where a product “launches,” “ships,” or starts really getting “used.”

We can distinguish between practical PS-alignment failures that occur before and after deployment. In the first case, that is, some of the inputs on which an APS system seeks power in misaligned ways (call these its “bad inputs”) occur in the set it encounters during training/development/testing; in the second case, it only hits bad inputs after deployment.

Other things equal, pre-deployment PS-alignment failures are preferable, because an AI system will generally be better controlled, and less able to cause harm, during training/development/testing than deployment. Indeed, some pre-deployment testing will likely aim to trigger any liability to misaligned power-seeking that an agent has.

That said, in sufficiently capable agents, misaligned power-seeking that occurs before deployment can still be quite harmful---especially if it goes uncorrected. Thus, for example, if a power-seeking system meant to be contained in some training environment, or limited in its means of influencing the outside world, manages to break out of that environment, and/or to obtain other forms of influence, it may succeed in gaining various types of real-world power even if it was never intentionally deployed (let’s call this ‘‘unintentional deployment’’). 127 127 127 See Critch and Krueger (2020) for a more detailed breakdown of possible deployment scenarios.

What’s more, and importantly, even with the heightened monitoring that training/development/testing implies, not all forms of pre-deployment misaligned power-seeking will necessarily be detected. This is especially concerning in the context of the possibly adversarial dynamics discussed in section 4.4.2 . That is, to the extent that an AI system is actively trying to get deployed (for example, because the real-world influence deployment grants would offer greater opportunity to achieve its objectives), it might do things like try deceive or manipulate its trainers, and/or to differentiate between training/testing inputs and deployment inputs, such that it behaves badly only on the latter, and once its prospects for gaining/maintaining power are sufficiently good. 128 128 128 See e.g. Christiano’s RSA-2048 example here .

Indeed, in general, as the accuracy with which practically PS-misaligned systems can anticipate the consequences of their actions grows, we should expect them, other things equal, to engage less frequently in behavior that results in forms of detection/correction that hinder their pursuit of their objectives (e.g., to make fewer “mistakes”). That said, other things may not be equal: for example, our ability to detect bad behavior can grow, too.

Pre-deployment practical alignment failures are one key route to post-deployment ones: if the AI system was already engaging in PS-misaligned behavior before deployment (for example, deceiving its trainers), it will likely continue to do so, given greater real-world influence (and again, the problem may continue to go undetected). But we can imagine post-deployment failures without pre-deployment failures, too. That is, even if the AI system was genuinely well-behaved during training/testing, deployment may bring new inputs, opportunities, and capabilities, which the training/testing process was ill-suited to anticipating.

### 5.2 Decisions

Let’s look at the factors relevant to deployment decisions in more detail.

Consider some set of decision-makers (at e.g. a lab, a company, a government, etc) deciding whether or not to deploy a given APS system, or to pursue some other alternative (running more tests, re-designing/retraining the system, scrapping the project, etc). And assume, further, that this system is in fact practically PS-misaligned: that is, if deployed, it will hit bad inputs (it may already have done so), and seek power in misaligned ways. Why would anyone ever choose to deploy such a system?

At a high-level, we can break down the factors influencing this decision into: (a) the decision-makers’ beliefs about the practical PS-alignment of the system in question; and (b) the costs and benefits they treat as relevant, conditional on the system being practically PS-aligned or misaligned.

Of course, in reality, there will likely be a diversity of beliefs and cost-benefit assessments at stake; but for simplicity I’ll treat the relevant decision-makers as unified.

With respect to beliefs: we are assuming the system is in fact practically PS-misaligned, but decision-makers need not know this, or even assign it significant probability. Rather, the accuracy of their beliefs in this respect depend on a variety of factors, including:

• the effort they have exerted to gather evidence about the agent’s alignment, and the power of the tools they have available for doing so;

• their ability to anticipate and control the circumstances to which the agent will be exposed during deployment, and the similarity of those circumstances to the training/testing distribution;

• deception/manipulation of the training/testing process on the part of the AI system;

• their previous experience with practical alignment problems, and the salience of such problems in the field/the broader culture;

• the strength of their epistemology overall.

Factors that might pull decision-makers to deploy the system, conditional on its practical PS-alignment, might include:

• profit,

• power (for themselves, for the lab/company, for country/allies, etc),

• helping solve social problems (curing diseases, designing greener technology),

• prestige/“credit”,

• the thrill and momentum of scientific progress, 129 129 129 See e.g. Geoffrey Hinton’s comments here about the prospects of discovery being “too sweet.”

• a (perceived) need to keep up with some competitor,

• and a desire to prevent someone else from deploying a comparable system first.

Note, though, that decision-makers might expect some of these benefits to apply even if the system is practically PS-misaligned. Thus:

• even if a system will seek power in misaligned ways later, decision-makers might still profit from deploying it, or use it to help solve various social problems, in the short term;

• decision-makers might expect that the misaligned power-seeking could be adequately contained/corrected, or that it would only occur on fairly small scales, or only on rare inputs.

Factors that might push decision-makers away from deploying, conditional on practical PS-misalignment, include:

• ways a given product won’t be successful/profitable if it’s unreliable or unsafe,

• legal/regulatory/reputational/economic costs from deploying systems that end up seeking power in misaligned and harmful ways,

• concern on the part of decision-makers to avoid any harm to themselves/their loved ones that could come from sufficiently large-scale PS-misalignment failures,

• altruistic concern to avoid the social costs of such failures.

Figure 1, on the following page, summarizes these factors. 130 130 130 Obviously, the decision-makers need not actually use an explicit cost-benefit/expected value framework in deciding; this is just a toy model.

### 5.3 Key risk factors

As even this quite simplified framework illustrates, many different factors affect deployment decisions. Their strength and interaction can vary, and their balance can shift over time (for example, as competitive dynamics alter, regulation increases, experience with PS-alignment problems grows, and so forth).

A few factors in particular, though, seem to me especially important, and worth highlighting.

#### 5.3.1 Externalities and competition

It can be individually rational for a given actor to deploy a possibly PS-misaligned AI system, but still very bad in expectation for society overall, if society’s interests aren’t adequately reflected in the actor’s incentives. Climate change might be some analogy. Thus, the social costs of carbon emissions are not, at present, adequately reflected in the incentives of potential emitters—a fact often thought key to ongoing failures to curb net-harmful emissions. Something similar could hold true of the social costs of actors risking the deployment of practically PS-misaligned APS systems for the sake of e.g. profit, global power, and so forth—especially given that profit, power, etc at stake could be very significant. In such a context, the risks and downsides that a less-than-fully altruistic actor would be willing to accept are one thing; the risk and downsides that human society as a whole (let alone all future generations of humans) would be willing to accept are quite another.

Of course, the personal costs to decision-makers of sufficiently high-impact forms of PS-misalignment failure (analogous, for example, to an engineered virus) could be quite high (and in some cases, immediate)---a fact that suggests important disanalogies from climate change, where personal costs to emitters are generally both minimal and delayed. 131 131 131 Thanks to Ben Garfinkel for emphasizing considerations in this vein, and for discussion. But (as with emissions) a given individual choice to deploy need not increase the risk of such high-impact harms by much. Thus, the probability of PS-misaligned behavior from the particular system in question might be low; and/or the scale of that behavior’s potential harm limited. 132 132 132 Though to the extent the harm is limited, the “stakes of error” discussed in section 4.4.3 , for individual deployment events, will be much lower. But over time, and across many actors (see next section), small risks can accumulate; and collectively, the PS-misaligned behavior of many individual (and indeed, uncoordinated) systems can add up to catastrophe, even if no single system or deployment event is the cause. 133 133 133 Christiano’s (2018) second scenario is one example of this. And in some cases (e.g., especially bad competitive dynamics, military conflict, cases where the upside of successful deployment is sufficiently high) some actors may knowingly accept non-trivial (though probably not overwhelming) risk of costs as severe as their own permanent disempowerment/death. 134 134 134 Many humans, for example, have proven willing to risk their lives for personal power, glory, national strength, military victory, a social cause, an ideology, even scientific discovery, etc. “X would involve someone knowingly risking death” doesn’t seem to me especially conclusive evidence for “X will never happen”—especially if the risk is thought small, and the upside great. Thanks to Ben Garfinkel, Nick Beckstead, and Carl Shulman for discussion of points in this vicinity.