Is Power-Seeking AI an Existential Risk? — part 11/16

Cleaned-text code points [153109, 169059). Bibliography omitted here; source.md includes it.

Dysfunctional forms of competition between actors could also incentivize risk-taking, especially to the extent that there are significant advantages to gaining/maintaining some relative position in an AI technology ‘‘race.’’ 135 135 135 See Askell et al (2019) for discussion of first-mover advantages. Bostrom (2014) ’s notion of a “decisive strategic advantage” at stake in the AI race is an extreme example. Because time and effort devoted to ensuring practical PS-alignment trade off against the speed with which one can scale up the capabilities of state of the art systems, an actor who might’ve otherwise decided to put in more of such time and effort, if the advantages of a given relative position (for example, first-mover advantages) were secure, could be incentivized, in a more competitive context, to accept increased risk in order to gain or maintain such a position. Indeed, in an especially bad version of this dynamic, the other competitors might then be likewise incentivized to take on increased risk as well, thereby creating further incentives for the first actor take on more risk, and so forth—an ongoing feedback loop of increasing pressure on all parties to either to up their risk tolerance, or drop out of the race.

That said, granted that dynamics like this are possible , it’s a substantially further question what sorts of externalities and competitive dynamics will actually apply to a given sort of AI technology in a given context. There are competitive dynamics and first-mover advantages in many industries (for example, pharmaceuticals ), but that doesn’t mean we see a race to the bottom on safety, possibly because various mechanisms---market forces, regulation, legal liability---have raised the ‘‘bottom’’ sufficiently high; 136 136 136 See Askell et al (2019, p. 9) ’s discussion of pharmaceuticals; and see Hunt (2020) on the possibilities of “races to the top.” and many of these mechanisms function to incorporate various potential externalities as well. 137 137 137 Though as the case of climate change shows, they do not always do this adequately—and negative externalities from deploying practically PS-misaligned AI systems could be extreme. And more generally: abstract, game-theoretical models are one thing; the concrete mess of social and political reality, quite another.

#### 5.3.2 Number of relevant actors

Once some actors can create APS systems, then over time, and absent active efforts to the contrary, a larger and larger number of actors around the world will likely become able to do so as well. Absent significant coordination among such actors, then, it seems likely that we will see substantial variation in the beliefs, values, and incentives that inform their decision-making.

Of course, safety-relevant properties of AI systems can be correlated, even without active coordination amongst relevant actors. The ease of ensuring practical PS-alignment, for example, represents one source of correlation, and there may be others (see Hubinger (2020) for discussion).

But to the extent that such correlation does not ensure practical PS-alignment by default (such that e.g. some substantive level of caution and social responsibility is required), then larger numbers of relevant actors increase the risk that some sort of failure will occur. That is: even if many actors are adequately cautious and responsible, some plausibly won’t be. For example, some could be overconfident in the practical PS-alignment of the systems they’ve developed, or dismissive of the level of harm that PS-misaligned behavior would cause. Others may have greater tolerance for risk, or act with more concern for profit and individual power than for the social costs of their actions, or see themselves as having more to gain from a particular type of AI-based functionality. Some may be operating in the absence of various market, regulatory, and liability-related incentives that apply to other actors. And so on. 138 138 138 Here we might think of analogies with the “ winner’s curse .”

Indeed, to the extent that resources invested in ensuring practical PS-alignment trade off against resources invested in increasing the capabilities of the systems one builds, over time we might expect to see actors who invest less in alignment, and who take more risks, to scale up the capabilities of their systems faster. This could result in the competitive dynamics discussed above (e.g., other actors cut back on safety efforts to keep up, and/or deploy systems that wouldn’t meet their own safety standards, but which are safer than the ones they expect competitors to deploy); but if other actors don’t cut back on safety as a result, the most powerful systems might end up increasingly in the hands of the least cautious and socially responsible actors (though there are also important correlations between social responsibility and factors like resource-access, talent, etc).

That said, as above, similar abstract dynamics plausibly apply in many industries, and it’s an empirical question, dependent on a wide variety of factors, what sorts of safety problems actually result. AI is no different.

#### 5.3.3 Bottlenecks on usefulness

The benefits of deployment listed in the box above—profit, power, prestige, solving social problems, etc—all require the APS system, once deployed, to be useful in various ways. If such a system is misaligned in a way that renders it obviously not useful, then, we shouldn’t expect to see it intentionally deployed.

As I noted at the beginning of the section, this is an important constraint on how we should expect alignment-like problems to show up in the real world, as opposed to the lab. In general, if we can’t get our AI systems to do things like understand what we want, follow instructions in charitable and common-sensical ways, check in with us when they’re uncertain, refrain from immediately trying to hack their reward systems, and so forth, then their usefulness to us will be severely limited. And even very incautious and socially-irresponsible actors are likely to test a system extensively before deploying it.

The question, then, isn’t whether relevant actors will intentionally deploy systems that are already blatantly failing to behave as they intend. The question is whether the standards for good behavior they apply during training/testing will be adequate to ensure that the systems in question won’t seek power in misaligned ways on any inputs post-deployment.

The issue is that good behavior during (even fairly extensive) training/testing doesn’t necessarily demonstrate this. This is partly due to possible deception/manipulation on the part of the AI systems (see next subsection). But even absent deception/manipulation of this kind, it can be extremely difficult for an actor to predict/test the AI’s behavior on the full range of post-deployment inputs—especially in a rapidly changing world, in the absence of deep understanding of how the system works (see section 4.4.1 ), and if the AI system might gain new knowledge and capabilities post-deployment (see section 4.3.2.2 ).

Indeed, I think that one of the central reasons we should expect to see practically PS-misaligned AI systems getting used/deployed is precisely that they will demonstrate a high degree of usefulness during training/testing—and consequently, it will be increasingly difficult to resist deploying them, especially in the context of competitive dynamics like the ones described in 5.3.1.

Here’s an analogy. Suppose that scientists create a new, genetically-engineered species of chimpanzee, whose cognitive capabilities significantly exceed those of humans. Initially, scientists confine these chimps in a laboratory environment, and incentivize them to perform various low-stakes intellectual tasks using rewards like food and entertainment. 139 139 139 Per my comments in section 1.1 , I’m leaving aside questions about the ethical implications of treating the chimps this way, despite their salience. And suppose, further, that these chimpanzees are clearly capable of generating things like vaccine designs, prototypes for new clean energy technology, cures for cancer, highly effective military/political/business strategies, and so forth—and that they will in fact do this, if you set up their incentives right (even though they don’t intrinsically value being helpful to humans, and so are disposed, in some circumstances, to seize power for themselves—for example, if they can get more food and entertainment by doing so).

In such a context, I think, it would become increasingly difficult for various actors around the world to resist drawing on the intellectual capabilities of the chimps in a manner that gives the chimps real-world forms of influence. If a new Covid-19 style pandemic started raging, for example, and we knew that the chimps could rapidly design a vaccine, there would be strong pressure to use them for doing so. If the chimps can help “users” win a senate race, or save the lives of millions, or end climate change, or make a billion dollars, or achieve military dominance, then some people, at least, will be strongly inclined to use them, even if there are risks involved—and those who don’t use them will end up losing their senate races, falling behind their business and military competitors, and so forth.

And even if the chimps, at the beginning, are appropriately contained and incentivized to be genuinely cooperative, it seems unsurprising if, as people draw on their capacities in more and more ways around the world, they get exposed to opportunities and circumstances that incentivize them to seek power for themselves, instead.

Something similar, I think, might apply to APS AI systems. Indeed, even if people know , or strongly suspect, that such systems would seek power in misaligned ways in some not-out-of-the-question circumstances, the pull towards using them for goals that matter a lot to us may simply be too great. When pandemics are raging, oceans are rising, parents and grandparents are dying of cancer, rival nations are gaining in power, and billions (or even trillions) of dollars are sitting on the table, concerns about science-fictiony risks from power-seeking AI systems may, especially for some relevant actors, take a backseat.

#### 5.3.4 Deception

This sort of issue—e.g., that “useful” may come apart from “practically PS-aligned”—could be importantly exacerbated by the fact, discussed in section 4.4.2 and elsewhere, that less-than-fully aligned APS systems with suitably long-term objectives may be actively optimizing for getting deployed, since deployment grants them greater influence in the world. This could incentivize them to deceive or manipulate relevant decision-makers—and if they are very capable, their abilities in this respect may exceed our ability to detect and correct the deceitful/manipulative behavior in question.

For example: we should expect suitably sophisticated and strategically aware systems to understand what sorts of behavior humans are looking for during training/testing, even if their objectives don’t intrinsically motivate such behavior. So if they are optimizing for getting deployed, they will have strong instrumental incentives to behave well, to demonstrate the type of usefulness (described above) that will pull us towards deploying them, and to convince us that their objectives are fully (or at least sufficiently) aligned with ours. Indeed, they’ll even have incentives to appeal to ethical concerns about how it is morally appropriate to treat them—incentives that will apply regardless of the legitimacy of those concerns (though I also expect such concerns to be legitimate in at least some cases).

Of course, human decision-makers will also be aware of the possibility of this sort of behavior. Indeed, if such behavior arises in sophisticated systems, we will likely see rudimentary forms of it in more rudimentary systems, too—akin, perhaps, to the types of lies that children tell, but that adults can easily spot. But just because we know about a problem, and/or have encountered and maybe even solved it with more rudimentary systems, this doesn’t mean we’ll have solved it for all levels of cognitive capability—especially levels much higher than our own. Detecting lies and manipulation attempts in your children is one thing; in adults much smarter and more strategically sophisticated than yourself, it’s quite another. And deceptive/manipulative AI systems will have incentives to make us think we’ve solved the problem of AI deception/manipulation, even if we haven’t.

This isn’t to say that humans will be actually fooled; and some AI systems might themselves be able to help with our efforts to detect deception in others. But unless we can develop deep understanding of and control over the objectives our AI systems are pursuing, evidence like “it performs well on all the tests we ran, including tests designed to detect deceptive/manipulative behavior” and “it clearly knows how to behave as we want” may tell us much less about its ultimate objectives, or about how it will behave once deployed, than we wish. And in the context of such uncertainty, some humans will be more willing to gamble than others.

### 5.4 Overall risk of problematic deployment

Summing up this section, then: I don’t think we should expect obviously non-useful, practically PS-misaligned APS systems to get intentionally deployed. Some systems might get deployed unintentionally, but the key risk, I think, is that an increasingly large number of relevant actors, with varying beliefs, incentives, and levels of social-responsibility, will be increasingly drawn to deploy strategically-aware AI agents that demonstrate their usefulness and apparent good behavior during training/testing (perhaps because they are deceptive/manipulative, or perhaps because their behavior is genuinely aligned on the training/testing inputs).

If, as seems plausible to me, it will be much easier to build less-than-fully PS-aligned systems that meet this standard than fully PS-aligned ones, we should expect some of the systems people are strongly pulled towards deploying to be liable to misaligned power-seeking on at least some physics-compatible inputs. And if, as seems plausible to me, it is difficult to adequately predict and control the full range of inputs a system will receive, and how it will behave in response (especially if the world is rapidly changing, you don’t deeply understand how the AI system works, and/or the AI has been actively deceptive or manipulative during the training/testing process), we should expect some such misaligned power-seeking to in fact occur—not just in a controlled laboratory/testing environment, but via channels of influence on the real world.

It is a further question, though, what happens then: that is, whether misaligned efforts on the part of strategically aware AI agents to gain and maintain power actually succeed, and on what scale. Let’s turn to that question now.

## 6 Correction

In many contexts, if an AI system starts seeking to gain/maintain power in unintended ways, the behavior may well be noticed, and the system prevented from gaining/maintaining the power it seeks. Let’s call this “correction.”

Some types of correction might be easy (e.g., a lab notices that an AI system tried to open a Bitcoin wallet, and shuts it down). Others might be much more difficult and costly (for example, an AI system that has successfully hacked into and copied itself onto an unknown number of computers around the world might be quite difficult to eradicate). 140 140 140 This is a point made in Ord (2020) , and by Bostrom in his TED talk.

Confronted with post-deployment PS-alignment failures, will humanity’s corrective efforts be enough to avert catastrophe? I think they might well; but it doesn’t seem guaranteed. Let’s look at some considerations.

### 6.1 Take-off