Is Power-Seeking AI an Existential Risk? — part 12/16

Cleaned-text code points [169059, 183988). Bibliography omitted here; source.md includes it.

Discussions of existential risk from misaligned AI often focus on the transition from some lower (but still more advanced than today) level of frontier AI capability (call it ‘‘A’’) to some much higher and riskier level (call it ‘‘B’’). Call this transition ‘‘take-off.’’ 141 141 141 See e.g. Bostrom (2014), Chapter 4 . Obviously, the “level” of AI capability available is highly multidimensional; but how we define it doesn’t much matter at present.

In particular, some of the literature focuses on especially dramatic take-off scenarios. We can distinguish between a number of related variants:

1. “Fast take-off”: that is, escalation from A to B that proceeds very rapidly;

2. ‘‘Discontinuous take-off’’: that is, escalation from A to B that proceeds much faster than some historical extrapolation would have predicted; 142 142 142 See the AI Impacts project on discontinuities for more.

3. “Concentrated take-off”: that is, escalation from A to B that leaves one actor or group (including a PS-misaligned AI system itself) with much more powerful AI-based capabilities than anyone else;

4. ‘‘Intelligence explosion’’: that is, AI-driven feedback loops lead to explosive growth in frontier AI capabilities, at least for some period (on my definition, this need not be driven by a single AI system ‘‘improving itself’’---see below; and note that the assumption that feedback loops explode, rather than peter out, requires justification). 143 143 143 This could in principle proceed via non-artificial forms of intelligence, too; but I’m leaving that aside.

5. “Recursive self-improvement”: that is, some particular AI system applying its capabilities to improving itself , then repeatedly using its improved abilities to do this more (sometimes assumed or expected to lead to an intelligence explosion; though as above, feedback loops can just peter out instead).

These are importantly distinct. Thus, for example, take-off can be fast, but still continuous (in line with previous trends), distributed (no actor or group is far ahead of another), and driven by factors other than AI-based feedback loops (let alone the self-improvement efforts of a single system). 144 144 144 Bostrom (2014) ’s notion of a “hardware overhang” is an example of fast take-off not driven by feedback loops.

Perhaps because of the emphasis in the previous literature, some people, in my experience, assume that existential risk from PS-misaligned AI requires some combination of (1)–(5). I disagree with this. I think (1)–(5) can make an important difference (see discussion of a few considerations below), but that serious risks can arise without them, too; and I won’t, in what follows, assume any of them.

### 6.2 Warning shots

Weaker systems are easier to correct, and more likely to behave badly in contexts where they will get corrected (more strategic systems can better anticipate correction). Plausibly, then, if practical PS-alignment is a problem, we will see and correct various forms of misaligned power-seeking in comparatively weak (but still strategically aware) systems (indeed, we may well devote a lot of the energy to trying to trigger tendencies towards misaligned power-seeking, but in contained environments we’re confident we can control). Let’s call these “warning shots.”

Warning shots should get us very worried. If early, strategically-aware AI agents show tendencies to try to e.g. lie to humans, break out of contained environments, get unauthorized access to resources, manipulate reward channels, and so forth, this is important evidence about the degree of PS-alignment the techniques used to develop such systems achieve—and plausibly, of the probability of significant PS-alignment problems more generally. Receiving such evidence should make us sit up straight.

Indeed, precisely because warning shots provide such tangible evidence, it seems preferable, other things equal, for them to occur earlier on in the process of AI development. Earlier warning shots are more easily controlled, and they leave more time for the research community and the world to understand their implications, and to try to address the problem. 145 145 145 Though note that warning shots for the most worrying types of misaligned power-seeking—for example, acting aligned in a training/testing environment, while planning to seek power once deployed—require fairly strategically sophisticated systems: e.g., ones that have models of themselves, humans, the world, the difference between training/testing and deployment, and so forth. For example: I don’t think GPT-3 giving false information qualifies.

By contrast, if there is very little calendar time between the first significant warning shots and the development of highly capable, strategically-aware agents, there will be less time for the evidence that warning shots provide to be reflected in the world’s AI-related research and decision-making. 146 146 146 Though note that some decisions—for example, to recall a product—can be made quickly. Thanks to Ben Garfinkel for suggesting this. This is one of the worrying features of scenarios where frontier capabilities escalate very rapidly.

It’s sometimes thought that in scenarios where frontier AI capabilities don’t escalate rapidly, warning shots will suffice to alert the world to PS-alignment problems (if they exist), and to prompt adequate responses. 147 147 147 Garfinkel (2020): “If you expect progress to be quite gradual, if this is a real issue, people should notice that this is an issue well before the point where it’s catastrophic. We don’t have examples of this so far, but if it’s an issue, then it seems intuitively one should expect some indication of the interesting goal divergence or some indication of this interesting phenomenon of this new robustness of distribution shift failure before it’s at the point where things are totally out of hand. If that’s the case, then people presumably or hopefully won’t plough ahead creating systems that keep failing in this horrible, confusing way. We’ll also have plenty of warning you need, to work on solutions to it.” But relying on this seems to me overoptimistic, for a number of reasons.

First, recognizing a problem is distinct from solving it. Warning shots may prompt more attention to PS-alignment problems, but that attention may not be enough to find solutions, especially if the problems are difficult. And certain sorts of “solutions” may function as band-aids; they correct a system’s observed behavior, but not the underlying issue with its objectives. For example, if you train a system by penalizing it for lying, you may incentivize “don’t tell lies that would get detected,” as opposed to “don’t lie” (and the training process itself might provide more information about which lies are detectable).

Second, there are reasons to expect fewer warning shots as the strategic and cognitive capabilities of frontier systems increase, regardless of whether techniques for ensuring the practical PS-alignment have adequately improved. This is because more capable systems, regardless of their PS-alignment, will be better able to model what sorts of behavior humans are looking for, and to forecast what attempts at power-seeking will be detected and corrected---a dynamic that could lead to a misleading impression that earlier problems have been adequately addressed; or even, that those problems stemmed from lack of intelligence rather than alignment. 148 148 148 See e.g. the roll-out scenario described in Bostrom (2014) . It’s worth noting that in principle, if the ability to accurately assess whether a given instance of misaligned power-seeking will succeed arises sufficiently early in the AI system’s we’re building/training, the period of time in which we see widespread and overt misaligned power-seeking from such agents could be quite short, or even non-existent. That is, it could be that the type of strategic awareness that makes an AI agent aware of the benefits of seeking real-world forms of power, resources, etc is closely akin to the type that makes an AI agent aware of and capable of avoiding the downsides of getting caught. If the former is in place, the latter may follow fast—even if the trajectory of AI capability development in general is more gradual.

Third, even if there is widespread awareness that existing techniques for ensuring practical PS-alignment are inadequate, various actors might still push forward with scaling up and deploying highly-capable AI agents, either because they have lower risk estimates, or because they are willing to take more risks for the sake of profit, power, short-term social benefit, competitive advantage, etc. Indeed, it seems plausible to me that at a certain point, basically all of the reasonably cautious and socially-responsible actors around the world will know full well that various existing highly-capable AI agents are prone to misaligned power-seeking in certain not-out-of-the-question circumstances. But this won’t be enough to prevent such systems from getting used (and as I discussed in 5.3.1, if incautious actors use such systems, this will put competitive pressure on cautious actors to do so as well).

Here, again, climate change might be an instructive analogy. The first calculations of the greenhouse effect occurred in 1896; the issue began to receive attention in the highest levels of national and international governance in the late 1960s; and scientific consensus began to form in the 1980s. 149 149 149 See Wikipedia . Yet here we are, more than 30 years later, with the problem unsolved, and continuing to escalate---thanks in part to the multiplicity of relevant actors (some of whom deny/minimize the problem even in the face of clear evidence), and the incentives and externalities faced by those in a position to do harm. There are many disanalogies between PS-alignment risk and climate change (notably, in the possible---though not strictly necessary---immediacy, ease of attribution, and directness of AI-related harms), but I find the comparison sobering regardless. 150 150 150 Thanks to Ben Garfinkel and Holden Karnofsky for suggesting disanalogies. At least in some cases, “warnings” aren’t enough.

### 6.3 Competition for power

Deployed APS systems seeking power in misaligned ways would be competing for power with humans (and with each other). This subsection discusses some of the relevant features of that competition, and what sorts of mechanisms for seeking power might be available to the AI systems involved.

A few points up front. First: discussion of existential risk from AI sometimes assumes that the central threat is a single PS-misaligned agent that comes to dominate the world as a whole—e.g., what Bostrom (2014) calls a “unipolar scenario.” But the key risk is broader: namely, that ~all humans are permanently and collectively disempowered , whether at the hands of one AI system, or many.

Thus, for example, existential catastrophe can stem from many PS-misaligned systems, each with comparable levels of capability, engaged in complex forms of competition and coordination (this is an example of a “multipolar” scenario): what matters is whether humans have been left out. Here we might think again of analogies with chimpanzees: no single human or human institution rules the world, but the chimps are still disempowered relative to humans.

In this sense, questions about e.g. what sorts of take-off lead to unipolar scenarios, and what sort will occur, don’t settle questions about risk levels. Indeed, regardless of take-off, if we reach a point where (a) basically all of the most capable APS systems are seeking power in misaligned ways (for example, because of widespread problems ensuring scalable and competitive forms of practical PS-alignment), and (b) such systems drive and control most of the scientific, technological, and economic growth occurring in the world, then the human position seems to me tenuous.

Second: the success or failure of a given instance of misaligned power-seeking depends both on the absolute capability of the power-seeking system, and on the strength of the constraints and opposition that it faces. 151 151 151 See e.g. Drexler (2019, Chapter 31) ’s distinction between “supercapabilities” and “superpowers.” And in this latter respect, the world that future power-seeking AI systems would be operating in would likely be importantly different from the world of 2021.

In particular, such a world would likely feature substantially more sophisticated capacities for detecting, constraining, responding to, and defending against problematic forms of AI behavior—capacities that may themselves be augmented by various types of AI technology, including non-agentic AI systems, specialized/myopic agents, and other AI systems that humans have succeeded in eliciting aligned behavior from, at least in some contexts. And even setting aside human opposition, a given PS-misaligned system might have other, different PS-misaligned systems to contend (or, perhaps, cooperate) with as well; and the dynamics of cooperation and competition between human and non-human agents could become quite complex.

Third, even in the context of absolute capability, there is an important difference between ‘‘better than human’’ and ‘‘arbitrarily capable’’---whether in one domain, or many---and PS-misaligned APS systems might fall on a wide variety of points in between. 152 152 152 See e.g. Arbital’s discussion of different variants here . We humans certainly can’t predict all of the options and strategies that would be available to such systems, and we should be wary of ruling out possibilities with too much confidence. But we shouldn’t assume that all physically possible forms of competence, knowledge, predictive ability, and so forth are in play, either.

#### 6.3.1 Mechanisms

With those points in mind, let’s look briefly at some factors and mechanisms that could be relevant to the scope and success of a given instance of PS-misaligned behavior. 153 153 153 This section draws on some of Bostrom’s (2014) discussion of possible AI “superpowers.” Below are a few that seem salient to me. I include these partly for the sake of concreteness, and partly to illustrate that power-seeking by AI systems---even very capable ones---may be subject to important constraints and limitations: the move from ‘‘seeks power’’ to ‘‘gets/keeps power,’’ especially on large scales, is non-trivial. 154 154 154 Though note that strategically sophisticated systems are less likely to make power-seeking efforts likely to fail.

Obviously, any discussion in this respect will be speculative and incomplete (not to mention strange and “science-fiction-y”), and any actual PS-misaligned scenarios will likely involve factors and mechanisms that we are in a very poor position to anticipate. The examples I give are just meant as illustrations of possible dynamics.