Is Power-Seeking AI an Existential Risk? — part 15/16

Cleaned-text code points [215663, 231633). Bibliography omitted here; source.md includes it.

• The people I talk to most about these issues (who are also, not coincidentally, my friends, colleagues, etc) are heavily selected for being concerned about them. 177 177 177 And note, reader, that I, too, am heavily selected for concern about this issue, as someone who chose to write this report, to work at Open Philanthropy on existential risk, and so forth. I expect this to color my epistemic processes in various non-truth-tracking ways, many of which are difficult to correct for. I have tried to hazily incorporate these and other “outside view” considerations into the probabilities reported here. But obviously, it’s difficult.

Here, then, is the argument:

By 2070:

1. It will become possible and financially feasible to build APS systems. 178 178 178 As a reminder, APS systems are ones with: (a) Advanced capability : they outperform the best humans on some set of tasks which when performed at an advanced level grant significant power in today’s world (tasks like scientific research, business/military/political strategy, engineering, hacking, and social persuasion/manipulation); (b) Agentic planning : they make and execute plans, in pursuit of objectives, on the basis of models of the world; and (c) Strategic awareness: the models they use in making plans represent with reasonable accuracy the causal upshot of gaining and maintaining different forms of power over humans and the real-world environment.

I’m going to say: 65%. This comes centrally from my own subjective forecast of the trajectory of AI progress (not discussed in this report), which draws on various recent investigations at Open Philanthropy, along with expert (and personal) opinion. I encourage readers with different forecasts to substitute their own numbers (and if you prefer to focus on a different milestone of AI progress, you can do that, too).

2. There will be strong incentives to build APS systems | (1). 179 179 179 As a reminder, I’m using “incentives” in a manner such that, if people will buy tables, and the only (or the most efficient) tables you can build are flammable, then there are incentives to build flammable tables, even if people would buy/prefer fire-resistant ones.

I’m going to say: 80%. This comes centrally from an expectation that agentic planning and strategic awareness will be either necessary or very helpful for a variety of tasks we want AI systems to perform. I also give some weight to the possibility that available techniques will push towards the development of systems with these properties; and/or that they will emerge as byproducts of making our systems increasingly sophisticated (whether we want them to or not).

The 20% on false, here, comes centrally from the possibility that the combination of agentic planning and strategic awareness isn’t actually that useful or necessary for many tasks—including tasks that intuitively seem like they would require it (I’m wary, here, of relying too heavily on my “of course task X requires Y” intuitions). For example, perhaps such tasks will mostly be performed using collections of modular/highly specialized systems that don’t together constitute an APS system; and/or using neural networks that aren’t, in the predictively relevant sense sketched in 2.1.2-3, agentic planning and strategically aware. (To be clear: I expect non-APS systems to play a key role in the economy regardless; in the scenarios where (2) is false, though, they’re basically the only game in town.)

3. It will be much harder to develop APS systems that would be practically PS-aligned if deployed, than to develop APS systems that would be practically PS-misaligned if deployed (even if relevant decision-makers don’t know this), but which are at least superficially attractive to deploy anyway | (1)–(2).

I’m going to say: 40%. I expect creating a fully PS-aligned APS system to be very difficult, relative to creating a less-than-fully PS-aligned one with very useful capabilities—especially in a paradigm akin to current machine learning, in which one searches over systems that perform well according to some measurable behavioral metric, but whose objectives one does not directly control or understand (though 50 years is a long time to make progress in this respect). However, I find it much harder to think about the difficulty of creating a system that would be practically PS-aligned, if deployed, relative to the difficulty of creating a system that would be practically PS-misaligned, if deployed, but which is still superficially attractive to deploy.

Part of this uncertainty has to do with the “absolute” difficulty of achieving practical PS-alignment, granted that you can build APS systems at all. A system’s practical PS-alignment depends on the specific interaction between a number of variables—notably, its capabilities (which could themselves be controlled/limited in various ways), its objectives (including the time horizon of the objectives in question), and the circumstances it will in fact exposed to (circumstances that could involve various physical constraints, monitoring mechanisms, and incentives, bolstered in power by difficult-to-anticipate future technology, including AI technology). I expect problems with proxies and search to make controlling objectives harder; and I expect barriers to understanding (along with adversarial dynamics , if they arise pre-deployment) to exacerbate difficulties more generally; but even so, it also seems possible to me that it won’t be “ that hard” (by the time we can build APS systems at all) to eliminate many tendencies towards misaligned power-seeking (for example, it seems plausible to me that selecting very strongly against (observable) misaligned power-seeking during training goes a long way), conditional on retaining realistic levels of control over a system’s post-deployment capabilities and circumstances (though how often one can retain this control is a further question).

Beyond this, though, I’m also unsure about the relative difficulty of creating practically PS-aligned systems, vs. creating systems that would be practically PS-misaligned, if deployed, but which are still superficially attractive to deploy . One commonly cited route to this is via a system actively pretending to be more aligned than it is. This seems possible, and predictable in some cases; but it’s also a fairly specific behavior, limited to systems with a particular pattern of incentives (for example, they need to be sufficiently non-myopic to care about getting deployed, there need to be sufficient benefits to deployment, and so on), and whose deception goes undetected. It’s not clear to me how common to expect this to be, especially given that we’ll likely be on the lookout for it.

More generally, I expect decision-makers to face various incentives (economic/social backlash, regulation, liability, the threat of personal harm, and so forth) that reduce the attraction of deploying systems whose practical PS-alignment remains significantly uncertain. And absent active/successful deception, I expect default forms of testing to reveal many PS-alignment problems ahead of time.

That said, even absent active/successful deception, there are a variety of other ways that systems that would be practically PS-misaligned if deployed can end up superficially attractive to deploy anyway: for example, because they demonstrate extremely useful/profitable capabilities , and decision-makers are wrong about how well they can predict/control/incentivize the systems in question; and/or because externalities , dysfunctional competitive dynamics , and variations in caution/social responsibility lead to problematic degrees of willingness to knowingly deploy possibly or actually practically PS-misaligned systems (especially with increasingly powerful capabilities, and in increasingly uncontrolled and/or rapidly changing circumstances).

4. Some deployed APS systems will be exposed to inputs where they seek power in misaligned and high-impact ways (say, collectively causing >$1 trillion 2021-dollars of damage) | (1)–(3). 180 180 180 For reference, Hurricane Katrina , according to Wikipedia, cost ~160 billion (2021 dollars); Chernobyl , ~$770 billion (2021 dollars); and Cutler and Summers (2020) put the cost of Covid-19 in the U.S. at ~$16 trillion. More disaster costs here .

I’m going to say: 65%. In particular, I think that once we condition on 2 and 3, the probability of high-impact post-deployment practical alignment failures goes up a lot, since it means we’re likely building systems that would be practically PS-misaligned if deployed, but which are tempting—to some at least, especially in light of the incentives at stake in 2—to deploy regardless.

The 35% on this premise being false comes centrally from the fact that (a) I expect us to have seen a good number of warning shots before we reach really high-impact practical PS-alignment failures, so this premise requires that we haven’t responded to those adequately, (b) the time-horizons and capabilities of the relevant practically PS-misaligned systems might be limited in various ways, thereby reducing potential damage, and (c) practical PS-alignment failures on the scale of trillions of dollars (in combination) are major mistakes, which relevant actors will have strong incentives, other things equal, to avoid/prevent (from market pressure, regulation, self-interested and altruistic concern, and so forth).

However, there are a lot of relevant actors in the world, with widely varying degrees of caution and social responsibility, and I currently feel pessimistic about prospects for international coordination (cf. climate change) or adequately internalizing externalities (especially since the biggest costs of PS-misalignment failures are to the long-term future). Conditional on 1-3 above, I expect the less responsible actors to start using APS systems even at the risk of PS-misalignment failure; and I expect there to be pressure on others to do the same, or get left behind.

5. Some of this misaligned power-seeking will scale (in aggregate) to the point of permanently disempowering ~all of humanity | (1)–(4).

I’m going to say: 40%. There’s a very big difference between >$1 trillion dollars of damage (~6 Hurricane Katrinas), and the complete disempowerment of humanity; and especially in slower take-off scenarios, I don’t think it at all a foregone conclusion that misaligned power-seeking that causes the former will scale to the latter. But I also think that conditional on reaching a scenario with this level of damage from high-impact practical PS-alignment failures (as well as the other previous premises), things are looking dire. It’s possible that the world gets its act together at that point, but it seems far from certain.

6. This will constitute an existential catastrophe | (1)–(5).

I’m going to say: 95%. I haven’t thought about this one very much, but my current view is that the permanent and unintentional disempowerment of humans is very likely to be catastrophic for the potential value of human civilization’s future.

Multiplying these conditional probabilities together, then, we get: 65 % ⋅ 80 % ⋅ 40 % ⋅ 65 % ⋅ 40 % ⋅ 95 % = 65\%\cdot 80\%\cdot 40\%\cdot 65\%\cdot 40\%\cdot 95\%= ~5% probability of existential catastrophe from misaligned, power-seeking AI by 2070. 181 181 181 In sensitivity tests, where I try to put in “low-end” and “high-end” estimates for the premises above, this number varies between ~.1% and ~40% (sampling from distributions over probabilities narrows this range a bit, but it also fails to capture certain sorts of correlations). And my central estimate varies between ~1-10% depending on my mood, what considerations are salient to me at the time, and so forth. This instability is yet another reason not to put too much weight on these numbers. And one might think variation in the direction of higher risk especially worrying. And I’d probably bump this up a bit---maybe by a percentage point or two, though this is especially unprincipled (and small differences are in the noise anyway)---to account for power-seeking scenarios that don’t strictly fit all the premises above. 182 182 182 I’m not including scenarios that don’t center on misaligned power-seeking: for example, ones where AI systems empower human actors in the wrong ways, or in which forms of misalignment that don’t involve power-seeking lead to existential catastrophe.

Note that these (subjective, unstable) numbers are for 2070 in particular . Later dates, or conditioning on the development of sufficiently advanced systems, would bump them up.

My main point here, though, isn’t the specific numbers. Rather, it’s that as far as I can presently tell, there is a disturbingly substantive risk that we (or our children) live to see humanity as a whole permanently and involuntarily disempowered by AI systems we’ve lost control over. What we can and should do about this now is a further question. But the issue seems serious.

### Acknowledgments

Thanks to Asya Bergal, Alexander Berger, Paul Christiano, Ajeya Cotra, Tom Davidson, Daniel Dewey, Owain Evans, Ben Garfinkel, Katja Grace, Jacob Hilton, Evan Hubinger, Jared Kaplan, Holden Karnofsky, Sam McCandlish, Luke Muehlhauser, Richard Ngo, David Roodman, Rohin Shah, Carl Shulman, Nate Soares, Jacob Steinhardt, and Eliezer Yudkowsky for input on earlier stages of this project; thanks to Sara Fish for formatting and bibliography help; and thanks to Nick Beckstead for guidance and support throughout the investigation. The views expressed here are my own.

## 9 Appendix

Here are a few reformulations of the argument above, with probabilities (but without commentary). I don’t think this does all that much to ward off possible biases/framing effects (especially since I’ve gotten used to thinking in terms of the six premises above), but perhaps it’s a start. 183 183 183 I found the exercise of cross-checking at least somewhat helpful.

Shorter negative:

By 2070:

1. It will become possible and financially feasible to build APS AI systems.

65%

2. It will much more difficult to build APS AI systems that would be practically PS-aligned if deployed than to build APS systems that would be practically PS-misaligned if deployed, but which are at least superficially attractive to deploy anyway | 1. 35% 184 184 184 This is somewhat lower than in my six premises above, because it’s not conditioning on strong incentives to build APS systems; and I expect that absent such incentives, it’s harder to hit the bar for “superficially attractive to deploy.”

3. Deployed, practically PS-misaligned systems will disempower humans at a scale that constitutes existential catastrophe | 1-2. 20%

Implied probability of existential catastrophe from scenarios where all three premises are true: ~5%

Shorter positive:

Before 2070:

1. It won’t be possible and financially feasible to build APS AI systems.

35%

2. It won’t be much more difficult to build APS AI systems that would be practically PS-aligned if deployed, than to build APS systems that would be practically PS-misaligned if deployed, but which are at least superficially attractive to deploy anyway | not (1). 65%

3. It won’t be the case that deployed practically PS-misaligned systems disempower humans at a scale that constitutes existential catastrophe | not (1 or 2). 80%

Implied probability that we’ll avoid catastrophe à la shorter negative: ~95%

Same-length positive:

Before 2070:

1. It won’t be both possible and financially feasible to build APS systems.

35%

2. There won’t be strong incentives to build APS systems | not 1. 20%

3. It won’t be much harder to develop APS systems that would be practically PS-aligned if deployed, than to develop APS systems that would be practically PS-misaligned if deployed (even if relevant decision-makers don’t know this), but which are at least superficially attractive to deploy anyway | not (1 or 2). 60%