Is Power-Seeking AI an Existential Risk? — part 4/16
Cleaned-text code points [46141, 61885). Bibliography omitted here; source.md includes it.
• Agentic planning seems like a very powerful and general way of interacting with an environment---especially a complex and novel environment that does not afford a lot of opportunities for trial and error---in a way that results in a particular set of favored outcomes. Many tasks humans care about (creating and selling profitable products; designing and running successful scientific experiments; achieving political and military goals; etc) have this structure, as do many of the sub-tasks involved in those tasks (e.g., efficiently gathering and synthesizing relevant information, communicating with stakeholders, managing resource-allocation, etc). 40 40 40 For more discussion, see Yudkowsky here , on the “ubiquity of consequentialism,” and on options for “subverting” it. Indeed, getting something done often seems like it requires, or at least benefits substantially, from (a) using a model of the world that reflects the relationship between action and outcome to (b) choose actions that lead to outcomes that score well according to some criteria. If our AI systems can’t do this, then the scope of what they can do seems, naively, like it will be severely restricted. • Strategic awareness seems closely related to a basic capacity to “understand what is going on,” interact with other agents (including human agents) in the real world, and recognize available routes to achieving your objectives—all of which seem very useful to performing tasks of the type just described. Indeed, to the extent that humans care about the strategic pursuit of e.g. business, military, and political objectives, and want to use AI systems in these domains, it seems like there will be incentives to create AI systems with the types of world models necessary for very sophisticated and complex types of strategic planning—including planning that involves recognizing and using available levers of real-world power. Note that the usefulness of agentic planning here is not limited to AI systems that are intuitively ‘‘acting directly in the world’’ (for example, via robot bodies, or without human oversight), as opposed to e.g. predicting the results of different actions, generating new ideas or designs---output that humans can then decide whether or not to act on. 41 41 41 See Karnofsky (2012) for discussion of “tool AI” that suggests such a contrast. Thus, for example, in sufficiently sophisticated cognitive systems, the task of predicting events or providing information might benefit from making and executing plans for how to process inputs, what data to gather and pay attention to, what lines of reasoning to pursue, and so forth. 42 42 42 See Bostrom (2014, p. 152-3, and p. 158) . Training new systems, and learning from previous experience, also plausibly involves decisions that benefit from this sort of planning. See Branwen (2016) for more. That said, I think we should be cautious in predicting what degree of agentic planning and/or strategic awareness will be necessary or uniquely useful for performing what types of cognitive tasks. • Humans often plan in pursuit of objectives, on the basis of big-picture models of what’s going on, and we are most familiar with human ways of performing tasks. But the constraints and selection processes that shape the development of our AI systems may be very different from the ones that shaped human evolution; and as a result, the future of AI may involve much less agentic planning and strategic awareness than anchoring on human intelligence might lead us to expect. Indeed, my impression is that the track record of claims of the form ‘‘Humans do X task using Y capabilities, so AI systems that do X will also need Y capabilities,’’ is quite poor. 43 43 43 Thanks to Owain Evans for discussion. • Naively construed, I can imagine arguments given about the usefulness/necessity of agentic planning and strategic awareness making the wrong predictions about current systems (or, e.g., about animal behaviors like squirrels burying nuts for the winter). Thus, for example, one might have expected writing complex code (see here , starting at 28:14), or playing StarCraft, to require a certain type of high-level planning; but whether GPT-3-like systems or AlphaStar in fact engage in such planning is unclear. Similarly, one might have expected automated driving to involve lots of big-picture understanding of “what’s going on”; but current self-driving cars don’t have much of this. In hindsight, perhaps this is obvious. But one wonders what future hindsight will reveal. Or put another way: we should be cautious about “obviously task X requires capability Y,” too. The compatibility of agentic planning and strategic awareness with modularity is also important. Suppose, for example, that you want to automate the long-term strategic planning performed by a CEO at a company. The best way of doing this may involve a suite of interacting, non-APS systems. Thus, as a toy example, one system might predict a plan’s impact on company long-term earnings; another system might generate plans that the first predicts will lead to high earnings; a third might predict whether a given plan would be deemed legal and ethical by a panel of human judges; a fourth might break a generated plan into sub-tasks to assign to further systems; and so forth. 44 44 44 Note, here, that generating a plan is not the same as agentic planning. None of these individual systems need be ‘‘trying’’ to optimize company long-term earnings in a way human judges would deem legal and ethical; but in combination, they create a system that exhibits agentic planning and strategic awareness with respect to this objective. 45 45 45 Ben Garfinkel suggested this sort of example in discussion. See also similar examples in Drexler (2019) . (Note that the agentic planning here is not “emergent” in the sense of “accidental” or “unanticipated.” Rather, we’re imagining a system specifically designed to perform an agentic planning function—albeit, using modular systems as components.) Modularity makes it possible for some sub-tasks, including various forms of oversight, to be performed by humans---an option that may well prove a useful check on the behavior of interacting non-APS systems, and a way of reducing the role of fully-automated APS-systems in the economy. 46 46 46 Though if the human’s performance of the sub-task does not meaningfully constrain or regulate the overall behavior of the system, then whether that task is performed by a human or an AI system may not make a difference. However, as AI systems increase in speed and sophistication, it also seems plausible that there will be efficiency pressures towards more and more complete automation, as slower and less sophisticated humans prove a greater and greater drag on what a system can do. 47 47 47 See Drexler (2019, Chapter 24) for discussion and disagreement. Overall, my main point here is that the space of tasks we want AI systems to perform plausibly involves a lot of ‘‘strategically-aware agentic planning shaped holes.’’ Leaving such holes unfilled would reduce risks of the type I discuss; but it would also substantially curtail the ways in which AI can be useful. 48 48 48 For related points, see Branwen (2018) : “Fundamentally, autonomous agent AIs are what we and the free market want ; everything else is a surrogate or irrelevant loss function. We don’t want low log-loss error on ImageNet, we want to refind a particular personal photo; we don’t want excellent advice on which stock to buy for a few microseconds, we want a money pump spitting cash at us; we don’t want a drone to tell us where Osama bin Laden was an hour ago (but not now), we want to have killed him on sight; we don’t want good advice from Google Maps about what route to drive to our destination, we want to be at our destination without doing any driving etc. Idiosyncratic situations, legal regulation, fears of tail risks from very bad situations, worries about correlated or systematic failures (like hacking a drone fleet), and so on may slow or stop the adoption of Agent AIs—but the pressure will always be there.” ### 3.2 Available techniques Even if some task doesn’t require agentic planning or strategic awareness, it may be that creating APS systems is the only route, or the most efficient route, to automating that task, given available techniques. For example, instead of automating tasks one by one, the best way to automate a wide range of tasks—especially ones where we lack lots of training data, which may be the majority of useful tasks—is to create AI systems that are able to learn new tasks with very little data. And we can imagine scenarios in which the best way to do that is by training agentic planners (for example, in multi-agent reinforcement learning environments, or via some process akin to evolutionary selection) with high-level, broad-scope pictures of ‘‘how the world works,’’ and then fine tuning them on specific tasks. 49 49 49 Richard Ngo suggested this point in conversation; see also his discussion of the “generalization-based approach” here . The training and fine-tuning used for GPT-3 may also be suggestive of patterns of this type. Indeed, with respect to strategic awareness in particular, various current techniques for providing AI systems information about the world—for example, training them on large text corpora from the internet—seem ill-suited to limiting their understanding of their strategic situation. ### 3.3 Byproducts of sophistication Even if you’re not explicitly aiming for or anticipating agentic planning and strategic awareness in an AI system, these properties could in principle arise in unexpected ways regardless, and/or prove difficult to prevent. Thus, for example: • Optimizing a system to perform some not-intuitively-agential task (for example, predicting strings of text) could, given sufficient cognitive sophistication, result in internal computation that makes and executes plans, in pursuit of objectives, on the basis of broad and informed models of the real world, even if the designers of the system do not expect this (they may even be unable to tell whether it has occurred). Indeed, the likelihood of this seems correlated with the strength of the ‘‘usefulness’’ considerations in 3.1: insofar as agentic planning and broad-scope world-modeling are very useful types of cognition, we might expect to see them cropping up a lot in sufficiently optimized systems, whether we want them to or not. 50 50 50 See e.g. Yudkowsky (undated) : “Another concern is that consequentialism may to some extent be a convergent or default outcome of optimizing anything hard enough. E.g., although natural selection is a pseudoconsequentialist process, it optimized for reproductive capacity so hard that it eventually spit out some powerful organisms that were explicit cognitive consequentialists (aka humans).” • A sophisticated system exposed to one source of information X might infer from this strategically important information Y, even if humans could not anticipate the possibility of such an inference. • Modular systems interacting in complex and unanticipated ways could give rise to various types of agential behavior, even if they weren’t designed to do so. Of these three reasons to expect APS systems—their usefulness, the pressures exerted by available techniques, and the possibility that they arise as byproducts of sophistication—I place the most weight on the first. Indeed, if it turns out that APS systems aren’t uniquely useful for some of the tasks we want AI systems to perform, relative to non-APS systems, this would seem to me a substantial source of comfort. ## 4 Alignment Let’s assume that it becomes possible and financially feasible to create APS systems by 2070, and that there are significant incentives to do so. This section examines why we might expect it to be difficult to create systems of this kind that don’t seek to gain and maintain power in unintended ways. ### 4.1 Definitions and clarifications Let’s start with a few definitions and clarifications. At a high-level, we want the impact of AI on the world to be good, just, fair, and so forth---or at least, not actively/catastrophically bad. Call this the challenge of ‘‘making AI go well.’’ 51 51 51 Here I’m drawing on the framework used by Christiano (2020) . This is a very broad and complex challenge, much of which lies well outside the scope of this report. A narrower challenge is: making sure AI systems behave as their designers intend. Of course, the intentions of designers might not be good. But if we can’t get our AI systems to behave as designers intend, this seems like a substantial barrier to good outcomes from AI more generally (though it is also, at least in many cases, a barrier to the usefulness of AI; see section 5.3.3 ). 52 52 52 Following the framework in Critch and Krueger (2020) , we can also draw additional distinctions, related to the number of human stake-holders and AI systems involved. I’m focused here on what that framework would call “single-single” and “multi-single” delegation: that is, the project of aligning AI behavior with the intentions of some human designer or set of designers. Critch argues (e.g., here ) that multi-multi delegation should be our focus. I disagree with this (indeed, if we solve single-single alignment, I personally would be feeling substantially better about the situation), but I won’t argue the point here. Granted, there are ambiguities about what sort of behavior counts as ‘‘intended’’ by designers (see footnote for discussion), but I’m going to leave the notion vague for now, and assume that behaviors like lying, stealing money, resisting shut-down by appropriate channels, harming humans, and so forth are generally ‘‘unintended.’’ 53 53 53 In particular, the relationship between “intended” and “foreseen” (for different levels of probability) is unclear. Thus, the designers of AlphaGo did not foresee the system’s every move, but AlphaGo’s high-quality play was still “intended” at some higher level (thanks to David Roodman for suggesting this example). And it’s unclear how to classify cases where a designer thinks it either somewhat or highly likely that a system will engage in a certain type of (unwanted-by-the-designers) power-seeking, but deploys anyway (thanks to Katja Grace for emphasizing this type of case in discussion; see also her post here ). I’ll count scenarios of this latter type as “unintended” (as distinct from “unforeseen”), but this isn’t particularly principled (indeed, I’m not sure if a principled carving is ultimately in the offing). A final problem is that the intentions of designers may simply not cover the range of inputs on which we wish to assess whether a system’s behavior is aligned (e.g., “did the designers intend for the robot to react this way to an asteroid impact?” may not have an answer: asteroid impacts weren’t within the scope of their design intentions). Here, perhaps, something like “unwanted” or “would be unwanted” is preferable. The reason I’m not using “unwanted” in general is that too many features of AI systems, including their capability problems, are “unwanted” in some sense (for example, I might want AlphaStar to play even better Starcraft, or to make me lots of money on the stock market, or to help me find love). And “catastrophic” seems both too broad (not all catastrophic behavior scales in the problematic way power-seeking does, nor is it clear why catastrophic behavior would imply power-seeking) and too narrow (not all relevant power-seeking need be “catastrophic” on its own, especially pre-deployment, and/or in “multipolar” scenarios in which no one AI actor is dominant).