Is Power-Seeking AI an Existential Risk? — part 3/16

Cleaned-text code points [30899, 46141). Bibliography omitted here; source.md includes it.

I take as my paradigm a certain type of human cognition—the type, for example, involved in e.g., planning and then taking a trip from New York to San Francisco; reasoning about the safest way to cut down a tree, then doing it; designing a component of a particle collider; and so on. When I talk about agentic planning, I’m talking, centrally, about that . (Though exactly how much of human cognition is like this is a further question---and similarity comes on a spectrum.) 26 26 26 Obviously, not everything humans do seems like agentic planning. Pulling your hand away from a hot stove, for example, seems more like a reflex—a basic “if-then” implemented by your nervous system—than an executed plan. And much of what we do in life can seem more like the stove than planning a trip to New York. Indeed, we often think of non-humans animals as acting on “instinct,” or on the basis of comparatively simple heuristics, instead of on the basis of explicit plans, models, and objectives. Humans, we might think, are more like this than sometimes supposed. But lines here can get blurry fast. Ultimately, all complex cognition consists of a large number of much simpler steps or “if-thens.” Whether a given behavior is built out of instincts, heuristics, if-thens, etc does not settle whether it is relevantly similar to travel-planning (indeed, contrasts between “heuristics” and more systematic forms of cognition often seem to me under-specified). And similarity comes along a multi-dimensional spectrum—reflecting, for example, the sophistication and flexibility of the models and plans involved.

We tend to think of this kind of cognition as (a) using a model of the world that represents causal relationships between actions/outputs/policies and outcomes to (b) select actions/outputs/policies that lead to outcomes that rate highly according to various (possibly quite complicated) criteria (e.g., objectives). 27 27 27 See Arbital (undated, and no author listed, but I believe it’s Yudkowsky) on “consequentialist cognition” for more. Thanks to Paul Christiano for discussion. And we predict and explain human action on this basis, using certain sorts of common-sense distinctions—for example, between the role that someone’s objectives play in explaining a given action, vs. their world-models and capabilities. There are also various algorithms that implement explicit procedures in the vein of (a) and (b)—algorithms, for example, that search over and evaluate possible sequences of actions, or that backwards-chain from a desired end-state.

People sometimes dispute whether humans “actually” do something like (a) and (b) (let alone, implement something akin to a more formal algorithm). For present purposes, though, what matters is that they do something close enough to justify predicting their behavior (at least sometimes) with (a) and (b) in mind.

This is the standard I’ll apply to the agentic planners I discuss in what follows. That is, I am not imposing any specific constraints on the cognitive algorithms or forms of representation required for “agentic planning”—but they need to be close enough , at some level of abstraction, to (a) and (b) above to justify similar predictions. Indeed, I will speak as though agentic planners are actually using models of the world to select and execute plans that promote their objectives (as opposed to, e.g., “behaving just like” they are doing this); by hypothesis, to the extent there is a difference here, it’s not a predictively relevant one.

I am not assuming, though:

• that the objectives need to involve an explicitly represented “ objective function ,’’ especially in the sense at stake in formal optimization problems; 28 28 28 Here I’m responding in part to the definition offered by Hubinger et al (2019) , and to subsequent debate.

• that an agentic planner need be helpfully understood as “maximizing expected utility”;

• that it engages in agentic planning in all circumstances;

• that it needs to be possible to easily tell whether a given system is engaging in agentic planning or not;

• that the system’s objectives are simple, “long-term” (see section 4.3.1.3 ), easily subsumable within a set of “intrinsic values,” or constant across circumstances;

• that agentic planners cannot be constituted by many interacting, non-agentic-planning systems; 29 29 29 Here, and in a number of subsequent bullet points, I’m responding in part to the contrasts discussed by Drexler (2019) .

• that the system is capable of self-modification or online learning (see section 4.3.2.2 );

• that the system has any particular set of opportunities for action in the world (for example, systems that can only answer question can count as agents in this sense);

• that the system’s plans are oriented towards the external environment, as opposed to e.g. plans for how to distribute its internal cognitive resources in performing a task;

• that the agentic planning in question is ‘‘ongoing’’ and ‘‘open-ended,’’ as opposed to ‘‘episodic’’ (see footnote for details). 30 30 30 For example, something can be an agent planner once it is given a task ; but that doesn’t mean that when it hasn’t been given a task, it has other, ongoing objectives. And a system can plan agentically in pursuit of taking a single action, as opposed to executing a series of actions.

MuZero —a system which learns a model of a game (Chess, Go, Shogi, Atari) in order to plan future actions—qualifies as an agentic planner in this sense, as do AlphaZero , AlphaGo , and (at least on my current understanding) various self-driving cars. Thermostats, bottlecaps, forest fires, balls rolling down hills, and robots twitching randomly don’t qualify---they’re not doing something close enough to planning, using a model of the world, in pursuit of the outcomes they cause. 31 31 31 These are examples references in e.g. Filan (2018) , Shah (2018) , and Flint (2020) . Fires are an example Katja Grace often uses of a non-agentic system that meets many proposed definitions of agency (for example, tending to bring about a certain type of outcome with a fair amount of robustness).

In various other systems, it’s less clear. For example: AlphaStar —an AI system that plays Starcraft—executes complex, flexible strategies over long time horizons, but the extent to which it’s doing something close enough to explicitly representing and acting in light of the long-term consequences of its actions seems (to me) like an open question. Similarly, we can imagine a system like GPT-3 ---that is, a large machine learning model trained to predict human-generated strings of text---that engages in something like agentic planning in generating its output: but the outputs we’ve observed need not tell us directly. 32 32 32 The extent of the world-modeling and planning involved in many animal behaviors is similarly unclear.

I’ll also note that a system that merely generates plans (for example, a GPT-3-like system responding to the prompt “The following is a statement of an excellent business plan:”), but which doesn’t do so in order to select outputs/actions/policies that promote its objectives , doesn’t qualify.

#### 2.1.3 Strategic awareness

I’ll say that an agentic planner has ‘‘strategic awareness’’ if the models it uses in making plans are broad, informed, and sophisticated enough to represent with reasonable accuracy the causal upshot of gaining and maintaining different forms of power over humans and the real-world environment (here, again, I am using the ‘‘actually this, or close enough to make no predictive difference’’ standard above). 33 33 33 See Yudkowsky here for the closely related concept of “big-picture strategic awareness.”

Clearly, strategic awareness comes in degrees. Broadly and loosely, though, we can think of a strategically aware, planning agent as possessing models of the world that would allow it to answer questions like “what would happen if I had access to more computing power” and “what would happen if I tried to stop humans from turning me off” about as well as humans can (and using those same models in generating plans).

AlphaGo does not meet this condition (its models are limited to the Go board). Nor, I think, do self-driving cars. 34 34 34 My understanding is that the planning performed by self-driving cars uses some combination of long-term, google-maps-like planning, which relies on a fairly impoverished model of the world; and short term predictions of behavior on the road. Neither of these seems broad and sophisticated enough to answer questions of the type above. However, I haven’t investigated this. And even if a GPT-3-like system had suitably sophisticated models, it would need to use those models in generating plans in order to qualify.

Let’s call a system with advanced capabilities that performs agentic planning with strategic awareness an APS (Advanced, Planning, Strategically aware) system.

### 2.2 Likelihood by 2070

How likely is it that it becomes possible and financially feasible to develop APS systems before 2070? Obviously, forecasts like this are difficult (and fuzzily defined), and I won’t spend much time on them here. My own views on this topic emerge in part from a set of investigations that Open Philanthropy has been conducting (see, for example, Cotra (2020) , Roodman (2020) , and Davidson (2021) , which I encourage interested readers to investigate.

I’ll add, though, a few quick data points:

• Cotra’s model , which anchors on the human brain and extrapolates from scaling trends in contemporary machine learning, puts >65% on the development of “transformative AI” systems by 2070 (definition here ).

• Metaculus, a public forecasting platform, puts a median of 54% on “ human-machine intelligent parity by 2040 ,” and a median of 2038 for the “ date the first AGI is publicly known ” (as of mid-April 2021, see links for definitions).

• Depending on how you ask them , experts in 2017 assign a median probability of >30% or >50% to “unaided machines can accomplish every task better and more cheaply than human workers” by 2066, and a 3% or 10% chance to the “full automation of labor” by 2066 (though their views in this respect are notably inconsistent, and I don’t think the specific numbers should be given much weight).

Personally, I’m at something like 65% on “developing APS systems will be possible/financially feasible before 2070.” I can imagine going somewhat lower, but less than e.g. 10% seems to me weirdly confident (and I don’t think the difficulty of forecasts like these licenses assuming that probability is very low, or treating it that way implicitly).

## 3 Incentives

Let’s grant, then, that it becomes possible and financially feasible to develop APS systems. Should we expect relevant actors to have strong incentives to do so, especially on a widespread scale? I’ll assume that there are strong incentives to automate advanced capabilities, in general. But building strategically-aware agentic planners may not be the only way to do this. Indeed, there are various reasons we might expect an automated economy to focus on systems without such properties, namely: 35 35 35 Here I’m drawing heavily on a list of considerations in an unpublished document written by Ben Garfinkel.

• Many tasks---for example, translating languages, classifying proteins, predicting human responses, and so forth---don’t seem to require agentic planning and strategic awareness, at least at current levels of performance. 36 36 36 Though what sorts of methods of performing those tasks emerge in the limits of optimization is a further question; see 3.3 below. Perhaps all or most of the tasks involved in automating advanced capabilities will be this way.

• In many contexts (for example, factory workers), there are benefits to specialization; and highly specialized systems may have less need for agentic planning and strategic awareness (though there’s still a question of the planning and strategic awareness that specialized systems in combination might exhibit). See section 4.3.2.1 for more on this.

• Current AI systems are, I think, some combination of non-agentic-planning and strategically unaware. Some of this is clearly a function of what we are currently able to build, but it may also be a clue as to what type of systems will be most economically important in future.

• To the extent that agentic planning and strategic awareness create risks of the type I discuss below, this might incentivize focus on other types of systems. 37 37 37 Though not all relevant actors will treat these risks with equal caution—see 5.3.2.

• Agentic planning and strategic awareness may constitute or correlate with properties that ground moral concern for the AI system itself (though not all actors will treat concerns about the moral status of AI systems with equal weight; and considerations of this type could be ignored on a widespread scale).

Indeed, for reasons of this type, I expect that to the extent we develop APS systems, we’ll do so in a context of a wide variety of non-APS systems, which may themselves have large impacts on the world, and which may help manage risk from APS systems. 38 38 38 Though note that an increasingly automated economy might also exacerbate some types of risks, since misaligned, power-seeking systems might be better-positioned to make use of automated rather than human-reliant infrastructure. I discuss this briefly in section 6.3.1 . And it seems possible that APS systems just won’t be a very important part of the picture at all.

Still, though, there are a number of reasons to expect AI progress to push in the direction of systems with agentic planning and strategic awareness. I’ll focus on three main types:

1. Agentic planning and strategic awareness both seem very useful . That is, many of the tasks we want AI systems to perform seem to require or substantially benefit from these abilities.

2. Given available techniques, it may be that the most efficient way to develop AI systems that perform various valuable tasks involves developing strategically aware, agentic planners, even if other options are in principle available.

3. It might be difficult to prevent agentic planning and strategic awareness from developing in suitably sophisticated and efficient systems.

One note: when I talk in what follows about “incentives to create APS systems,” I mean this in a sense that covers 2 and 3 as well as 1. Thus, on this usage, if people will pay lots of money for tables (including flammable tables), and the only (or the cheapest/most efficient) way to make tables is out of flammable wood, then I’ll say that there are “incentives” to make flammable tables, even if people would pay just as much, or more, for fire-resistant tables.

Let’s look at 1-3 in turn.

### 3.1 Usefulness

I think that the strongest reason to expect APS systems is that they seem quite useful . 39 39 39 See Drexler (2019), Chapter 12 , for discussion and disagreement. Many of Drexler’s arguments, though, focus on the value of self-improving and/or “unbounded” agents, which aren’t aren’t my focus in this section (though see 4.3.2.2 for more on self-improvement, and 4.3.1.3 on something akin to boundedness). In particular: