Is Power-Seeking AI an Existential Risk? — part 5/16

Cleaned-text code points [61885, 77742). Bibliography omitted here; source.md includes it.

I’ll understand misaligned behavior as a particular type of unintended behavior: namely, unintended behavior that arises specifically in virtue of problems with an AI system’s objectives. Thus, for example, a designer might intend for an AI system to make money on the stock market, and the system might fail because it was mistaken about whether some stock would go up or down in price. But if it was trying to make money on the stock market, then its behavior was unintended but not misaligned: the problem was with the AI’s competence in pursuing its objectives, not with the objectives themselves. 54 54 54 Note that there is a sense in which the AI here is “trying to do something we don’t want it to do”—e.g., if we hold fixed its false beliefs about whether a given stock will go up or down, we’d prefer for it to be trying to lose money on the stock market (thanks to Paul Christiano for suggesting this point). Plausibly, calling this behavior “aligned,” then, requires reference to some holistic, common-sensical interpretation of the combination of objectives and capabilities we ultimately “have in mind.” But as ever, settling on a perfect conceptualization is tricky.

I don’t, at present, have a rigorous account of how to attribute unintended behavior to problems with objectives vs. other problems; and I doubt the distinction will always be deep or easily drawn (this doesn’t make it useless). I’ll lean on the intuitive notion for now; but if necessary, perhaps we could cease talk of “alignment” entirely, and simply focus on unintended behavior (perhaps of a suitably agentic kind), or perhaps simply catastrophic behavior, whatever its source.

A characteristic feature of misaligned behavior is that it is unintended but still competent. 55 55 55 See Mikulik (2019) for related discussion of “2-D robustness.” That is, it looks less like an AI system “breaking” or “failing in its efforts to do what designers want,” and more like an AI system trying, and perhaps succeeding, to do something designers don’t want it to do. In this sense, it’s less like a nuclear plant melting down, and more like a heat-seeking missile pursuing the wrong target; less like an employee giving a bad presentation, and more like an employee stealing money from the company.

I’ll say that a system is ‘‘fully aligned’’ if it does not engage in misaligned behavior in response to any inputs compatible with basic physical conditions of our universe (I’ll call these ‘‘physics-compatible’’ inputs). 56 56 56 Thus, it is physics-compatible for a randomly chosen bridge in Indiana to get hit, on a randomly chosen millisecond in May 2050, by a nuclear bomb, a 10 km-diameter asteroid, and a lightning bolt all at once; but not for the laws of physics to change, or for us all to be instantaneously transported to a galaxy far away. Obviously the scope here is very broad: but note that misaligned behavior is a different standard than “bad” or even “catastrophic” behavior. It will always be possible to set up physics-compatible inputs where a system makes a mistake, or gets deceived, or acts in a way that results in catastrophic outcomes. To be misaligned, though, this behavior needs to arise from problems with the system’s objectives in particular. Thus, for example, if Bob is a paper-clip maximizer, and he builds Fred, who is also a paper-clip maximizer, Fred will (on my definition) be fully-aligned with Bob as long as Fred keeps trying to maximize paperclips on all physics compatible-inputs (even though some of those inputs are such that trying to maximize paperclips actually minimizes them, kills Bob, etc). Thanks to Eliezer Yudkowsky, Rohin Shah, and Evan Hubinger for comments on the relevant scope here (which isn’t to say they endorse my choice of definition). By ‘‘inputs,’’ I mean information the system receives via the channels intended by its designers (an input to GPT-3, for example, would be a text prompt). 57 57 57 There are some possible edge-cases here where a system is getting information via channels other than those the designers intended, but I’ll leave these to the side. The point here is that we’re interested in how the system responds to inputs, not in how the world might change the system into something else. I am not including processes that intervene in some other way on the internal state of the system---for example, by directly changing the weights in a neural network (analogy: a soldier’s loyalty need not withstand arbitrary types of brain surgery). 58 58 58 Though here, too, lines may get hard to draw.

Sometimes, something like full alignment is framed by asking whether we would be OK with a system’s objectives being pursued with ~arbitrary amounts of capability. 59 59 59 See Yudkowsky on the “ omnipotence test for AI safety ”: “The Omni Test is that an advanced AI should be expected to remain aligned, or not lead to catastrophic outcomes, or fail safely, even if it suddenly knows all facts and can directly ordain any possible outcome as an immediate choice. The policy proposal is that, among agents meant to act in the rich real world, any predicted behavior where the agent might act destructively if given unlimited power (rather than e.g. pausing for a safe user query ) should be treated as a bug .” Talk of an “objective” such that the “optimal policy” on that objective leads to good outcomes is also reminiscent of something like the Omni Test. See e.g. Hubinger’s (2020) definition of “intent alignment”: “An agent is intent aligned if the optimal policy for its behavioral objective is aligned with humans.” This isn’t the concept I have in mind. 60 60 60 Indeed, I’m skeptical that it will generally be well-defined, even for agentic planners, what an arbitrarily capable system pursuing that agent’s objectives looks like (for example, I’m skeptical that there’s a single version of “omnipotent Joe”—and still less, “omnipotent MuZero,” “omnipotent Coca-Cola company,” “omnipotent version of my mom’s dog” and so forth). If we want to talk about alignment properties that are robust to improvements in capability, I think we should talk about what sorts of behavior will result from a particular process for improving that system’s capabilities (e.g,. a particular type of retraining, a particular way of scaling up its cognitive resources, and so forth). Thanks to Paul Christiano, Ben Garfinkel, Richard Ngo, and Rohin Shah for discussion. Rather, what matters is how the actual system, with its actual capabilities, responds to physics-compatible inputs. If, on some such inputs, it seeks to improve its own capabilities in misaligned ways; or if some inputs improve its capabilities in a way that results in misaligned behavior; then the system is less-than-fully aligned. But if there are no physics-compatible inputs that the actual system responds to in misaligned ways, then I see preventing changes in capability that might alter this fact as a separate problem (though it may be a problem that designers intentionally improving the system’s capabilities should be especially concerned about). 61 61 61 But so, too, should designers be concerned about altering the system’s objectives as they improve it. Note that I’m also setting aside the problem (as it relates to a given system A) of how to make sure that, to the extent that system A builds a new system B, system B is fully-aligned, too (for example, if system B isn’t fully aligned, but not because of any problems with system A’s objectives, that’s not, on my view, a problem with system’s A’s alignment—though it might be a problem more generally).

Nor does full alignment require that the system ‘‘share the designer’s values’’ (and still less, the designer’s ‘‘utility function,’’ to the extent it makes sense to attribute one to them) in any particularly complete sense. 62 62 62 See e.g. Bostrom’s 2015 TED talk : “The point here is that we should not be confident in our ability to keep a superintelligent genie locked up in its bottle forever. Sooner or later, it will out. I believe that the answer here is to figure out how to create superintelligent A.I. such that even if—when—it escapes, it is still safe because it is fundamentally on our side because it shares our values. I see no way around this difficult problem… The initial conditions for the intelligence explosion might need to be set up in just the right way if we are to have a controlled detonation.” For example, we can imagine AI systems that just undergo some kind of controlled shut-down, or query relevant humans for guidance, if they receive an input designers didn’t intend for them to operate on. Indeed, extreme motivational harmony seems like a strange condition to impose on e.g. a house cleaning robot, or a financial manager—neither of which, presumably, need know the designer’s or the user’s preferences about e.g. politics, or romantic partners.

Another property in the vicinity of “full alignment” and “sharing values,” but distinct from both, is Christiano’s (2018) notion of “intent alignment,” on which an AI system A is intent aligned with a human H iff “A is trying to do what H wants it to do,” where A’s relationship to H’s desires is, importantly, de dicto . That is, A doesn’t just want the same things as H (e.g., ‘‘maximize apples’’). 63 63 63 My sense is that this difference between Christiano’s notion of “intent alignment,” and a broader notion of “sharing values,” is sometimes overlooked. Rather, A has to have something like a hypothesis about what H wants (e.g., it asks itself: ‘‘would H want me to maximize apples, or oranges?’’), and to act on that hypothesis. 64 64 64 Russell’s (2019) proposal—e.g., to make AI systems that pursue our objectives, but are uncertain what those objectives are (and see our behavior as evidence)—has a somewhat similar flavor. Intent alignment seems a promising route to full alignment; but I won’t focus on it exclusively.

I’ll say that a system is “practically aligned” if it does not engage in misaligned behavior on any of the inputs it will in fact receive. 65 65 65 In the sense, practical alignment is a property that holds relative to a set of inputs. Full alignment implies a highly reliable form of practical alignment—e.g., one that does not depend at all on predicting or controlling the inputs a system receives. From a safety perspective, this seems a very desirable property, especially given the unique challenges and stakes that APS systems present (see section 4.4 for discussion). Indeed, if you are building AI systems that would e.g. harm or seize power over you, due to problems with their objectives, in some physics-compatible circumstances (see next section), this seems, at the least, a red flag about your project. 66 66 66 See Yudkowsky’s discussion of “AI safety mindset” here . Yudkowsky sometimes frames this as: you should build systems that never, in practice, search for ways to kill you, rather than systems where the search comes up empty. Ultimately, though, it’s practical alignment that we care about.

### 4.2 Power-seeking

Not all misaligned AI behavior seems relevant to existential risk. Consider, for example, an AI system in charge of an electrical grid, whose designers intend it to send electricity to both town A and town B, but whose objectives have problems that cause it, during particular sorts of storms, to only send electricity to town A. This is misaligned behavior, and it may be quite harmful, but it poses no threat to the entire future.

Rather, as I discussed in section 1.2.4 , the type of misaligned AI behavior that I think creates the most existential risk involves misaligned power-seeking in particular: that is, active efforts by an AI system to gain and maintain power in ways that designers didn’t intend, arising from problems with that system’s objectives. In the electrical grid case, the AI system hasn’t been described as trying to gain power (for example, by trying to hack into more computing resources to better calculate how to get electricity to town A) or to maintain the power it already has (for example, by resisting human efforts to remove its influence over the grid). And in this sense, I think, it’s much less dangerous.

I’ll say that a system is “fully PS-aligned” if it doesn’t engage in misaligned power-seeking in particular in response to any physics-compatible inputs. And I’ll say that a system is “practically PS-aligned” if it doesn’t engage in misaligned power-seeking on any of the inputs it will in fact receive.

A key hypothesis, some variant of which underlies much of the discourse about existential risk from AI, is that there is a close connection, in sufficiently advanced agents, between misaligned behavior in general, and misaligned power-seeking in particular. 67 67 67 See e.g. Bostrom (2014, chapter 7) ; Russell (2019, Chapter 5) , on “you can’t fetch the coffee if you’re dead”; and Yudkowsky (undated) . I’ll formulate this hypothesis as:

Instrumental Convergence : If an APS AI system is less-than-fully aligned, and some of its misaligned behavior involves strategically-aware agentic planning in pursuit of problematic objectives, then in general and by default, we should expect it to be less-than-fully PS-aligned, too.

Why think this? The basic reason is that power is extremely useful to accomplishing objectives---indeed, it is so almost by definition. 68 68 68 For a formal version of a similar argument, see Turner et al (2019) . So to the extent that an agent is engaging in unintended behavior in pursuit of problematic objectives, it will generally have incentives, other things equal, to gain and maintain forms of power in the process---incentives that strategically aware agentic planning puts it in a position to recognize and respond to. 69 69 69 Of course, systems with non-problematic objectives will have incentives to seek forms of power, too; but the objectives themselves can encode what sorts of power-seeking are OK vs. not OK.

One way of thinking about power of this kind is in terms of the number of ‘‘options’’ an agent has available. 70 70 70 Thanks to Ben Garfinkel for helpful discussion. Thus, if a policy seeks to promote some outcomes over others, then other things equal, a larger number of options makes it more likely that a more preferred outcome is accessible. Indeed, talking about “options-seeking,” instead of “power-seeking,” might have less misleading connotations.

What sorts of power might a system seek? Bostrom (2014) (following Omohundro (2008) ) identifies a number of “convergent instrumental goals,” each of which promotes an agent’s power to achieve its objectives. These include:

• self-preservation (since an agent’s ongoing existence tends to promote the realization of those objectives);

• “goal-content integrity,” e.g. preventing changes to its objectives (since agent’s pursuit of those objectives in particular tends to promote them);

• improving its cognitive capability (since such capability tends to increase an agent’s success in pursuing its objectives);

• technological development (since control over more powerful technology tends to be useful);

• resource-acquisition (since more resources tend to be useful, too). 71 71 71 Note that we can extend the list to include an agent’s instrumental incentives to promote the existence, goal-content integrity, cognitive capability, resources, etc of agents whose objectives are sufficiently similar to its own—for example, future agents in some training process, copies of itself, slightly modified versions of itself, and so forth. And we can imagine cases in which an agent is intrinsically motivated to gain/maintain various types of power—for example, because this was correlated with good performance during training (see Ngo (2020) for discussion). I’ll focus, though, on the instrumental usefulness of power.