Is Power-Seeking AI an Existential Risk? — part 7/16

Cleaned-text code points [92040, 107553). Bibliography omitted here; source.md includes it.

Indeed, to avoid confusions in this vein, we might hope to jettison such concepts altogether. 86 86 86 See Wijk (2019) for one effort. And it’s possible to formulate arguments for something like instrumental convergence without them (see footnote for details). 87 87 87 That is, we can say things like: “This system behaves in a manner that promotes outcomes of a certain type in a wide range of complex circumstances. Circumstance C is one to which we should expect this behavior to generalize, and in circumstance C, power-seeking will promote outcomes of type X. Therefore, we should expect power-seeking in C” (thanks to Paul Christiano for discussion). Here, a lot of work is done by “circumstance C is one to which we should expect this behavior to generalize”—but I expect this to be plausible in the context of system optimized across a sufficient range of circumstances. That said, if the system was optimized, in those circumstances, to cause X without (observable) misaligned power-seeking , the case for expecting power-seeking in C seems weaker. So the question looms large: how much can selecting against observed power-seeking, in creating a system that causes X in many circumstances, eliminate behavior responsive to power-seeking’s usefulness for causing X (and many other types of outcome) across all physics-compatible inputs? And this question seems similar to the questions raised above (and discussed more in the next section) about the ease of selecting, in training, against forms of agentic planning that license misaligned power-seeking (and note, too, that one salient way in which training a system to cause X might lead to power-seeking behavior is if a suitable generalized ability to cause X tends to involve or require something like agentic-planning and strategic awareness). But the reasoning in play seems to me at least somewhat different.

### 4.3 The challenge of practical PS-alignment

Let’s grant that less-than-fully aligned APS systems will have at least some tendency towards misaligned, power-seeking behavior, by default. The challenge, then, is to prevent such behavior in practice—whether through alignment (including full alignment), or other means. How difficult will this be?

I’ll break down the challenge as follows:

1. Designers need to cause the APS system to be such that the objectives it pursues on some set of inputs X do not give rise to misaligned power-seeking.

2. They (and other decision-makers) need to restrict the inputs the system receives to X.

The larger the set of inputs X where 1 succeeds, the less reliance on 2 is required (and in the limit of full PS-alignment, 2 plays no role at all).

I’ll consider a variety of ways one might try to control a system’s objectives, capabilities, and circumstances to hit the right combination of 1 and 2. Some of these (or some mix) might well work. But all, I think, face problems.

#### 4.3.1 Controlling objectives

Much work on technical alignment focuses on controlling the objectives the AI systems we build pursue. I’ll start there.

At any given point in the progress of AI technology, we will have some particular level of control in this respect, grounded in currently available techniques. Thus, for example, some methods of AI development require coding objectives by hand. Others (more prominent at present) shape an AI’s objectives via different sorts of training signals—whether generated algorithmically, or using human feedback/demonstration. And we can imagine future methods that allows us to e.g. control an AI’s objectives by editing the weights of a neural network directly; or to cause a system to pursue (for its own sake) any objective that we can articulate using English-language sentences that will be accurately, common-sensically, and charitably construed.

The challenge of (1) is to use what methods in this respect are available to ensure PS-alignment on all inputs in X. But note that because available methods change, (1) is a moving target: new techniques and capabilities open up new options. I emphasize this because sometimes the challenge of AI alignment is framed as one of shaping an AI’s objectives in a particular way —for example, via hand-written code, or via some sort of reward signal, or via English-language sentences that will be interpreted in literalistic and uncharitable terms. And this can make it seem like the challenge is centrally one of, e.g., coding, measuring, or articulating explicitly everything we value, or getting AI systems to interpret instructions in common-sensical ways. These challenges may be relevant in some cases, but the core problem is not method-specific.

##### 4.3.1.1 Problems with proxies

That said, many ways of attempting to control an AI’s objectives share a common challenge: namely, that giving an AI system a “proxy objective”—that is, an objective that reflects properties correlated with, but separable from, intended behavior—can result in behavior that weakens or breaks that correlation, especially as the power of the AI’s optimization for the proxy increases.

For example: the behavior I want from an AI, and the behavior I would rate highly using some type of feedback, are well-correlated when I can monitor and understand the behavior in question. But if the AI is too sophisticated for me to understand everything that it’s doing, and/or if it can deceive me about its actions, the correlation weakens: the AI may be able to cause me to give high ratings to behavior I wouldn’t (in my current state) endorse if I understood it better---for example, by hiding information about that behavior, or by manipulating my preferences. 88 88 88 Though what sorts of misaligned power-seeking in particular these behaviors would give rise to is a further question.

This general problem---that optimizing for a proxy correlated with but not identical to some intended outcome can break the correlation in question---is familiar from human contexts. 89 89 89 Thus, paying railroad builders by the mile of track that they lay incentivizes them to lay unnecessary track (from Wikipedia, here ); trying to lower the cobra population by paying people to turn in dead cobras leads to people breeding cobras (from Wikipedia, here ); if teachers take “cause my students to get high scores on standardized tests” as their objective, they’re incentivized to “ teach to the test ”—an incentive that can work to the detriment of student education more broadly; and so on (see here for more examples). See Manheim and Garrabrant (2018) for an abstract categorization of dynamics of this kind. And we already see analogous problems in existing AI systems. Thus:

• If we train an AI system to complete a boat race by rewarding it for hitting green blocks along the way, it learns to drive the boat in circles hitting the same blocks over and over.

• If we train an AI system on human feedback on a grasping task , it learns to move its hand in a way that looks to the human like it’s grasping the object, even though it isn’t.

• If we reward an AI system for picking up apples in a simulated environment , but in a manner that depends on the locations of certain blocks, the agent learns to tamper with the blocks.

See here for a much longer list of examples in this vein.

These examples may seem easy to fix—and indeed, in the context of fairly weak systems, on a limited range of inputs, they generally are. But they illustrate the broader problem: systems optimizing for imperfect proxies often behave in unintended ways.

Indeed, this tendency is closely connected to a core property that makes advanced AI useful: namely, the ability to find novel solutions and strategies that humans wouldn’t think of. 90 90 90 Krakovna et al (2020) make this point well. When you don’t know how an AI will achieve its objective, and that objective doesn’t capture everything that you really want, then even for comparatively weak systems and simple tasks, it’s hard to anticipate how the system’s way of achieving the objective will break its correlation with what you really want. And as the AI’s capacity to generate solutions we can’t anticipate grows, the problem becomes more and more challenging.

We can see a variety of techniques for controlling an AI system’s objectives as mediated by some kind of “proxy” or other. Thus: hand-coded objectives, simple metrics (clicks, profits, likes), algorithmically generated training signals, human-generated data/feedback, and English-language sentences can all be seen as information structures that shape/serve as the basis for the AI’s optimization, but which can also fail to contain the information necessary to result in intended behavior across inputs. The challenge is to find a technique adequate in this respect.

Human feedback seems likely to play a key role here. 91 91 91 This paragraph draws heavily on discussion in Ngo (2020) . And it may, ultimately, be enough. But notably, we need ways of drawing on this feedback that don’t require unrealistic amounts of human supervision and human-generated data; 92 92 92 See e.g. Christiano et al (2017) . we need to ensure that such feedback captures our preferences about behavior that we can’t directly understand and/or whose consequences we haven’t yet seen; 93 93 93 See Christiano and Amodei (2018) for discussion. Iterative amplification and distillation ; debate ; and recursive reward modeling can all be seen as efforts in this vein. we need ways of eliminating incentives to manipulate or mislead the human feedback mechanisms in question; and we need such methods to scale competitively as frontier AI capabilities increase. 94 94 94 See 4.3.2.3 for more on scaling, and 5.3.1 for more on competition.

Would it help if our AI systems could understand fuzzy human concepts like “helpfulness,” “obedience,” “what humans would want,” and so forth? I expect it would, in various ways (though as I discuss below, this also opens up new opportunities for deception/manipulation). But note that the key issue isn’t getting our AI systems to understand what objectives we want them to pursue—indeed, such understanding is plausibly on the critical path to increasing their capability, regardless of their alignment. Rather, the key issue is causing them to pursue those objectives for their own sake (though if they understand those objectives, but don’t share them, we might also be able to incentivize them to pursue such objectives for instrumental reasons). 95 95 95 This is a point from Ord (2020) . For example, if we train some set of sophisticated agents to get bananas, in a complex environment that requires understanding and modeling humans, they may end up capable of understanding quite accurately (even more accurately than us) what we have in mind when we talk about “aligned behavior,” and of behaving accordingly (for example, when we give them bananas for doing so). But their intrinsic objectives could still be focused centrally on bananas (or something else), and our abilities to control those objectives directly might remain quite limited.

##### 4.3.1.2 Problems with search

Many techniques shape an AI’s objectives using proxies of one form another. But some—namely, those that involve searching over and selecting AI systems that perform well on some evaluation criteria, without controlling their objectives directly—have an additional problem: namely, even if those criteria fully capture the behavior we want, the resulting systems may not end up intrinsically motivated by the criteria in question. 96 96 96 My discussion in this section is centrally inspired by the discussion in Hubinger et al (2019) , though I don’t assume their specific set-up. Rather, they may end up with other objectives, pursuit of which correlated with good performance during the selection process, but which lead to unintended behavior on other inputs.

Some think of human evolution as an example. 97 97 97 See e.g. Hubinger et al (2019, p. 6) . Someone interested in creating agents who pass on their genes to the next generation could run a training process similar to evolution, which searches over different agents, and selects for ones who pass on their genes (for example, by allowing ones who don’t to die out). But this doesn’t mean the resulting agents will be intrinsically motivated to pass on their genes. Humans, for example, are motivated by objectives that were correlated with passing on genes (for example, avoiding bodily harm, having sex, securing social status, etc), but which they’ll pursue in a manner that breaks such correlations, given the opportunity (for example, by using birth control, or remaining childless to further their careers).

Rudimentary, evolved AI systems display analogous tendencies. Thus, when Ackley and Littman ran an evolutionary selection process in an environment with trees that allowed agents to hide from predators, the agents developed such a strong attraction to trees that (after reproductive age) they would starve to death in order to avoid leaving tree areas (what Ackley called ‘‘tree senility’’). 98 98 98 See Christian (2020) , quote here . I’m going off of Christian’s description, here, and haven’t actually investigated these experiments.

And we can imagine other cases, with less evolution-like techniques. Suppose, for example, that we use gradient descent to train an AI system to reach the exit of a maze. 99 99 99 This is an example from Hubinger (2020) . If the exit was marked by a green arrow on all the training data, the system could learn the objective “find the green arrow” rather than “find the exit to the maze.” If we then give it a maze where the green arrow isn’t by the exit, it will search out the green arrow instead.

Note that in both the evolution and the maze cases, we can imagine that the evaluation criteria (“pass on genes,” “exit maze”) fully capture and operationalize the intended behavior. Still, the designers lack the required degree of control over the objectives of the agents they create.

How often will problems like this arise? It’s an open empirical question. But some considerations make the possibility salient. Namely:

• Proxy goals correlated with the evaluation criteria may be simpler and therefore easier to learn, especially if the evaluation criteria are complex. In the context of evolution, for example, it seems much harder to evolve an agent whose mind represents a concept like ‘‘passing on my genes,’’ and then takes doing this as its explicit goal---humans, after all, didn’t even have the concept of ‘‘genes’’ until very recently---than to evolve an agent whose objectives reflect the relevance of things like bodily damage, sex, power, knowledge, etc to whether its genes get passed on (though starting with cognitively sophisticated agents might help in this respect). 100 100 100 This is a point I believe I heard Evan Hubinger make on a podcast (either this one , or this one ).

• Relatedly: if the ‘‘true’’ objective function provides slower feedback, agents that pursue faster-feedback proxies have advantages. For example: in the game Montezuma’s Revenge, it helps to give an agent a direct incentive analogous to ‘‘curiosity’’ (e.g., it receives reward for finding sensory data it can’t predict very well), because the game’s ‘‘true’’ objective (e.g., exiting a level by finding keys that require a large number of correct sequential steps to reach) is too difficult to train on. 101 101 101 See Burda and Edwards (2018) , and discussion in Christian (2020) .