Is Power-Seeking AI an Existential Risk? — part 6/16

Cleaned-text code points [77742, 92040). Bibliography omitted here; source.md includes it.

We see examples of rudimentary AI systems “discovering” the usefulness of e.g. resource acquisition already. For example : when OpenAI trained two teams of AIs to play hide and seek in a simulated environment that included blocks and ramps that the AIs could move around and fix in place, the AIs learned strategies that depended crucially on acquiring control of the blocks and ramps in question---despite the fact that they were not given any direct incentives to interact with those objects (the hiders were simply rewarded for avoiding being seen by the seekers; the seekers, for seeing the hiders). 72 72 72 Thus, the hiders learned to move and lock blocks to prevent the seekers from entering the room where the hiders were hiding; the seekers, in response, learned to move a ramp to give them access anyway; adjusting for this, the hiders learned to take control of that ramp before the seekers can get to it, and to lock it in the room as well. In another environment, the seekers learned to “surf” on boxes, and the hiders, to prevent this, learned to lock all boxes and ramps before the seekers can get to them. See videos in the blog post for details.

Why did this happen? Because in that environment, what the AIs do with the boxes/ramps can matter a lot to the reward signal, and in this sense, boxes and ramps are ‘‘resources,’’ which both types of AI have incentives to control---e.g., in this case, to grab, move, and lock. 73 73 73 Indeed, in a competitive environment like this, agents have an incentive not just to control resources that are directly useful to their own goals, but also resources that would be useful to their opponents. Thus, the ramps aren’t obviously useful to the hiders directly; but ramps can be used to bypass hider defenses, so hiders have an incentive to lock ramps in their room, or to remove from the environment entirely, all the same. This type of dynamic could be relevant to adversarial dynamics between humans and AIs, too. The AIs learned behavior responsive to this fact.

Of course, this is a very simple, simulated environment, and the level of agentic planning it makes sense to ascribe to these AIs isn’t clear. 74 74 74 And importantly, the trainers weren’t trying to disincentivize resource-seeking behavior; quite the contrary, the set-up seems (I haven’t investigated the history of the experiments) to have been designed to test whether “emergent tool-use” would occur. But the basic dynamic that gives rise to this type of behavior seems likely to apply in much more complex, real-world contexts, and to more sophisticated systems, as well. If, in fact, the structure of a real-world environment is such that control over things like money, material goods, compute power, infrastructure, energy, skilled labor, social influence, etc would be useful to an AI system’s pursuit of its objectives, then we should expect the planning performed by a sufficiently sophisticated, strategically aware AI agent to reflect this fact. And empirically, such resources are in fact useful for a very wide variety of objectives.

Concrete examples of power-seeking (where unintended) might include AI systems trying to: break out of a contained environment; hack; get access to financial resources, or additional computing resources; make backup copies of themselves; gain unauthorized capabilities, sources of information, or channels of influence; mislead/lie to humans about their goals; resist or manipulate attempts to monitor/understand their behavior, retrain them, or shut them off; create/train new AI systems themselves; coordinate illicitly with other AI systems; impersonate humans; cause humans to do things for them; increase human reliance on them; manipulate human discourse and politics; weaken various human institutions and response capacities; take control of physical infrastructure like factories or scientific laboratories; cause certain types of technology and infrastructure to be developed; or directly harm/overpower humans. (See 6.3.1 for more detailed discussion).

A few other clarifications:

• Not all misaligned power-seeking, even in APS systems, is particularly harmful, or intuitively worrying from an existential risk perspective. 75 75 75 Ben Garfinkel suggests the example of a robo-cop pinning down the wrong person, but remaining amenable to human instruction otherwise. And in general, it seems most dangerous if an APS system is using its advanced capabilities in pursuing power; e.g., if I’m a greater hacker, but a poor writer, and I’m trying to get power via my journalism, I’m less threatening. But I’m not going to try to narrow down any further.

• Instrumental convergence is not a conceptual claim, but rather an empirical claim that purports to apply to a wide variety of APS systems. In principle, for example, we can imagine APS systems that plan in pursuit of problematic objectives on some inputs, but which are nevertheless fully PS-aligned (or very close to it). 76 76 76 Thanks to Ben Garfinkel for suggesting examples in this vein. Consider, for example, an APS version of the electrical grid AI system above, which plans strategically in pursuit of directing electricity only to town A, but which just doesn’t consider plans that involve seeking power. The in-principle possibility of strategic, agentic misalignment without PS-misalignment is important, though, since it might be realized in practice. Perhaps, for example, the type of training we should expect by default will reinforce cognitive habits in APS systems that steer away from searching over/evaluating plans that involve misaligned power-seeking, even if other types of misaligned behavior persist. Training, after all, will presumably strongly select against observable power-seeking behavior, and perhaps power-seeking cognition, too. 77 77 77 Thanks to Ben Garfinkel, Rohin Shah, and Tom Davidson for emphasizing points in this vicinity. If it proves easy to create/train APS systems of this kind (even if they aren’t fully aligned), this would be great news for existential safety (see section 4.3.1 for more discussion).

Note, though, this requires that the planning performed by an APS system engaged in misaligned behavior be limited or “pruned” in a specific way. That is, by hypothesis, such a system is using a broad-scope world model, capable of accurately representing the causal upshot of different forms of power-seeking, to plan in pursuit of problematic objectives. To the extent that power-seeking would , in fact, promote those objectives, the APS system’s planning has to consistently ignore/skip over this fact, despite being in an epistemic position to recognize it. 78 78 78 We can also imagine cases where the system’s tendency to rule out certain plans is best interpreted as part of its objectives (e.g., such plans in fact aren’t good, on its values). Perhaps this type of planning is easy, in practice, to select for; but naively, it feels to me like a delicate dance. 79 79 79 Ben Garfinkel suggests the example of humans considering plans that involve murdering other humans—something that plausibly happens quite rarely, perhaps because “don’t murder” has been successfully reinforced. But for various reasons, murdering—in most of the relevant contexts—really doesn’t promote someone’s objectives (both because they intrinsically disvalue murder, and because of many other constraints and instrumental incentives). So it makes sense for them to adopt a policy of not considering it at all, or only very rarely. In contexts where murder does consistently promote a human’s objectives (perhaps sociopaths embedded in much more violent/lawless human contexts would be an example), I expect humans to consider plans that involve murdering much more often.

Relatedly, we can imagine systems whose objectives are problematic in some way, but which wouldn’t be promoted by unintended forms of power-seeking. 80 80 80 Distinctions between this and “it doesn’t consider bad plans” also seem blurry. Perhaps, for example, the electrical grid system plans strategically in order to direct electricity only to town A, but any plans that involve unintended power-seeking would score very low on its criteria for choice. The ease with which we can create systems of this type seems closely related to the ease with which we can create systems with non-problematic objectives more generally (again, see section 4.3.1 for discussion).

Let’s look at two other objections to instrumental convergence.

The first is that humans don’t always seem particularly ‘‘power-seeking.’’ 81 81 81 See e.g. Cegloski’s (2016) “Argument From My Roommate”; and also Pinker (2018, Chapter 19) : “There is no law of complex systems that says that intelligent agents must turn into ruthless conquistadors. Indeed, we know of one highly advanced form of intelligence that evolved without this defect. They’re called women” (p. 297). Thanks to Rohin Shah for discussion of the humans example. Humans care a lot about survival, and about certain resources (food, shelter, etc), but beyond that, we associate many forms of power-seeking with a certain kind of greed, ambition, or voraciousness, and with intuitively “resource-hungry” goals, like “maximize X across all of space and time.” Some strategically-aware human planners are like this, we might think, but not all of them: so strategically aware, agentic planning isn’t, itself, the problem.

I think there is an important point in this vicinity: namely, that power-seeking behavior, in practice , arises not just due to strategically-aware agentic planning, but due to the specific interaction between an agent’s capabilities, objectives, and circumstances. But I don’t think this undermines the posited instrumental connection between strategically-aware agentic planning and power-seeking in general. Humans may not seek various types of power in their current circumstances —in which, for example, their capabilities are roughly similar to those of their peers, they are subject to various social/legal incentives and physical/temporal constraints, and in which many forms of power-seeking would violate ethical constraints they treat as intrinsically important. But almost all humans will seek to gain and maintain various types of power in some circumstances, and especially to the extent they have the capabilities and opportunities to get, use, and maintain that power with comparatively little cost. Thus, for most humans, it makes little sense to devote themselves to starting a billion dollar company—the returns to such effort are too low. But most humans will walk across the street to pick up a billion dollar check.

Put more broadly: the power-seeking behavior humans display, when getting power is easy, seems to me quite compatible with the instrumental convergence thesis. And unchecked by ethics, constraints, and incentives (indeed, even when checked by these things) human power-seeking seems to me plenty dangerous, too. That said, the absence of various forms of overt power-seeking in humans may point to ways we could try to maintain control over less-than-fully PS-aligned APS systems (see 4.3 for more).

A second objection (in possible tension with the first) is: humans (or, some humans) may be power-seeking, but this is a product of a specific evolutionary history (namely, one in which power-seeking was often directly selected for), which AI systems will not share. 82 82 82 See e.g. Zador and LeCun (2019) (and follow-up debate here ), and Pinker (2018, Chapter 19) . One can also imagine non-evolutionary versions of this—e.g., ones that attribute human power-seeking tendencies to our culture, our economic system, and so forth. Indeed, Ceglowski (2016) can be read as suggesting something like this in the context of a particular demographic: after listing Bostrom’s convergent instrumental goals, he writes: “If you look at AI believers in Silicon Valley, this is the quasi-sociopathic checklist they themselves seem to be working from.” Some versions of this objection simply neglect to address the instrumental convergence argument above 83 83 83 That is, the argument isn’t “humans seek power, therefore AIs will too”; it’s “power is useful for pursuing objectives, so AIs pursuing problematic objectives will have incentives to seek power, by default.” (and note, regardless, that some proposed ways of training AI systems resemble evolution in various respects). 84 84 84 Large, multi-agent reinforcement learning environments might be one example. And the “league” used to train AlphaStar seems reminiscent of evolution in various ways. But we can see stronger versions as pointing at a possibility similar to the one I mentioned above: namely, perhaps it just isn’t that hard to train APS systems not to seek power in unintended ways, across a large enough range of inputs, if you’re actively trying to do so (evolution wasn’t). Again, I discuss possibilities and difficulties in this regard in the next section.

Ultimately, I’m sympathetic to the idea that we should expect, by default, to see incentives towards power-seeking reflected in the behavior of systems that engage in strategically aware agentic planning in pursuit of problematic objectives. However, this part of the overall argument is also one of my top candidates for ways that the abstractions employed might mislead.

In particular, it requires the agentic planning and strategic awareness at stake be robust enough to license predictions of the form: ‘‘if (a) a system would be planning in pursuit of problematic objectives in circumstance C, (b) power-seeking in C would promote its objectives, and (c) the models it uses in planning put it in a position to recognize this, then we should expect power-seeking in C by default.’’ I’ve tried to build something like the validity of such predictions into my definitions of agentic planning and strategic awareness; but perhaps for sufficiently weak/loose versions of those concepts, such predictions are not warranted; and it seems possible to conflate weaker vs. stronger concepts at different points in one’s reasoning, and/or to apply such concepts in contexts where they confuse rather than clarify. 85 85 85 Thanks to Rohin Shah for emphasizing possible objections in this vein, and for discussion.