Is Power-Seeking AI an Existential Risk? — part 8/16

Cleaned-text code points [107553, 123149). Bibliography omitted here; source.md includes it.

• To the extent that many objectives would instrumentally incentivize good behavior in training (for example, because many objectives, when coupled with strategic awareness, incentivize gaining power in the world, and doing well in training leads to deployment/greater power in the world), but few involve intrinsic motivation to engage in such behavior, we might think it more likely that selecting for good behavior leads to agents who behave well for instrumental reasons. 102 102 102 Thanks to Carl Shulman for discussion. See also Christiano (2018) : “One reason to be scared is that a wide variety of goals could lead to influence-seeking behavior, while the ‘intended’ goal of a system is a narrower target, so we might expect influence-seeking behavior to be more common in the broader landscape of ‘possible cognitive policies.’ ” That said, it could also be that the agents who perform best according to some criteria, especially once they’re sophisticated enough to understand what those criteria are, are the ones who are intrinsically motivated by those criteria. And even if such agents aren’t selected for automatically, various techniques might help detect and address problematic cases. Notably, for example, we might actively search for inputs that will reveal problematic objectives, 103 103 103 See Christiano’s discussion of “adversarial training” here . and we might learn how to read off a system’s objectives from its internal states. 104 104 104 This is closely related to work on “interpretability” and “transparency”—see, e.g., Olah (2020) and related work.

Overall, though, ensuring robust forms of practical PS-alignment seems harder if available techniques search over systems that meet some external evaluation criteria, with little direct control over their objectives. And much of contemporary machine learning fits this bill.

##### 4.3.1.3 Myopia

Some broad types of objectives seem to incentivize power-seeking on fewer physics-compatible inputs than others. Perhaps, then, we can aim at those, even if we lack more fine-grained control.

Short-term (or, ‘‘myopic’’) objectives seem especially interesting here. The most paradigmatically dangerous types of AI systems plan strategically in pursuit of long-term objectives, since longer time horizons leave more time to gain and use forms of power humans aren’t making readily available, they more easily justify strategic but temporarily costly action (for example, trying to appear adequately aligned, in order to get deployed) aimed at such power. 105 105 105 Non-myopic agents also seem more likely to want to “keep” power for a very long time, whereas myopic agents are more likely to “give up” a given type of power once they are “done with it.” Myopic agentic planners, by contrast, are on a much tighter schedule, and they have consequently weaker incentives to attempt forms of misaligned deception, resource-acquisition, etc that only pay off in the long-run (though even short spans of time can be enough to do a lot of harm, especially for extremely capable systems---and the timespans ‘‘short enough to be safe’’ can alter if what one can do in a given span of time changes). 106 106 106 Thanks to Rohin Shah, Paul Christiano, and Carl Shulman for discussion. And note that a given operationalization of “time” can itself be vulnerable to various forms of manipulation (an AI could, for example, find ways to stop and start its internal clock). Thanks to Carl Shulman for suggesting this possibility.

I think myopia might well help. But I see at least two problems with relying on it:

• Plausibly, there will be demand for non-myopic agents. Humans (and human institutions) have long-term objectives, and will likely want long-term tasks—running factories and companies, managing scientific experiments, pursuing political outcomes—automated. Of course, myopic and/or non-APS systems can perform sub-tasks (including sub-tasks that involve generating long-term plans), and humans can stay in the loop; but as discussed section 3.1 , there will plausibly be competitive pressures towards automating our pursuit of long-term objectives more and more fully.

• The “search” techniques discussed in the previous section may make ensuring myopia challenging. And various types of long-term training processes—for example, reinforcement learning on tasks that involve many sequential steps—seem likely to result in non-myopia by default (that said, myopia is a fairly coarse-grained property for an objective to possess, and may be easier to cause/check for than others).

We can also imagine other ways of attempting to shrink (even if not to zero) the set of physics-compatible inputs on which an APS system engages in PS-misaligned behavior. For example: we might aim for objectives that penalize ‘‘high-impact’’ action, 107 107 107 See, e.g., the discussion of “impact penalties” in Krakovna et al (2019) . or that prohibit lying in particular, or that give intrinsic weight to various legal and ethical constraints, or that benefit less from marginal resources. But these face the same challenges re: proxies and search discussed in the last two sections.

#### 4.3.2 Controlling capabilities

AI alignment research typically focuses on controlling a system’s objectives. But controlling its capabilities can play a role in practical PS-alignment, too.

In particular: the less capable a system, the more easily its behavior—including its tendencies to misaligned power-seeking—can be anticipated and corrected. Less capable systems will also have a harder time getting and keeping power, and a harder time making use of it, so they will have stronger incentives to cooperate with humans (rather than trying to e.g. deceive or overpower them), and to make do with the power and opportunities that humans provide them by default.

Preventing agentic planning and strategic awareness in the first place would be one example of “controlling capabilities” (see section 3 ); but there are other options, too.

##### 4.3.2.1 Specialization

In particular, we might focus on building APS systems whose competence is as narrow and specialized as possible (though they are still, by hypothesis, agentic planners with strategic awareness).

Discussion of AI risk often assumes the relevant systems are very “general”—e.g., individually capable of performing (or learning to perform) a very wide variety of tasks. But automating a wide variety of tasks doesn’t require creating a single AI system that can perform (or learn to perform) all of them. Rather, we can create different, specialized systems for each—and since such systems are less capable, the project of practical PS-alignment may be easier, and lower-stakes. An APS system skilled at a specific kind of scientific research, for example, but not at e.g. hacking, social persuasion, investing, and military strategy, seems much less dangerous—but it seems comparably useful for curing cancer. And even if a system’s competencies are broad, we can imagine that its strategically-aware agentic planning is only operative on some narrow set of inputs.

Indeed, specialized systems have many benefits. 108 108 108 Here, and in the bulleted list of “benefits of specialization” below, I’m drawing on a list of benefits in an unpublished document by Ben Garfinkel. See also Drexler (2019) for more discussion of the value of specialized systems. For example, they can be optimized more heavily for specific functions (to borrow an example from Ben Garfinkel, there is a reason that the flashlight, camera, speakers, etc on an iPhone are inferior to the best flashlights, cameras, etc). And we see incentives towards specialization and division of labor in human economies and organizations as well. What’s more, we will likely have much greater abilities to optimize AI systems for particular tasks than we do with humans.

That said, generality has benefits, too. In particular:

• Human workers with quite general skill-sets—CEOs, generals, Navy Seals, researchers with a broad knowledge of many domains, flexibly competent personal assistants—are prized in various contexts (even while specialization is prized in others). Automated systems need not be human-like in this respect (farmers, too, have quite general skill sets, but automated agriculture need not involve “farmer bots”), but it seems suggestive, at least, of economically-relevant environments in which general competence is useful.

• Specialized systems may be worse at responding flexibly to changing environments and task-requirements (e.g., it’s helpful not to have to buy new robots every time you redesign the factory or change the product being produced). 109 109 109 Thanks to Carl Shulman for suggesting this example.

• Multiple specialized systems can be less efficient to store and create (there is a reason you carry around an iPhone, rather than separate flashlights, cameras, microphones, etc);

• If a task requires multiple competencies, specialized systems can be harder to coordinate (e.g., it’s helpful to have a single personal assistant, rather than one for email, one for scheduling, one for travel planning, one for research, etc). And a suitably coordinated set of specialized systems can end up acting as a quite general and agentic system.

What’s more, just as available techniques may push the field towards agentic planning and strategic awareness (see section 3.2 ), so too might they push towards generality. 110 110 110 This paragraph draws centrally on the discussion in Ngo (2020) . GPT-3, for example, is trained to a fairly general level capability via predicting text, and then later fine-tuned on specific tasks like coding. Indeed, such an approach might be necessary for tasks where data is too hard to come by or learn from directly (consider, for example, tasks like ‘‘designing a good railway system’’ or ‘‘be an effective CEO’’); and more broadly, the most efficient route to wide-spread automation may be the creation of general-purpose agents that can learn ~arbitrary new tasks very efficiently (though those agents could also end up quite specialized later). 111 111 111 This is a point from Bostrom (2014) .

And note, too, that even specialized APS systems can be very dangerous. A system highly skilled at hacking into new computers and copying itself, for example, can spread far and wide; a system skilled in science can design a novel virus; a system with control over automated weapons can use them; a system skilled at social manipulation can turn an election; and so forth. Indeed, this is part of why I focused on “advanced capabilities” in 2.1.1, rather than something like “AGI” or “superintelligence.”

##### 4.3.2.2 Preventing problematic improvements

New capabilities can put a system in a position to gain and maintain power in ways it couldn’t before—and hence, make new incentives action-relevant (if Bob learns how to hack into bank accounts, for example, his likelihood of considering and executing plans that involve such hacking will change). Practical PS-alignment may therefore require controlling the extent to which the inputs a system receives result in improved capabilities.

This seems easier if the variables in the system that determine how it responds to inputs (for example, the weights in a neural network) stay fixed. But we may also want systems that mix task-performance and learning together, that “remember” previous events, and so forth; and predicting and controlling the capabilities such systems will develop could be difficult (especially if we don’t understand well how they work—see 4.4.1).

Note, though, that this is a narrower challenge than making sure a system’s PS-alignment is robust to any increase in capabilities—including, for example, increases that result from interventions other than exposure to physics-compatible inputs. Ultimately, we need to make sure that a system isn’t exposed to non-input interventions that cause it (or, a new version of it) to seek power in misaligned ways, too; and efforts by designers to scale up a system’s cognitive resources, training, and so forth will need to grapple with challenges in this vein. But as I discussed in 4.1, this sort of robustness is not a requirement for PS-alignment in my sense.

##### 4.3.2.3 Scaling

Strategies for practical PS-alignment that rely on limiting a system’s capabilities face a general problem: namely, that there are likely to be strong incentives to scale up the capabilities of frontier systems. 112 112 112 Though note that PS-alignment problems with more capable systems could complicate these dynamics—see 5.3.3 for more. PS-alignment strategies that can’t scale accordingly (and competitively) therefore risk obsolescence as state of the art capabilities advance.

A key question for any such strategy, then, is whether it can translate, given success at some level of capability, into a different strategy that scales better. For example, we might try to achieve practical PS-alignment with some fairly advanced systems (including, perhaps, quite specialized ones—or, indeed, non-APS ones), and then use them to create new and superior PS-alignment strategies (indeed, as AI development itself becomes increasingly automated, automating alignment research will plausibly be necessary regardless).

But note that plans of the form “create some practically PS-aligned systems, and ask them what the plan should be” might just not work. For example, the new systems might not have adequate plans either. One might therefore need to create even more capable systems, which also might not have adequate plans, and so forth, until one pushes up against (or perhaps, past) the limits of one’s capacity to ensure practical PS-alignment.

#### 4.3.3 Controlling circumstances

So far in section 4.3, I’ve been talking about controlling “internal” properties of an APS system: namely, its objectives and capabilities. But we can control external circumstances, too—and in particular, the type of options and incentives a system faces.

Controlling options means controlling what a circumstance makes it possible for a system to do, even if it tried. Thus, using a computer without internet access might prevent certain types of hacking; a factory robot may not be able to access to the outside world; and so forth.

Controlling incentives, by contrast, means controlling which options it makes sense to choose, given some set of objectives. Thus, perhaps an AI system could impersonate a human, or lie; but if it knows that it will be caught, and that being caught would be costly to its objectives, it might refrain. Or perhaps a system will receive more of a certain kind of reward for cooperating with humans, even though options for misaligned power-seeking are open.

Human society relies heavily on controlling the options and incentives of agents with imperfectly aligned objectives. Thus: suppose I seek money for myself, and Bob seeks money for Bob. This need not be a problem when I hire Bob as a contractor. Rather: I pay him for his work; I don’t give him access to the company bank account; and various social and legal factors reduce his incentives to try to steal from me, even if he could.

A variety of similar strategies will plausibly be available and important with APS systems, too. Note, though, that Bob’s capabilities matter a lot, here. If he was better at hacking, my efforts to avoid giving him the option of accessing the company bank account might (unbeknownst to me) fail. If he was better at avoiding detection, his incentives not to steal might change; and so forth.