Is Power-Seeking AI an Existential Risk? — part 9/16

Cleaned-text code points [123149, 138317). Bibliography omitted here; source.md includes it.

PS-alignment strategies that rely on controlling options and incentives therefore require ways of exerting this control (e.g., mechanisms of security, monitoring, enforcement, etc) that scale with the capabilities of frontier APS systems. Note, though, that we need not rely solely on human abilities in this respect. For example, we might be able to use various non-APS systems and/or practically-aligned APS systems to help. 113 113 113 One could even imagine trying to use practically PS- mis aligned systems, though this seems dicey.

One other note: ensuring practical PS-alignment in a deployed system seems easier the more similar its deployment circumstances to the ones on which humans have observed and verified PS-aligned behavior (for example, during training, or pre-deployment testing). Indeed, ideally, one would want the deployment inputs to come from the same distribution as the training inputs. But in practice, and especially in strategically-aware systems, ensuring a close-to-identical distribution seems very difficult (if not impossible). This is partly because the world changes (and indeed, the actions of the APS system can themselves change it). 114 114 114 See Krueger et al (2020) and Christian (2020) for some discussion of this possibility. But also, to the extent that the distinction between training and deployment reflects some real difference in the agent’s level of influence on the world, this difference is itself a change in distribution—one that a sufficiently sophisticated agent might recognize.

### 4.4 Unusual difficulties

From our current vantage point, ensuring PS-aligned behavior from APS systems across a wide range of inputs seems, to me, like it could well be difficult. But so, too, does building any kind of APS system appear difficult. Is there reason to think that by the time we figure out how to do the latter, we won’t have figured out the former as well?

It’s generally easier to create technology that fulfills some function F, than to create technology that does F and meets a given standard of safety and reliability, for the simple reason that meeting the relevant standard is an additional property, requiring additional effort. 115 115 115 See Bostrom (2015) : “Making superintelligent A.I. is a really hard challenge. Making superintelligent A.I. that is safe involves some additional challenge on top of that. The risk is that if somebody figures out how to crack the first challenge without also having cracked the additional challenge of ensuring perfect safety.” Thus, it’s easier to build a plane that can fly at all than one that can fly safely and reliably in many conditions; easier to build an email client than a secure email client; easier to figure out how to cause a nuclear chain reaction than how to build a safe nuclear reactor; and so forth.

Of course, we often reach adequate safety standards in the end. But at the very least, we expect some safety problems along the way (plane crashes, compromised email accounts, etc). We might expect something similar with ensuring PS-aligned behavior from powerful AI agents.

But ensuring such behavior also poses a number of challenges that (most) other technologies don’t. Here are a few salient to me.

#### 4.4.1 Barriers to understanding

Ensuring safety and reliability requires understanding a system well enough to predict its behavior. But plausibly, this is uniquely challenging in the context of a strategically aware agentic planner whose cognitive capabilities significantly exceed those of humans in a given domain. That is, the thinking and strategic decision-making of such an agent will likely reach a quite alien and opaque level of sophistication, which humans may be poorly positioned to anticipate and understand at the level required for ensuring PS-alignment. For example, it may consider many options humans never would; it may understand physical and social dynamics that humans do not; and so forth. 116 116 116 See Yudkowksy on “ strong cognitive uncontainability .”

This issue seems especially salient in the current, machine-learning dominated AI paradigm, in which our ability to create an AI system that can perform some task (e.g., predicting text) often far exceeds our ability to understand how the system does what it does. We set various key high-level variables (the system’s architecture, the number of parameters, the training process, the evaluation criteria), but the system that results is still, in many (though not all) respects, a black box. We must rely on further experiments to try to get some handle on what it knows, what it can do, and how it is liable to behave. 117 117 117 See Flint (2020) on “search vs. design” for related discussion.

If this lack of understanding of the systems we’re building persists (for example, if we end up creating APS systems by training very large machine learning models on complex tasks, using techniques fairly similar to those used today), this could be an important and safety-relevant difference between AI systems and other types of technology. That is, we achieve high degrees of reliability and safety with technology like bridges, planes, rockets, and so forth in part via an understanding of the physical principles that govern the behavior of those systems; and we design them, part by part, in a manner that reflects and responds to those principles. This allows us to understand and predict their behavior in a wide range of circumstances. Searching over opaque/poorly-understood AI systems allows no such advantage.

Of course, our understanding of how ML systems work will likely improve over time---indeed, active research in this area (sometimes called ‘‘interpretability’’) is ongoing. 118 118 118 See e.g. Olah (2020) and Goh et al (2021) . But interpretability is no bottleneck to training bigger models on more complex tasks—or, plausibly, to the commercial viability of such models. And even as some researchers work on it, much of the field’s effort focuses on pushing forward with developing whatever capabilities we can, interpretable or no.

That said, understanding comes in many varieties and degrees; and it’s an empirical question what mix of experiment/search vs. first-principles understanding/design has actually been involved in ensuring the safety of different technologies (for example, in biology, or before advanced scientific understanding). 119 119 119 And defining these distinctions may prove challenging. I expect that investigation of case studies in this respect would prove revealing.

Ultimately, though, “mechanistic understanding” isn’t the fundamental issue. Even if we had such understanding on some level, the basic difficulty—that is, understanding and predicting the behavior of an agent much more cognitively sophisticated than humans—might well persist.

#### 4.4.2 Adversarial dynamics

Part of the challenge of ensuring PS-alignment is detecting problems with an APS system’s objectives---for example, via various processes of testing and evaluation. However, if a system behaves in PS-misaligned ways in some circumstances, those circumstances could well include the very processes of testing and evaluation we’re trying to use to detect problems. Thus, for example, if a strategically-aware AI system has problematic objectives that it could better achieve if it passed some training/evaluation process, it may have incentives to try to manipulate that process, and/or deceive relevant humans about its degree of alignment. 120 120 120 Bostrom (2014) calls this a “treacherous turn.” And if it is better at this type of manipulation/deception than we are at detecting its bad behavior, the appearance of safety and reliability on various tests may not tell us very much about how the system is liable to behave in other circumstances. 121 121 121 Muelhauser (2021) describes a rudimentary form of this behavior occurring in simulated evolution experiments: “As part of a project studying the evolution of (simulated) organisms, computer scientist Charles Ofria wanted to limit the replication rate of a digital organism. So, he programmed the system to pause after each mutation, measure the mutant’s replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and ‘play dead’ (pause replication) so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, Ofria randomized the inputs of the test environment so that it couldn’t be so easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.” See also Lewis (2017) for examples of possible deceptive dynamics learned in the context of bargaining: “Analysing the performance of our agents, we find evidence of sophisticated negotiation strategies. For example, we find instances of the model feigning interest in a valueless issue, so that it can later ‘compromise’ by conceding it. Deceit is a complex skill that requires hypothesising the other agent’s beliefs, and is learnt relatively late in child development (Talwar and Lee, 2002). Our agents have learnt to deceive without any explicit human design, simply by trying to achieve their goals.… Deception can be an effective negotiation tactic. We found numerous cases of our models initially feigning interest in a valueless item, only to later ‘compromise’ by conceding it.” (h/t to Luke Muelhauser for pointing out this example).

This sort of dynamic applies to very few of the technologies we’re familiar with (disciplines like computer security, which involve actively anticipating the strategies available to adversaries, may be the closest analog). That is: planes, rockets, nuclear plants, and so forth may be dangerous and complicated—but they are never actively trying to appear safer than they are, or to manipulate our processes of understanding and evaluating them. But at least in principle, sufficiently sophisticated AI systems could be doing this (though whether they will display this behavior in practice is another question).

That is, PS-aligning APS systems requires dealing, not with passive tools that might malfunction, but with possibly-adversarial strategic agents that could be actively optimizing in opposition to your efforts to ensure safety—and doing so using unprecedented degrees of cognitive capability, far exceeding that of humans. This seems a very significant additional challenge.

#### 4.4.3 Stakes of error

A final challenge comes from the escalating impact of certain types of mistakes. If a bridge fails, or a plane crashes, or a rocket explodes, the harm done is limited, contained, and passive. If an engineered virus escapes from the lab, however, it can spread rapidly, and become more and more difficult to contain as it goes. Viruses of this kind seem better analogs for practically PS-misaligned APS systems than planes and rockets.

That is, practical PS-alignment failures involve highly-capable, strategically-aware agents applying their capabilities (including, perhaps, the ability to copy themselves) to gaining and maintaining power in the world—and they may become more and more difficult to stop as their power grows. In dealing with systems that pose this sort of threat, there is much less room for the “error” component of trial-and-error, because the stakes of error are so much higher. And whatever their present safety, most current technologies involved many errors (plane crashes, rocket explosions, etc) along the way.

Indeed, if you’re trying to store an engineered virus that has a significant chance of killing ~the entire global population if it gets released, you need safety standards much higher than those we use, even now (after generations of trial and error), for bridges or planes—much higher, indeed, than we use for approximately anything (this is one key reason to never, ever create such a virus). For example, bridges need not be robust to nuclear attack; but the storage facility for such a virus should be—such attacks just aren’t sufficiently unlikely.

We might view the threat of PS-misaligned behavior from sufficiently capable APS systems in similar terms. And human track records of ensuring safety and security in our highest-stakes contexts---BSL-4 labs, nuclear power plants, nuclear weapons facilities---seem very far from comforting. 122 122 122 See Ord (2020) for some discussion of BSL-4 accidents.

### 4.5 Overall difficulty

Overall, my current best guess is that ensuring the full PS-alignment of APS systems is going to be very difficult, especially if we build them by searching over systems that satisfy external criteria, but which we don’t understand deeply, and whose objectives we don’t directly control.

It’s harder to reason in the abstract about the difficulty of practical PS-alignment, because it’s a much more flexible and contingent property: e.g., it depends crucially on the interactions between an agent’s capabilities, its objectives, and the circumstances it gets exposed to. And I think various of the tools discussed in section 4.3 —for example: focusing on specialized and/or myopic agents; restricting an agent’s capabilities; creating various sorts of incentives towards cooperation; using various types of non-agentic, strategically unaware, and/or practically aligned systems to help with oversight, incentive design, safety testing, etc—may well prove useful. And even if these tools don’t, themselves, scale in a way that can ensure practical PS-alignment at very high levels of capability, they may help us discover techniques that do.

However, there are problems with those tools, too, and more general problems that make practical PS-alignment seem like it may be unusually challenging—for example, difficulties understanding of the systems we’re building, the possibility of adversarial dynamics, and the extreme stakes of failure. And at a high-level, if you don’t have full PS-alignment, you’re engaged in an effort to control powerful, strategically-aware agents who don’t fully share your objectives, and who would seize power given certain opportunities. It seems, in general, an extremely dangerous game.

## 5 Deployment

Let’s turn, now, to whether we should expect to actually see practically PS-misaligned APS systems deployed in the world.

The previous section doesn’t settle this. In particular: if a technology is difficult to make safe, this doesn’t mean that lots of people will use it in unsafe ways. Rather, they might adjust their usage to reflect the degree of safety achieved. Thus, if we couldn’t build planes that reliably don’t crash, we wouldn’t expect to see people dying in plane crashes all the time (especially not after initial accidents); rather, we’d expect to see people not flying. And such caution becomes more likely as the stakes of safety failures increase.