Is Power-Seeking AI an Existential Risk? — part 16/16

Cleaned-text code points [231633, 232429). Bibliography omitted here; source.md includes it.

4. APS systems won’t be exposed to inputs where they seek power in misaligned and high-impact ways (say, collectively causing >$1 trillion 2021-dollars of damage) | not (1 or 2 or 3). 35%

5. This power-seeking won’t scale (in aggregate) to the point of permanently disempowering ~all humans | not (1 or 2 or 3 or 4). 60%

6. Such disempowerment won’t constitute an existential catastrophe | not (1 or 2 or 3 or 4 or 5). https://www.overleaf.com/project/626a43173c7082e1d1e2036c 5%

Implied probability that we’ll avoid scenarios like the one discussed in the report : ~95%

We can also imagine expanding the argument into many further premises, so as to bring out/highlight conjunctiveness that might be hiding within it. I think doing this might well be valuable; but I won’t attempt it here.