Levels of AGI for Operationalizing Progress on the Path to AGI — part 3/3

Cleaned-text code points [30750, 43266). Bibliography omitted here; source.md includes it.

Discussions of AGI often include discussion of risk, including “x-risk” – existential ( for AI Safety, 2023 ) or other very extreme risks ( Shevlane et al., 2023 ) . A leveled approach to defining AGI enables a more nuanced discussion of how different combinations of performance and generality relate to different types of AI risk. While there is value in considering extreme risk scenarios, understanding AGI via our proposed ontology rather than as a single endpoint (per Principle 6) can help ensure that policymakers also identify and prioritize risks in the near-term and on the path to AGI.

### 6.1 Levels of AGI as a Framework for Risk Assessment

As we advance along our capability levels toward ASI, new risks are introduced, including misuse risks, alignment risks, and structural risks ( Zwetsloot & Dafoe, 2019 ) . For example, the “Expert AGI” level is likely to involve structural risks related to economic disruption and job displacement, as more and more industries reach the substitution threshold for machine intelligence in lieu of human labor. On the other hand, reaching “Expert AGI” likely alleviates some risks introduced by “Emerging AGI” and “Competent AGI,” such as the risk of incorrect task execution. The “Exceptional AGI” and “ASI” levels are where many concerns relating to x-risk are most likely to emerge (e.g., an AI that can outperform its human operators on a broad range of tasks might deceive them to achieve a mis-specified goal, as in misalignment thought experiments ( Christian, 2020 ) ).

Systemic risks such as destabilization of international relations may be a concern if the rate of progression between levels outpaces regulation or diplomacy (e.g., the first nation to achieve ASI may have a substantial geopolitical/military advantage, creating complex structural risks). At levels below “Expert AGI” (e.g., “Emerging AGI,” “Competent AGI,” and all “Narrow” AI categories), risks likely stem more from human actions (e.g., risks of AI misuse, whether accidental, incidental, or malicious). A more complete analysis of risk profiles associated with each level is a critical step toward developing a taxonomy of AGI that can guide safety/ethics research and policymaking.

Whether an AGI benchmark should include tests for potentially dangerous capabilities (e.g., the ability to deceive, to persuade ( Veerabadran et al., 2023 ) , or to perform advanced biochemistry ( Morris, 2023 ) ) is controversial. We lean on the side of including such capabilities in benchmarking, since most such skills tend to be dual use (having valid applications to socially positive scenarios as well as nefarious ones). Dangerous capability benchmarking can be de-risked via Principle 4 (Potential, not Deployment) by ensuring benchmarks for any dangerous or dual-use tasks are appropriately sandboxed and not defined in terms of deployment. However, including such tests in a public benchmark may allow malicious actors to optimize for these abilities; understanding how to mitigate risks associated with benchmarking dual-use abilities remains an important area for research by AI safety, AI ethics, and AI governance experts.

Concurrent with this work, Anthropic released Version 1.0 of its Responsible Scaling Policy (RSP) ( Anthropic, 2023b ) . This policy uses a levels-based approach (inspired by biosafety levels ( Richmond & McKinney, 2009 ) ) to define the level of risk associated with an AI system, identifying what dangerous capabilities may be associated with each AI Safety Level (ASL), and what containment or deployment measures should be taken at each level. Current SOTA generative AIs are classified as an ASL-2 risk. Including items matched to ASL capabilities in any AGI benchmark would connect points in our AGI taxonomy to specific risks and mitigations.

### 6.2 Capabilities vs. Autonomy

While capabilities provide prerequisites for AI risks, AI systems (including AGI systems) do not and will not operate in a vacuum. Rather, AI systems are deployed with particular interfaces and used to achieve particular tasks in specific scenarios. These contextual attributes (interface, task, scenario, end-user) have substantial bearing on risk.

Consider, for instance, the affordances of user interfaces for AGI systems. Increasing capabilities unlock new interaction paradigms, but do not determine them . Rather, system designers and end-users will settle on a mode of human-AI interaction ( Morris et al., 2023 ) that balances a variety of considerations, including safety. We propose characterizing human-AI interaction paradigms with six Levels of Autonomy , described in Table 2 .

These Levels of Autonomy are correlated with the Levels of AGI. Higher levels of autonomy are “unlocked” by AGI capability progression, though lower levels of autonomy may be desirable for particular tasks and contexts even as we reach higher levels of AGI. Carefully considered choices around human-AI interaction are vital to safe and responsible deployment of frontier AI models.

Unlike prior taxonomies of computer automation ( Sheridan et al., 1978 ; Sheridan & Parasuraman, 2005 ; Parasuraman et al., 2000 ) that take a computer-centric perspective (framing automation in terms of how much control the designer relinquishes to computers), we characterize the concept of autonomy through the lens of the nature of human-AI interaction style; further, our ontology considers how AI capabilities may enable particular interaction paradigms and how the combination of level of autonomy and level of AGI may impact risk. Shneiderman ( Shneiderman, 2020 ) observes that automation is not a zero-sum game, and that high levels of automation can co-exist with high levels of human control; this view is compatible with our perspective of considering automation through the perspective of varying styles of human-AI partnerships.

We emphasize the importance of the “No AI” paradigm for many contexts, including for education, enjoyment, assessment, or safety reasons. For example, in the domain of self-driving vehicles, when Level 5 Self-Driving technology is widely available, there may be reasons for using a Level 0 (No Automation) vehicle. These include for instructing a new driver (education), for pleasure by driving enthusiasts (enjoyment), for driver’s licensing exams (assessment), or in conditions where sensors cannot be relied upon such as technology failures or extreme weather events (safety). While Level 5 Self-Driving ( SAE International, 2021 ) vehicles would likely be a Level 4 or 5 Narrow AI under our taxonomy, the same considerations regarding human vs. computer autonomy apply to AGIs. We may develop an AGI, but choose not to deploy it autonomously, or choose to deploy it with differentiated autonomy levels in distinct circumstances as dictated by contextual considerations.

Certain aspects of generality may be required to make particular interaction paradigms desirable. For example, the Autonomy Levels 3, 4, and 5 (“Collaborator,” “Expert,” and “Agent”) may only work well if an AI system also demonstrates strong performance on certain metacognitive abilities (learning when to ask a human for help, theory of mind modeling, social-emotional skills). Implicit in our definition of Autonomy Level 5 (“AI as an Agent”) is that such a fully autonomous AI can act in an aligned fashion without continuous human oversight, but knows when to consult humans ( Shah et al., 2021 ) . Interfaces that support human-AI alignment through better task specification, the bridging of process gulfs, and evaluation of outputs ( Terry et al., 2023 ) are a vital area of research.

### 6.3 Human-AI Interaction and Risk Assessment

Table 2 illustrates the interplay between AGI Level, Autonomy Level, and risk. Advances in model performance and generality unlock additional interaction paradigm choices (including full autonomy). These interaction paradigms in turn introduce new classes of risk. The interplay of model capabilities and interaction design will enable more nuanced risk assessments and responsible deployment decisions than considering model capabilities alone.

Table 2 also provides concrete examples of each of our six proposed Levels of Autonomy. For each level of autonomy, we indicate the corresponding levels of performance and generality that “unlock” that interaction paradigm (i.e., the level of AGI at which it is possible or likely for that paradigm to be successfully deployed and adopted).

Our predictions regarding “unlocking” levels tend to require higher levels of performance for Narrow than for General AI systems; for instance, we posit that the use of AI as a Consultant is likely with either an Expert Narrow AI or an Emerging AGI. This discrepancy reflects the fact that for General systems, capability development is likely to be uneven; for example, a Level 1 General AI (“Emerging AGI”) may have Level 2 or perhaps even Level 3 performance across some subset of tasks. Such unevenness of capability for General AIs may unlock higher autonomy levels for particular tasks that are aligned with their specific strengths.

Considering AGI systems in the context of use by people allows us to reflect on the interplay between advances in models and advances in human-AI interaction paradigms. The role of model building research can be seen as helping systems’ capabilities progress along the path to AGI in their performance and generality, such that an AI system’s abilities will overlap an increasingly large portion of human abilities. Conversely, the role of human-AI interaction research can be viewed as ensuring new AI systems are usable by and useful to people such that AI systems successfully extend people’s capabilities (i.e., “intelligence augmentation” ( Brynjolfsson, 2022 ; Englebart, 1962 ) ).

## 7 Conclusion

Artificial General Intelligence is a concept of both aspirational and practical consequences. We analyzed nine definitions of AGI, identifying strengths and weaknesses. Based on this analysis, we introduced six principles for a clear, operationalizable definition of AGI: focusing on capabilities, not processes; focusing on generality and performance; focusing on cognitive and metacognitive (rather than physical) tasks; focusing on potential rather than deployment; focusing on ecological validity for benchmarking; and focusing on the path to AGI rather than a single endpoint.

With these principles in mind, we introduced our Levels of AGI ontology, which offers a more nuanced way to define progress toward AGI by considering generality (either Narrow or General) in tandem with five levels of performance (Emerging, Competent, Expert, Exceptional, and Superhuman). We reflected on how current AI systems and AGI definitions fit into this framing. Further, we discussed the implications of our principles for developing a living, ecologically valid AGI benchmark, and argue that such an endeavor, while sure to be challenging, is vital to engage with.

Finally, we considered how our principles and ontology can reshape discussions around the risks associated with AGI. Notably, we observed that AGI is not necessarily synonymous with autonomy. We introduced Levels of Autonomy that are unlocked, but not determined by, progression through the Levels of AGI. We illustrated how considering AGI Level jointly with Autonomy Level can provide more nuanced insights into risks associated with AI systems, underscoring the importance of investing in human-AI interaction research in tandem with model improvements.

We hope our framework will prove adaptable and scalable – for instance, how we define and measure progress toward AGI might change with technical advances such as improvements in interpretability that provide insight into models’ inner workings. Additionally, parts of our ontology such as Human-AI Interaction paradigms and associated risks might evolve as society itself adapts to advances in AI.

## Impact Statement

This position paper introduces a novel ontology that supports discussing progress toward AGI in a nuanced manner, with the aim of supporting clear communication among researchers, practitioners, and policymakers about systems’ capabilities and associated risks.

## Acknowledgements

Thank you to the members of the Google DeepMind PAGI team for their support of this effort, and to Martin Wattenberg, Michael Terry, Geoffrey Irving, Murray Shanahan, Dileep George, Blaise Agüera y Arcas, and Ben Shneiderman for helpful discussions about this topic.