Open-Endedness is Essential for Artificial Superhuman Intelligence — part 1/5

Cleaned-text code points [0, 14899). Bibliography omitted here; source.md includes it.

# Open-Endedness is Essential for Artificial Superhuman Intelligence

###### Abstract

In recent years there has been a tremendous surge in the general capabilities of AI systems, mainly fuelled by training foundation models on internet-scale data. Nevertheless, the creation of open-ended, ever self-improving AI remains elusive. In this position paper, we argue that the ingredients are now in place to achieve open-endedness in AI systems with respect to a human observer. Furthermore, we claim that such open-endedness is an essential property of any artificial superhuman intelligence (ASI). We begin by providing a concrete formal definition of open-endedness through the lens of novelty and learnability. We then illustrate a path towards ASI via open-ended systems built on top of foundation models, capable of making novel, human-relevant discoveries. We conclude by examining the safety implications of generally-capable open-ended AI. We expect that open-ended foundation models will prove to be an increasingly fertile and safety-critical area of research in the near future.

###### Keywords:

## 1 Introduction

Recent years have seen impressive progress in AI, mainly driven by foundation models ( Bommasani et al., 2021 ) . These models are increasingly used as agents in various applications ( Wang et al., 2023a ; Wu et al., 2023 ; Lifshitz et al., 2023 ; Wang et al., 2023c ; Liu et al., 2023b ; Zheng et al., 2024 ; Ahn et al., 2022 , e.g., ) . This represents significant progress towards artificial general intelligence (AGI), in the sense of reaching human-level performance on a wide range of tasks ( Legg and Hutter, 2007 ) . However, we are still missing a formal description of what it would take for an autonomous system to self-improve towards increasingly creative and diverse discoveries without end —a Cambrian explosion of emergent capabilities, behaviors, and artifacts. This kind of open-ended invention is the mechanism by which human individuals and society at large accumulates new knowledge and technology. Therefore, open-endedness must be a property of an artificial superhuman intelligence ( Morris et al., 2023 , ASI,) that can, by definition, accomplish a wide range of tasks at a level which no human can match. By the very nature of superhuman intelligence, open-ended discovery of innovative solutions is essential to empower humanity to manage its risks, just as society evolves norms and institutions to govern increasingly capable humans across generations ( Richerson et al., 2001 ) .

Foundation models such as large language models (LLMs) have scaled learning to large, static datasets scraped from the internet. Extrapolating, we may soon be running out of high-quality textual and visual data for training such models ( Villalobos et al., 2022 ) . Thus, open-endedness is unlikely to arise for free by training on ever-larger datasets. Rather, a system endowed with the open-endedness necessary for ASI will eventually have to create, refute and refine its own explanatory knowledge, in interaction with a source of evidence ( Deutsch, 2011 ) , as well as learning what data to learn from ( Jiang et al., 2022 ) . Moreover, for ASI to be useful and safe, it is important that open-endedness be guided towards knowledge that is understandable by and beneficial for humanity. Foundation models and open-endedness are orthogonal dimensions, whose combination is particularly powerful ( Lehman et al., 2022 ; Huang et al., 2022 ; Chen et al., 2023a ; Meyerson et al., 2023 ; Zhang et al., 2023 ; Wu et al., 2023 ; Wang et al., 2023a , cf.) . Open-ended algorithms endow foundation models with the ability to uncover new knowledge, while foundation models guide the search space for open-ended AI towards discovering human-relevant artifacts efficiently ( Liu et al., 2023a ; Ma et al., 2023 ; Romera-Paredes et al., 2024 ) . A formal definition of open-endedness can catalyze progress in this direction, offering clarity and focus to galvanize the research community.

We provide a new and precise definition of open-endedness in Section 2 , inspired by the open-ended systems in nature that have created life, the human brain, culture, and technology, as well as open-ended systems in silico that, for instance, have achieved superhuman level at the game of Go ( Silver et al., 2016 ) , generated human-level adaptation to novel 3D tasks ( Bauer et al., 2023 ) , self-improved language models ( Fernando et al., 2023 ; Yang et al., 2023a ) , unlocked the tech tree in Minecraft ( Wang et al., 2023a ) , and discovered new results in pure mathematics ( Romera-Paredes et al., 2024 ) . Open-endedness has been understood in a wide variety of ways ( Earle et al., 2021 ) ever since it gained prominence as a term in the study of artificial life ( Bedau, 1992 ; Bedau et al., 1998 ) and biological evolution ( Holland, 1992 ; McShea, 1996 ; Waddington, 2008 ) . Contrary to Stepney and Hickinbotham (2023) , we believe quantifying open-endedness is both possible and important going forward, and, akin to Sigaud et al. (2023) , we believe it can be achieved via the help of an observer external to the system. Our definition makes formal the aphorism of Lisa B. Soros that, as observers of an open-ended system, “we’ll be surprised but we’ll be surprised in a way that makes sense in retrospect”. Concretely, open-ended systems produce increasingly novel and surprising artifacts that are hard to predict, even for an observer who has learned to better predict by examining past artifacts. Once a system exhibits these characteristics, i.e. producing learnable but novel artifacts, we call it an open-ended system. This allows us to pinpoint the sense in which open-endedness is essential for ASI, to provide examples illustrating how existing open-ended AI systems lack generality, and to argue that present-day foundation models are not yet open-ended.

Historically, the field of open-endedness has faced numerous challenges. Principal among these has been the problem of structuring the search space so as to regularly produce artifacts which are both novel and interesting to humans ( Ma et al., 2023 ) . When humans make discoveries, they do so by “standing on the shoulders of giant human datasets” ( Clune, 2022 ) ; that is to say, utilising prior world, domain and commonsense knowledge, which they have acquired biologically or culturally. Since foundation models have been trained on vast amounts of human data, they capture human notions of interestingness ( Zhang et al., 2023 ) . Furthermore, they are general sequence modellers ( Mirchandani et al., 2023 ) and can generate variations from existing examples ( Meyerson et al., 2023 ) , thus serving as general mutation operators. This is compelling since with more advanced foundation models, practical implementations of open-ended systems become increasingly feasible. Taken together, open-ended foundation models can both vary (i.e., mutate) data and assess novelty and interestingness of real and generated data to decide what data to further explore (i.e., select) ( Jiang et al., 2022 ) .

In Section 3 we provide some concrete research directions for this marriage between open-endedness and foundation models, for example leveraging evolutionary algorithms and reinforcement learning. Generally capable open-ended systems may be both extremely powerful and increasingly prevalent, prompting pressing safety considerations ( Ecoffet et al., 2020 ) . In Section 4 , we argue that research into open-ended systems will be essential to safely and beneficially deploy any increasingly general and autonomous AI.

## 2 Defining Open-Endedness

### 2.1 Formal Definition

The notion of an open-ended system has received many colloquial definitions ( Soros and Stanley, 2014 ; Stanley and Lehman, 2015 ; Stanley et al., 2017 ; Stanley, 2019 ) . More formal approaches have often focused on the case of evolutionary systems, quantifying the increasing complexity ( McShea, 1996 ; Waddington, 2008 ) and perpetual novelty ( Holland, 1992 ) of biological evolution. Intuitively, an open-ended system endlessly produces novel and interesting artifacts. But novelty and interestingness have generally been characterised without sufficient precision, or in an overly narrow way. We provide a general-purpose, formal definition of open-endedness, as follows.

More formally, a system S S produces a sequence of artifacts X t X_{t} , indexed by time t t . An observer O O processes a new artifact X T X_{T} to determine its predictability given a history X 1 : t X_{1:t} of past ones. O O possesses a statistical model X ^ t \hat{X}_{t} which predicts an arbitrary future artifact based on its observations of the artifacts it has seen up to time t t . The observer judges the quality of their prediction based on a loss metric ℓ ⁡ ( X ^ t , X T ) \ell(\hat{X}_{t},X_{T}) , or ℓ ⁡ ( t , T ) \ell(t,T) for short. A natural implementation of X ^ t \hat{X}_{t} is as a learning algorithm.

A system displays novelty if artifacts become increasingly unpredictable with respect to the observer’s model at any fixed time t t , namely: ∀ t , ∀ T > t , ∃ T ′ > T : 𝔼 ⁡ [ ℓ ⁡ ( t , T ′ ) ] > 𝔼 ⁡ [ ℓ ⁡ ( t , T ) ] . \forall t,\forall T>t,\exists T^{\prime}>T:\mathbb{E}\left[\ell(t,T^{\prime})\right]>\mathbb{E}\left[\ell(t,T)\right]\,. In other words, there is always a less predictable artifact coming further in the future. 1 1 1 We take the expectation over any stochasticity in the artefacts; practically speaking, were the observer to make observations from identical copies of the system S S , the expectation of ℓ \ell would be approximated by the empirical mean.

The system is learnable whenever conditioning on a longer history makes artifacts more predictable, namely: ∀ T , ∀ t ⁡ < T , ∀ T > ​ t ′ > t : 𝔼 ⁡ [ ℓ ⁡ ( t ′ , T ) ] < 𝔼 ⁡ [ ℓ ⁡ ( t , T ) ] . \forall T,\forall t<T,\forall T>t^{\prime}>t:\mathbb{E}\left[\ell(t^{\prime},T)\right]<\mathbb{E}\left[\ell(t,T)\right]\,. Finally, a system is open-ended from the perspective of the observer O O if and only if it generates sequences of artifacts that are both novel and learnable (see Figure 1 ). The novelty aspect ensures the presence of information gain within the system, while learnability guarantees that this information gain holds meaning and is “interesting” to the observer.

For example, imagine that the system is a noisy TV producing uniform random noise ( Burda et al., 2018 ) . A noisy TV is learnable, allowing the observer to learn a statistical model that approximates the uniform distribution increasingly well; however, once the observer’s model converges to uniform the system loses its novelty: all that is left is aleatoric uncertainty, which is collapsed by the expectation. Now imagine that the system is a noisy TV switched periodically by a remote control to a random, arbitrary distribution. Every time the channel is changed, the observer may experience novelty; however, the system is now not learnable, because the history of artifacts (previous TV channels) are not correlated with the distribution of the next channel, so the model loss will not decrease in general. We provide an informal positive example in Appendix A.1 .

Our definition makes no explicit mention of “interestingness”. More precisely, interestingness is represented in our definition by the observer’s choice of loss function ℓ \ell . Thus, for us, the interesting parts of artifacts are precisely those features which the observer decides are useful to learn about. Different observers can, and do, find different artifacts interesting, by virtue of the different parts of the feature space they choose to learn with their statistical model.

We hope that our definition will serve as a useful grounding for future work. On the theoretical side, it provides a basis for proving whether a system is open-ended. On a practical note, it raises the prospect of searching for open-ended systems. In this paper, we shall use it to underpin the argument that open-endedness lies on the critical path towards ASI, and in particular that the combination of open-ended algorithms and foundation models is ripe to yield significant progress towards that aim. We examine some subtleties of our definition in Appendix A.2 .

### 2.2 Related Definitions

In the interests of space, we review the definitions of open-endedness most closely related to ours, covering more distantly related work in Appendix C . Soros and Stanley (2014) provided four necessary conditions for an evolutionary process to be open-ended, namely (1) that individuals must meet a minimal criterion in order to reproduce, (2) that evolution of individuals should create novel opportunities to meet the minimal criterion, (3) that individuals themselves should make decisions about how to interact with the world, and (4) that the potential complexity of the phenotype should not be limited by its representation. Our definition overlaps with these necessary conditions, but relaxes the constraint that the open-ended system is evolutionary. Our requirement that learnability is increasing can be seen as a generalisation of the minimal criterion in condition (1). Our requirement that the observer cannot intervene on the system is analogous to condition (3). Our requirement that novelty is increasing is analogous to conditions (2) and (4). Indeed, conditions (2) and (4) suggest that an open-ended system cannot be learned from a fixed data distribution.

To our knowledge, the most recent paper offering a definition of open-endedness is Sigaud et al. (2023) . The authors write: “an observer considers a process as open-ended if, for any time t t , there exists a time t ′ > t t^{\prime}>t at which the process generates a token that is new according to this observer’s perspective”. This definition has considerable overlap with ours. Like us, Sigaud et al. define open-endedness with respect to an observer. They consider the observer examining a sequence of tokens from a process, while we equivalently have the observer consider a sequence of artifacts from a system. Our requirement of novelty and learnability is compatible with their statement that the process should generate a token that is “new according to the observer’s perspective”. Our definition differs by being more precise about what this phrase means. In particular, we specify that what an observer considers “new” should be artifacts that are unpredictable according to their current statistical model of the system under consideration. Moreover, we specify that the observer’s “perspective” is generated by learning that statistical model on the history of artifacts thus far presented by the system. In particular, our definition can rule out systems that display continual “novelty” but are otherwise uninteresting, like white noise on a TV screen, for instance.

### 2.3 Types of Observer