Open-Endedness is Essential for Artificial Superhuman Intelligence — part 2/5

Cleaned-text code points [14899, 30371). Bibliography omitted here; source.md includes it.

The choice of observer is a free parameter of great importance for our definition. From the perspective of AI research, there is a pre-eminent class of observers, namely humans. In other words, we wish to generate artifacts that are valuable to individual humans and to society. This provides a level of grounding for the open-ended system which narrows the search space considerably, as we shall argue in Section 3 . Nevertheless, our definition deliberately admits arbitrary observers, for several reasons. Firstly, it allows our definition to encompass open-ended systems which are not anthropocentric, such as biological evolution. Secondly, it allows us to reason about open-ended systems which might exceed human capabilities, so-called ASI. Thirdly, it allows us to determine whether systems can be open-ended with respect to any observer, as we did with the noisy TV. 2 2 2 There is one constraint on an observer which must be adhered to for our definition to make sense. The loss function must treat artifacts X X and predictions X ^ \hat{X} on an equal footing. In particular it must be fixed in advance without any knowledge of the system S S . Otherwise, an observer O O could find a system S S to be open-ended purely by discarding the artifacts from S S and constructing its own artifacts that it finds to be both novel and learnable.

Practically speaking, any given observer will have some time horizon τ \tau which bounds their observations of a system, i.e. t , T < τ t,T<\tau . This concept allows us to distinguish between systems which are open-ended on different timescales. We say that a system is infinitely open-ended with respect to an observer O O if it remains open-ended on any timescale τ → ∞ \tau\to\infty . We say that a system is finitely open-ended with time horizon τ \tau with respect to an observer O O if it is open-ended for t , T < τ t,T<\tau . Consider, for example, an agent trained in simulation with an automatic curriculum over tasks. In principle, a human observer might find observations of the agent behaviour to be infinitely open-ended, for the agent may accrue the ability to solve ever more diverse and surprising tasks. In practice ( Bauer et al., 2023 , cf. AdA,) , novelty starts to plateau after about 1 1 month of training, due to limitations in the richness of the task space and in the size of the agent’s neural network. Thus AdA is finitely open-ended with time horizon ≈ 1 \approx 1 month.

Similarly, an observer’s judgement will be influenced by the limitations of their cognitive abilities relative to the breadth of the domain. For example, a human observer who reads a curriculum of ever more complex articles from a current snapshot of Wikipedia may find such a system open-ended, but only until they reach the limit of their memory. A suitable ordering of Wikipedia articles will present novel information, in the sense that every now and then an article will be more unpredictable than we have hitherto seen. We might also expect that this information will be learnable, because human knowledge is interlinked, in the sense that knowing more about one topic makes it easier to understand other topics that may crop up later. However, once human memory capacity is saturated, the human observer will start to forget previous articles. This violates learnability: in calculus, for instance, once one has forgotten the definition of a derivative, one will find it harder to understand an article about the chain rule. Therefore, conditioning on a history longer than an observer’s recall doesn’t necessarily make the current artifact more predictable.

This example brings to light three interesting threads. Firstly, the open-endedness of human technology, as observed by humans, relies on our ability to compress knowledge into a form that can be maintained within our collective memory: indeed, we present an alternative definition of open-endedness in the language of compression in Appendix B . Secondly, an artificial superhuman intelligence may have less stringent memory constraints than humans, and therefore may judge itself to be open-ended beyond the point at which humans assess it to be so, re-emphasising that human observers must be considered pre-eminent for the purposes of safety, as we explore further in Section 4 . Thirdly, the open-endedness in this example is a function of the breadth of the domain. In a narrower domain, elliptic curve cryptography say, the set of relevant Wikipedia articles would be much smaller, so a human observer would find this open-ended only until they had understood every article, at which point novelty would be violated. Nevertheless, humans can, and frequently do, make new discoveries in narrow domains via experimentation and reasoning; amassing a vast, static trove of data is not the be all and end all of open-endedness.

### 2.4 Examples

In this section, we discuss some popular systems that are open-ended but not general, or that are general but not open-ended, with respect to a human observer. This serves two purposes. Firstly, it demonstrates that our definition is not so restrictive as to rule out systems that are intuitively open-ended, and is not so loose as to include systems that intuitively lack open-endedness. Secondly, it motivates the benefits that foundation models can provide in addressing the limitations of current open-ended systems and vice versa.

Our first archetypal open-ended system is AlphaGo ( Silver et al., 2016 ) . Consider as artifacts the sequence of policies produced across training by AlphaGo. After sufficient training, AlphaGo produces policies which are novel to human expert players, in the sense that they play moves which would be low probability for human professionals but which nevertheless are winning against the best humans. Furthermore, humans can improve their win rate against AlphaGo by learning from AlphaGo’s behavior ( Shin et al., 2023 ) . Yet, AlphaGo keeps discovering new policies that can beat even a human who has learned from previous AlphaGo artifacts. Thus, so far as a human is concerned, AlphaGo is both novel and learnable. AlphaGo is just one representative from a class of open-ended algorithms that augment reinforcement learning with self-play ( Samuel, 1959 ) , achieving or exceeding human-level play in Go, Chess, Shogi ( Silver et al., 2017 ) , StarCraft II ( Vinyals et al., 2019 ) Stratego ( Perolat et al., 2022 ) , DotA ( Berner et al., 2019 ) , and Diplomacy ( Bakhtin et al., 2022 ) .

AlphaGo is an example of an open-ended system that achieves narrow superhuman intelligence ( Morris et al., 2023 ) . This limits its utility: self-play of this kind cannot by itself help us to discover new science or technology that requires combining insight from disparate fields, or taking actions across a range of modalities, timescales and contexts. The constraints of the game rules make the search for novel and learnable artifacts tractable, and these artifacts are found to be novel and learnable by human observers largely because it was humans who invented the game.

Our second archetypal open-ended system is AdA ( Bauer et al., 2023 ; OEL Team et al., 2021 ) . AdA is a large-scale agent that learns to solve tasks in an 3D-environment called XLand2. In XLand2 there are 25B possible task variants, corresponding to different world topologies and a variety of possible games within each world, that are prioritized for learning potential ( Jiang et al., 2021 ) . Checkpoints of the AdA agent across training are open-ended with respect to a human observer who attempts to predict what capabilities the agent might show. Across training, the agent gradually accumulates zero-shot and few-shot capabilities over an ever wider set of held-out environments, requiring ever more complex skills. Thus the human continually observes novel capabilities in the agent. Furthermore, the prioritization of task variants provides an interpretable ordering to the accumulation of skills in the agent, rendering this learnable by a human. AdA represents a wider class of open-ended algorithms driven by unsupervised environment design ( Dennis et al., 2020 ; Justesen et al., 2018 , UED,) , which establish an automatic curriculum ( Leibo et al., 2019 ; Baker et al., 2020 ) of environments in the zone of proximal development for agent learning ( Vygotsky and Cole, 1978 ) .

It is natural to ask whether AdA would continue to be judged as open-ended by a human observer should training be continued indefinitely. Results in Bauer et al. (2023) suggest that novelty starts to plateau, implying that with an order of magnitude more compute AdA would almost certainly not be open-ended. Indeed, the authors show that both increasing the size of the agent and increasing the number of tasks allow the agent to generalize to a wider range of environments. Thus, in order for this system to be open-ended on longer timescales, one would need an even richer environment and an even more capable agent to sustain the agent-environment co-evolution inherent in UED.

Our third archetypal open-ended system is POET ( Wang et al., 2019 ; Wang et al., 2020 ) . POET trains a population of agents, each of which is paired with an environment that is evolving over the course of training. These paired agent-environment artifacts are open-ended with respect to a human observer seeking to model the features of the environments that arise, or equivalently the skills the paired agents possess. A Quality Diversity algorithm ( Pugh et al., 2016 ; Mouret and Clune, 2015 , QD,) is deployed with respect to the environments, hunting for challenging problems that lead to diverging performance across the population. QD is an example of a wider class of open-ended algorithms, namely evolutionary algorithms, which we encounter again in Section 3.4 .

Crucially, POET periodically transfers agents from one environment to another, which results in an empirical example of the stepping stone phenomenon ( Stanley and Lehman, 2015 ) : agents can eventually solve incredibly challenging environments that are not possible to solve with direct optimization. As a result of training for billions of environment steps, POET produces a diverse population of highly capable specialist agents, which can solve novel environments that are created through coevolution with the population ( Brant and Stanley, 2017 ) . Novelty arises because of the mutation operator in the QD algorithm, which yields new and unpredictable environments. Learnability arises because each mutation is small, so the past lineage of an environment is a good guide to its current features. Just as for AdA, the key limitation on open-endedness is the environment parameterization itself: eventually POET will plateau once the agent can solve all possible terrains.

Our final example is contemporary foundation models . These are a negative example; they are not open-ended by our definition with respect to any observer who can model their training dataset. The justification for this follows immediately from our consideration of the noisy TV in Section 2.1 . Contemporary foundation models are typically trained on fixed datasets. If the distribution of this data is learnable, which it must be, for the foundation model learned it in the first place, then it cannot be endlessly novel, because eventually the observer will have modelled the epistemic uncertainty. As we saw in Section 2.3 , foundation models may appear open-ended to human observers if the domain of enquiry is sufficiently broad, by virtue of the memory limitations of the human brain. However, if the focus is narrowed, for instance to tasks that require planning ( Momennejad et al., 2024 ; Pallagani et al., 2023 ; Valmeekam et al., 2023 ) , the limitations of the foundation model in generating novel, correct solutions are exposed.

Since foundation models are periodically retrained on new data, including data generated by their own interactions with humans and the real world, one could argue that the data distribution is not really fixed. In some quarters, this kind of distributional shift is seen as an annoyance, even one which threatens “model collapse” ( Shumailov et al., 2023 ) . We flip this argument on its head, and contend that augmenting foundation models with open-endedness offers a path towards ASI. Similarly, the fact that foundation models are typically conditional on context breaks the logic that they cannot be open-ended. In principle, the context of a foundation model can be recruited to recombine concepts in an open-ended way by leveraging some external measure of validity. This brings us neatly to some concrete suggestions for how to build open-ended foundation models.

## 3 Open-Ended Foundation Models

We have defined open-endedness and discussed why the current foundation model training paradigm is not open-ended. We believe that the trend of improving foundation models trained on passive data by scaling alone will soon plateau, and it will not be enough to reach ASI. Our position is that open-endedness is a property of any ASI, and that foundation models provide the missing ingredient required for domain-general open-endedness. Further, we believe that there may be only a few remaining steps required to achieve open-endedness with foundation models. In the following subsections, we sketch four overlapping paths towards open-ended foundation models that lend credence to this belief. The paths are neither intended to be prescriptive nor exhaustive. Indeed, recent publications such as ( Wong et al., 2023b ; Sharma et al., 2023 ) point to other paths.

Before proceeding, we must justify our claim that a future foundation model trained passively on some large corpus of human data is unlikely to spontaneously acquire open-endedness. In principle, should we reach ASI, there will be some sum total of data which the model has consumed during its training, possibly via several intermediate stages. Therefore, our claim is not about the impossibility of assembling such a dataset. Rather, we suggest that it is unlikely that this dataset can be pre-collected offline in an efficient way. The reason is that open-endedness is fundamentally an experiential process: producing novelty and learnability in the eyes of an observer requires continual online adaptation on the basis of the artifacts already produced, in the context of that observer’s evolving prior beliefs.

What would it take to collect offline a static dataset from which such an experiential skill could be learned? Such a dataset must contain a treasure trove of artifacts which themselves crisply show novelty and learnability. Yet the process by which culture evolves, ideas develop, inventions arise and technologies proliferate is seldom recorded neatly and comprehensively. The alternative paradigm, in which experience is “built in” to the open-ended system, is well illustrated by the scientific method. Since the Enlightenment, the simple process of making hypotheses on the basis of current knowledge, falsifying them with experiments based on a source of evidence, and codifying the results into new knowledge has yielded unprecedented progress in science and technology ( Deutsch, 2011 ) . In our view, the fastest path to ASI will take inspiration from the scientific method, compiling a dataset online by the explicit combination of foundation models and open-ended algorithms.

### 3.1 Reinforcement Learning