Open-Endedness is Essential for Artificial Superhuman Intelligence — part 4/5

Cleaned-text code points [45524, 60513). Bibliography omitted here; source.md includes it.

There are significant non-technical concerns in ensuring that society can understand, prepare for, and appropriately react to new technological capabilities emerging from open-ended foundation models. Indeed, the impact of AI systems is not just felt at the individual level, but also at the level of the collectives that structure our society—communities, organisations, markets and nation states, to name a few. Since the artifacts arising from open-ended foundation models will by definition appear novel, we must devote prospective attention to the ways in which these could harm or benefit the cooperative infrastructure of society ( Dafoe et al., 2020 ) . Likewise, we must develop mechanisms to avoid tipping points driven by feedback loops, like flash crashes ( Aldrich et al., 2017 ) . Decision-makers should be prepared to adapt governance rapidly and retrospectively in response to open-ended artifacts, finding a good balance between collecting information and avoiding entrenchment of undesirable artifacts ( Collingridge, 1980 ) .

### 4.5 Emergent Risks of Open-Ended Systems

Even if each subcomponent of Figure 2 can be made safe, it may still be the case that the aggregate joint human-AI open-ended system leads to unforeseen problems. For instance, two systems that are open-ended in isolation could negatively interact to cause neither to be open-ended. This would mean a cessation of progress and an inability to collectively respond to new challenges. While such emergent effects have been studied in multi-agent systems ( Johanson et al., 2022 ) and ASI safety ( Critch and Krueger, 2020 ) solutions are still elusive, and an understanding of these effects is critical to the safe deployment of open-ended systems.

If such problems are inevitable and unpredictable, we would need our human-AI open-ended systems to adapt to solve novel ASI safety failures as they arise. Due to the inherent unpredictability of knowledge creation, these problems may be both unavoidable and solvable once as they arise ( Deutsch, 2011 ) . We should be building an open-ended system whose safety is anti-fragile ( Taleb, 2014 ) , adapting to emerging safety risks and getting stronger for it. This entails designing techniques for understanding, monitoring, and rapidly coordinating responses to emerging risks.

## 5 Conclusion and Outlook

Foundation models have led to a rapid increase in the generality of current AI systems. However, current foundation models are limited in their capability to discover new knowledge. In this paper, our position is that to further advance in levels of AGI towards ASI, we require systems that are open-ended —endowed with the ability to generate novel and learnable artifacts for a human observer. There has never been a more exciting time to build such systems, with foundation models already exhibiting general human-like knowledge that both accelerates further learning and guides this learning towards human-relevant artifacts.

As we develop and deploy more generally-capable open-ended systems, novel safety concerns arise that will be critical to address. In order to realise the benefits of such systems, it is important that the human observer remains able to learn from the novel artifacts, bringing fields such as explainability to the forefront of open-endedness research. If these endeavors are successful, then we believe open-ended foundation models could lead to advances that drastically enhance modern society.

## Impact Statement

Our work provides a formal definition of open-endedness, and provides a discussion on its significance for the pursuit of ASI. We explore current research directions in the field, emphasising the potential of combining open-endedness with foundation models as a pre-eminent path towards achieving ASI. Developed responsibly, we believe that such open-ended foundation models can have tremendous positive impact on the society, accelerating scientific and technological breakthroughs, enhancing human creativity through a collaborative feedback loop, and acting as an engine for general knowledge expansion across many fields. Recognising the profound implications of this concept, we dedicate the entirety of Section 4 to an initial analysis of potential risks and societal impacts, offering frameworks for the responsible and ethical development of ASI. We hope that highlighting these issues early will help to promote safety, responsibility and accountability as the field grows.

## Acknowledgements

We gratefully acknowledge Dave Abel for providing valuable feedback on an early draft of this paper. We are thankful to the designers at the Noun Project , from which we sourced graphics under the CC BY 3.0 licence as follows: “ tick ” icon by kareemovic, “ Delete ” icon by kareemovic, “ alien ” icon by Artem Yurov, “ girl ” icon by Teewara soontorn, “ year of rat ” icon by DailyPM, “ aircraft ” icon by mikicon, “ concorde ” icon by mikicon, “ Plane ” icon by CAMB, “ humans ” icon by Ifanicon, and “ Robot ” icon by Deemak Daksina.

## Appendix A Illustrating Open-Endedness

### A.1 An Informal Example

To illustrate our definition informally, we provide a relatable real-world example. Let S S be a research lab and the x t x_{t} be academic papers published by the lab. A natural choice of observer O O is a research student in the field at a different lab. Roughly speaking, a research student sees novelty in a line of work if, based on their knowledge of the literature up to time t t , given any subsequent paper x T x_{T} they can always find a later paper x T ′ x_{T^{\prime}} that is more surprising than x T x_{T} . This is intuitively sensible, a putative student with knowledge of Newtonian mechanics will find Maxwell’s equations hard to predict, quantum mechanics even more surprising, and contemporary particle physics very far outside their current level of comprehension. A research student sees learnability in a line of work if they find that reading the previous papers helps them better to predict the contents of the current paper. Again, this appeals to our intuition: part of the purpose of citations, for instance, is to point new researchers at previous works that will help to deepen their understanding of the current work.

Our interpretation of “interestingness” as learnability also makes sense from the perspective of a research student. A research student may choose to ignore a paper’s choice of font, but will likely pay close attention to the details of a novel method that yields state-of-the-art results. Thus the student finds interesting the parts of the paper from which they can learn the most. Similarly, the requirement that the loss metric ℓ \ell be chosen without knowledge of S S finds a natural interpretation here. A research student cannot judge the open-endedness of a stack of papers by choosing to never read the papers and instead inventing their own line of research with no reference to previous works.

### A.2 Definitional Subtleties

Self-play illustrates some subtleties in our definition. The first subtlety is the dependence of open-endedness on the choice of observer. Suppose that O O is an oracle who knows the Nash strategy to play in Go. Assuming that the oracle is modelling the win-rate of AlphaZero’s artifacts against its own policy, it will never find any AlphaZero policy to be novel. Therefore the oracle does not find AlphaZero to be open-ended. The second subtlety is the dependence of open-endedness on the learning limitations of the observer. To an average human Go player, as opposed to an expert, AlphaZero becomes novel earlier in training, and at some point ceases to be learnable, because the average player cannot figure out how to improve their own play with reference to very unusual style of a superhuman policy. Thus, open-ended systems only remain open-ended while they can “educate” their observers. We posit that superhuman intelligence will be interesting to humans only as far as humans can learn to understand it. The third subtlety is that open-ended systems need not explore a problem space fully to qualify as open-ended. Recently, adversarial search was shown to yield policies that beat reimplementations of AlphaZero and which are so simple that even amateur humans can learn them [ Wang et al., 2023b ] . Novelty and learnability give no guarantee of coverage.

Because our definition is based on the perspective of an external observer, one could worry that this makes it impossible to make any sort of objective claims about the open-endedness of any particular system, in harmony with the arguments of Stanley and Soros [2016] , Stepney and Hickinbotham [2023] . There are two factors which mitigate this concern. Firstly, the definition of open-endedness becomes objective given any fixed observer, and so it becomes a measurable claim, in the sense that theorems can be written and experiments conducted. For instance, if we care about open-endedness with respect to humans, open-endedness can be measured experimentally by how well humans can predict the system. By having observer-dependence explicit in our definition, we make precise the intuition that different observers, with different prior knowledge, different cognitive capabilities and different timescales, are likely to judge the same system in different ways. Thus our definition gracefully encompasses the diversity in perspectives of human individuals and groups (such as companies or governments), as well as the possibility that AI systems themselves could be observers.

Secondly, while our definition of open-endedness depends on an external observer, it is an open question as to whether all “reasonable” observers would judge the same systems to be open-ended. Since our definition rests on a notion of predictability with respect to the observer, our definition will be as subjective as the underlying notion of predictability. One may believe that predictability can be accurately and objectively modeled as Solomonoff induction [ Solomonoff, 1960 ] . Thus if reasonable observers are taken to be those whose predictions eventually follow something approximating Solomonoff induction, then any observer in this class would eventually agree on which systems are open-ended.

Practically speaking, there are various existing methods in the literature which can immediately be adapted to assess the open-endedness of a system. First, one might elicit direct human feedback on learnability and novelty of artifacts, in the same spirit as RLHF [ Ouyang et al., 2022 ] or PicBreeder [ Secretan et al., 2008 ] . Second, one can use large language models themselves as judges of novelty and learnability, as argued for in OMNI [ Zhang et al., 2023 ] . Finally, one could explicitly learn a model of the artifacts with an online learning method like Follow-the-Regularized-Leader [ Hazan and Kale, 2010 ] .

Can an open-ended system be its own observer? In principle, there is nothing in our definition that rules out self-observing open-ended systems. For example, an individual self-improving agent could generate a series of artifacts, each one of which is novel (surprising compared to the previous artifacts) and learnable (increasingly predictable given the more history of the past artifacts). When the feedback from self-observation is used to improve the system itself, we call the observer a proxy observer for it no longer sits outside the system.

For example, AlphaGo can be seen as an example of a self-observing system, in that the agent trains in self-play i.e. it observes its own policy as an opponent, is challenged by the novel discoveries of search, and learns from them to improve the policy. Likewise, humans can experience “Eureka moments”, when an individual suddenly reconceptualizes a problem in a ways that yields a solution [ Sternberg and Davidson, 1995 ] . A series of Eureka moments, each building on the last, is a self-observing open-ended system: the human generates discoveries which are novel to themselves, but which are also predictive of the next discovery.

Our notions of learnability is rather strict, in that it requires that the loss be decreasing for all t ′ > t t^{\prime}>t . A weaker and more practical notion of learnability might state that it should be probabilistically unlikely that the loss will increase as a function of t t : ∀ T , ∀ t ⁡ < T , ∀ T > ​ t ′ > t : ℙ ⁡ ( ℓ ⁡ ( t ′ , T ) ≥ ℓ ⁡ ( t , T ) ) < δ . \forall T,\forall t<T,\forall T>t^{\prime}>t:\mathbb{P}\left(\ell(t^{\prime},T)\geq\ell(t,T)\right)<\delta\,.

It would be interesting to compare the consequences of δ \delta being a constant with the situation in which δ \delta has some appropriate dependence on the variables ( t , t ′ , T ) (t,t^{\prime},T) . Similarly, one could weaken the notion of novelty to state that it should be probabilistically unlikely that the loss will decrease as a function of T T . We believe that there may be several related and differently useful variants on our definition that would be interesting to independently study, in a similar way that there are many notions of convergence which are interesting, related, and differently useful.

## Appendix B Alternative Definition

In Section 2.1 we provided a formal definition of open-endedness in the language of statistical learning. Here we give an alternative definition which we conjecture is equivalent under appropriate conditions. The alternative definition is phrased in the language of compression, a topic with known formal connections to statistical learning [ Hutter, 2004 , David et al., 2016 , Campi and Garatti, 2023 , Delétang et al., 2023 ] .

A system S S produces a sequence of artifacts X t ∈ 𝒳 X_{t}\in\mathcal{X} , indexed by time t t . An observer O O processes a new artifact X T X_{T} to determine its information content given a history h t = X 1 : t h_{t}=X_{1:t} of past ones. O O possesses a history-dependent compression map C h t : 𝒳 → { 0 , 1 } ∗ C_{h_{t}}:\mathcal{X}\rightarrow\{0,1\}^{*} which encodes X T X_{T} into a binary string of length | C h t ​ ( X T ) | |C_{h_{t}}(X_{T})| .

The system displays novelty if the information content increases, namely: ∀ t , ∀ T > t , ∃ T ′ > T : | C h t ​ ( X T ′ ) | > | C h t ​ ( X T ) | . \forall t,\forall T>t,\exists T^{\prime}>T:|C_{h_{t}}(X_{T^{\prime}})|>|C_{h_{t}}(X_{T})|. In other words, the complexity of the artifacts grows, according to the observer.

The system is learnable if conditioning on a longer history increases compressibility, namely: ∀ T , ∀ t ⁡ < T , ∀ T > ​ t ′ > t : | C h t ′ ​ ( X T ) | < | C h t ​ ( X T ) | . \forall T,\forall t<T,\forall T>t^{\prime}>t:|C_{h_{t^{\prime}}}(X_{T})|<|C_{h_{t}}(X_{T})|. In other words, as its history grows, the observer must be able to keep extracting additional patterns that help it compress future artifacts.

Finally, a system is open-ended from the perspective of O O if and only if it generates sequences of artifacts that are both novel and learnable.