Open-Endedness is Essential for Artificial Superhuman Intelligence — part 3/5

Cleaned-text code points [30371, 45524). Bibliography omitted here; source.md includes it.

The framework of Reinforcement Learning (RL) has been at the forefront of achieving superhuman performance in narrow domains, such as AlphaGo’s groundbreaking strategies that have enriched the human understanding of the game of Go. RL agents act deliberately so as to shape their stream of experience for both accumulating reward (exploitation) and learning about how to increase expected reward in the future (exploration). A nuanced extension are agents that set their own goals to (learn to) pursue; and generating the sequence of these goals can itself be an open-ended process, which drives open-ended experience generation ( Colas et al., 2022 ) . Voyager ( Wang et al., 2023a ) provides an early example of how RL-like self-improvement can be built on top of foundation models, without the need for explicit parameter updates or established RL algorithms. Instead, Voyager assembles an LLM-powered curriculum, uses iterative prompting as an improvement operator, and assembles verified skills into a library for hierarchical reuse.

A key problem in RL is how to shape exploration towards novel and learnable behaviors in high-dimensional domains, as discussed in Jiang et al. (2022) . Exploration can be guided, for instance, by pseudo-rewards ( Bellemare et al., 2016 ; Burda et al., 2018 ; Du et al., 2023b ) , modulation ( Schaul et al., 2019 ) or an automated curriculum that selects relevant tasks ( Jiang et al., 2021 ; Parker-Holder et al., 2022 ; Samvelyan et al., 2023 ) . To generalize this, a useful abstraction may be the notion of a proxy observer , which sits within the system and proactively guides it to generate novel and learnable content for the true external observer. In the past this guidance was provided on the basis of simple metrics such as TD-error, but now we can leverage foundation models to guide exploration towards artifacts that more closely align with what a human observer deems to be novel and interesting ( Jiang et al., 2022 ) . There is already evidence that this approach may be effective, with LLMs providing agent rewards from text in an environment ( Klissarov et al., 2023 ) and compiling a curriculum of tasks based on their interestingness ( Zhang et al., 2023 ; Faldor et al., 2024 ) .

While RL considers the first-person perspective of an agent interacting with an environment, a different perspective centers on multi-agent dynamics, and the additional richness arising from all the ways that different (possibly heterogeneous) agents can interact with each other, adapt to each other, or learn from each other. The presence of multiple learning agents provides a source of non-stationarity, such that the optimal strategy for each individual will change over time, potentially in an open-ended manner. Non-stationary dynamics been used to achieve or exceed human-level performance in games like StarCraft, DotA and Stratego. There is early evidence that multi-agent systems may help to improve factuality and reasoning in LLMs via debate ( Du et al., 2023c ; Tang et al., 2023 ) , although there is much more research needed before superhuman capability is reached.

### 3.2 Self-Improvement

To achieve open-endedness, a model must not only consume knowledge from pre-collected feedback as in, for example, RLHF ( Ziegler et al., 2019 ) , but also generate new knowledge, in form of hypotheses, insights or creative outputs beyond the human curated training data. A self-improvement loop should allow the agent to actively engage in tasks that push the boundary of its knowledge and capabilities, for example via leveraging tools such as search engines, simulated environments, calculators or interpreters and interacting with other agents ( Jiang et al., 2022 ; Schick et al., 2024 ) . This requires the model to have a scalable mechanism to evaluate its own performance, identify areas for improvement, and adapt its learning process accordingly.

There is growing evidence that foundation models can be leveraged for feedback in place of humans, and can significantly amplify data generated by humans. Examples include self-critique and revision for training harmless assistants ( Bai et al., 2022 ) and guiding human evaluators ( Saunders et al., 2022 ) , self-correction for tool-use ( Gou et al., 2023 ) , self-instruction for instruction following ( Wang et al., 2022 ) , self-debugging for code generation ( Chen et al., 2023b ) , self-rewarding for instruction following ( Yuan et al., 2024 ) , and leveraging VLMs as reward functions for control ( Baumli et al., 2023 ) . These works hint at the possibility of foundation models generating their own samples and refining them in an open-ended way.

### 3.3 Task Generation

Closely related to both RL and self-improvement is the problem of task generation, also known as the “problem problem” ( Leibo et al., 2019 ) . One great candidate approach for open-endedness is to keep adapting the difficulty of tasks to an agent’s capability so that they remain forever challenging yet learnable. Past examples of this type of system include setter-solvers ( Schmidhuber, 1991b ) and unsupervised environment design ( Dennis et al., 2020 ; Justesen et al., 2018 ; Wang et al., 2019 ) . With the advent of foundation models, it has become feasible to use the Internet itself as an environment ( Jiang et al., 2022 ; Gur et al., 2021 ) via web-based APIs, affording agents with an incredibly rich, ever-growing and human-relevant task domain ( Zhou et al., 2023 ) .

Another possibility is to instead learn world models—predictive simulators that can generate future outputs conditioned on text or actions. A promising approach is to consider a foundation model to be a world model itself, since it is capable of predicting the future ( Wong et al., 2023a ; Gurnee and Tegmark, 2023 ; Park et al., 2023 ) . Learned world models like Genie ( Bruce et al., 2024 ) , and text-to-video generation models like Sora ( Brooks et al., 2024 ) demonstrate that foundation video models can be used as learned simulators, including in real-world settings like robotics ( Yang et al., 2023b ) and autonomous driving ( Hu et al., 2023 ) . If these works combine with learned multi-modal reward models ( Chan et al., 2023 ; Du et al., 2023a ) , they could be used to generate an open-ended curriculum of tasks, scaling to task spaces far larger and more photorealistic than can currently be achieved. At sufficient scale, this may provide a path to generating AI agents with superhuman adaptability across a wide range of previously unseen tasks, which can be deployed in the real world across the rapidly closing Sim-to-Real gap ( Huang et al., 2023 ) .

### 3.4 Evolutionary Algorithms

Evolutionary methods offer a promising path to generate open-ended systems with foundation models ( Wu et al., 2024 ) . LLMs are well-placed to act as selection and mutation operators, as they have been trained on vast datasets of human knowledge, culture and preferences. For example, LLMs offer a mechanism through which to make semantically meaningful mutations via text ( Lehman et al., 2022 ; Meyerson et al., 2023 ; Chen et al., 2023a ) . The simplest such approach may be via prompts , which already allow foundation models to further improve their performance. Recent works have shown it is possible to far surpass human designed prompts, leading to stronger models ( Fernando et al., 2023 ; Yang et al., 2023a ; Guo et al., 2023 ) . More recently, Bradley et al. (2023) and Samvelyan et al. (2024) went further, using an evolutionary algorithm and LLMs to both generate variation and evaluate the quality and diversity of candidate text, making it possible to guide the search for creative and novel outputs. In the future it may be possible to further refine a model on these outputs, or use them for planning ( Gandhi et al., 2023 ) , to achieve self-improvement.

Another angle of attack for evolutionary methods is in the space of code (also known as genetic programming). Foundation models have proven to be competent at producing diverse and novel programs, providing a means of iterating upon an archive of candidate solutions. For example, Eureka ( Ma et al., 2023 ) evolves code-based reward functions to learn complex control behaviors. Similarly, FunSearch ( Romera-Paredes et al., 2024 ) evolves programs that represent new mathematical knowledge. These examples are focused on specific domains, and it remains an open problem to scale code evolution to a more general setting.

## 4 Achieving ASI Responsibly

Now that we have foundation models, designing a truly general open-ended learning system may be within our grasp. However, the power of open-endedness comes with a swathe of notable safety risks—beyond existing safety considerations facing foundation models ( Ecoffet et al., 2020 ) . Finding solutions to these challenges are interesting and important core problems in open-endedness research. Because the solutions to these problems may well depend on the design of the open-ended system, it is critical that safety and open-endedness are pursued in tandem. We cover them here not to hold them separate from other directions in open-endedness—in fact many of these problems are current practical limitations of artificial open-ended systems. Rather, this section is intended to draw specific attention to these problems as some of the most fundamental and exciting directions for research in the field. Of course, this short section cannot do justice to the breadth of concerns. Hence, where possible, we provide references to the wealth of knowledge in the ASI safety community.

We organize our understanding of these risks similar to ( Critch and Krueger, 2020 ) by focusing on the ways knowledge is created and transmitted through the joint human-AI open-ended process in Figure 2 . A powerful open-ended system which has the problems listed in this section is not a beneficial open-ended system, and we believe it is not one we should be striving to build. Solving these problems is not just making open-ended systems safer, but also making them usable by humans. As such, addressing these problems should be thought of as minimum specifications of an open-ended system that we would want to build.

### 4.1 AI Creation and Agency

AI systems powering the open-ended creation of new knowledge could lead to powerful new affordances. Without direction, these creations could be the source of dual-use dangers ( Urbina et al., 2022 ) . The danger is magnified when the open-ended systems take immediate action in an environment. Current state-of-the-art systems operate in narrow, simulated environments ( Wang et al., 2023a ; OEL Team et al., 2021 ; Bauer et al., 2023 ) . However, as AI is trained in broader, more diverse simulations or is even deployed (and continues to learn) in the real world, it becomes critical to understand the dangers. The agency of open-ended AI poses several safety risks, such as goal misgeneralization ( di Langosco et al., 2022 ; Shah et al., 2022 ) and specification gaming ( Clark and Amodei, 2016 ) . Open-ended search can be seen as an ambitiously aggressive form of exploration; thus one could hope to use similar approaches to mitigate the dangers of exploration as in RL, like safe exploration ( Garcıa and Fernández, 2015 ) and impact regularization ( Krakovna et al., 2018 ; Turner et al., 2020 ) .

### 4.2 Humans Understanding AI Creations

In order to provide informed oversight and direction when guiding an open-ended system, human observers need to at least partially understand the significance of the new artifacts that the system produces. This becomes increasingly challenging as the complexity of these artifacts grows, leading to the inability to give informed oversight and guidance. Such a system may not only be unsafe, but would no longer be open-ended for human observers, since it would no longer be learnable. As such, any open-ended system we want to build should have the ability to bring human observers along with it—understanding and interpreting these systems is not only a core problem to make them safe, it is also a core problem to make them useful.

One approach would be to try to understand the policy generated by open-ended systems through interpretability. With current approaches this would require a formidable interpretability effort for each domain of interest. However, with the advent of automated interpretability ( Bills et al., 2023 ) , one may hope to build increasingly good explanations of the systems’ behaviors which match the increasing complexity of the open-ended system. This presents an sizeable challenge, as such a system would be a universal explainer ( Deutsch, 2011 ) , by definition.

An alternative approach is to prefer designs for open-ended systems which promote interpretability and explainability, or whose goal is to teach human observers. Already, there are efforts to train systems which directly inform the user of implicit knowledge ( Christiano et al., 2021 ) . One might aim to design systems that at least maintain informed oversight ( Amodei et al., 2016 ; Bowman et al., 2022 ) . This approach may be especially effective if the design of the open-ended system automatically facilitates understanding and control by human users ( Irving et al., 2018 ) .

### 4.3 Humans Guiding AI Creation

Even if we assume that human observers can understand enough of the behavior of an open-ended system to be in a position to give informed feedback, we arrive at the question of how a human designer could meaningfully guide an open-ended system. This challenge goes beyond the difficulties of directing individual RL agents, as not only do open-ended systems often lack well-defined objectives that could be modified, but they are increasingly unpredictable by design. One possibility would be to use humans in the loop to drive open-endedness ( Secretan et al., 2008 ) , a kind of open-endedness from human feedback ( Zhang et al., 2023 ) . A complete solution to this problem not only needs to be directable, but must actively raise unexpected and possibly important artifacts to the user’s attention.

If open-ended systems could be made as directable as individual RL agents, then work defining objectives which preserve controllability ( Hadfield-Menell et al., 2016 ; Hadfield-Menell et al., 2017 ; Carey and Everitt, 2023 ) might be a promising path towards more controllable open-ended systems. However, directing an open-ended system towards any objective effectively while maintaining the open-endedness is an open problem. This problem is not only important for safety, but is important for open-ended systems to be useful. In sufficiently broad domains—such as all of mathematics, all proteins, or all behaviors on a computer—an open-ended system may rabbit-hole into the obscure theorems, useless proteins, or only certain computer applications. Thus, building mechanisms that allow us to direct open-ended systems to not just the safe artifacts, but the interesting and useful artifacts, is a fruitful avenue for collaboration between safety and open-endedness researchers.

### 4.4 Human Society Adapting