A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents — part 1/4

Cleaned-text code points [0, 15802). Bibliography omitted here; source.md includes it.

# A Definition of Open-Ended Learning for Goal-Conditioned Agents

###### Abstract

A lot of recent machine learning research papers have “open-ended learning” in their title. But very few of them attempt to define what they mean when using the term. Even worse, when looking more closely there seems to be no consensus on what distinguishes open-ended learning from related concepts such as continual learning, lifelong learning or autotelic learning. In this paper, we contribute to fixing this situation. After illustrating the genealogy of the concept and more recent perspectives about what it truly means, we outline that open-ended learning is generally conceived as a composite notion encompassing a set of diverse properties. In contrast with previous approaches, we propose to isolate a key elementary property of open-ended processes, which is to produce elements from time to time (e.g., observations, options, reward functions, and goals), over an infinite horizon, that are considered novel from an observer’s perspective. From there, we build the notion of open-ended learning problems and focus in particular on the subset of open-ended goal-conditioned reinforcement learning problems in which agents can learn a growing repertoire of goal-driven skills. Finally, we highlight the work that remains to be performed to fill the gap between our elementary definition and the more involved notions of open-ended learning that developmental AI researchers may have in mind.

## 1 Introduction

Most existing software agents and robots suffer from insufficient versatility, limiting the potential introduction of these agents and robots in our everyday life Plappert et al., (2018) . In most cases, some expertise is required from a human engineer to design their behavior in anticipation of the situations they may encounter. As a consequence, these agents cannot address novel or unforeseen situations.

An alternative to this specific expertise requirement would be to build an agent which would be ready to address any problem, fulfilling the long-standing dream of Artificial General Intelligence (AGI) Fjelland, (2020) .The corresponding design effort would be extraordinary, so there is a consensus that the way to reach AGI should be through learning. But the AGI perspective does not account for one of the core limitations of human intelligence: some humans can solve problems that others cannot, and vice versa, because humans have a limited lifespan for learning whereas being proficient in all potential tasks would require a potentially infinite amount of learning time.

Taking into account the above core limitation, open-ended learning (OEL) is a framework where autonomous agents face new tasks and learn how to solve them over their lifespan, driven by some intrinsic motivations or external guidance. Doing so, they build their own competence along some developmental trajectory which endows them with some capabilities, but not all capabilities that one may dream of Lungarella et al., (2003) . Their developmental trajectory consists of a curriculum of tasks, and the curriculum learning challenge consists in finding when to learn which tasks so as to better extend the agent’s capabilities.

Though the OEL topic has been around for more than a decade (e.g., see the Intrinsically Motivated Open-ended Learning Workshops and Community 1 1 1 IMOL - Intrinsically Motivated Open-ended Learning Workshops and Community: https://www.imol-community.org/archive/ and Section 2 ), the interest for this framework is growing very fast in the machine learning community, as illustrated by more than 200 machine learning papers using the term in 2022, the emergence of new workshops (e.g. ALOE 2 2 2 Agent Learning in Open-Endedness (ICLR 2022 Workshop): https://sites.google.com/view/aloe2022 ) or even the very related COLLAS top-tier conference 3 3 3 Conference on Lifelong Learning Agents: https://lifelong-ml.cc/ . This growth is simultaneous with the emergence of related topics in Computational Neuroscience (e.g., see Niv, (2019) ; Rmus et al., (2021) ).

In this paper we start by showing that, despite a growing number of research works using the term “open-ended learning”, there have been very few attempts at providing a formal definition for the corresponding concept. This results in a lack of consensus on what OEL truly is and in a risk of confusion with several very related topics such as lifelong learning, continual learning or autotelic learning.

## 2 Defining OEL: Elements from the literature

In this section, we draw from the literature some elements of a definition of OEL. We investigate the genealogy of the notion through its first occurrences in Artificial Intelligence (AI) and Artificial Life (AL) papers and we highlight the conceptual shift that arose in more recent papers. This investigation helps us set the stage for providing a better definition than the few existing ones in a subsequent section.

### 2.1 Genealogy of the notion

Searching for the very first instances of AI or AL works mentioning open-endedness is difficult because OEL is also a concept used in education sciences, with a far longer history. The first reference we could find where the term “open-ended” referred to intelligent machines within a cognitive science context is from Reader, (1969) :

“So far as is known, the range of tasks which the human intellect can master is open-ended (infinite), and therefore an intelligent machine can never be proved to be intelligent by comparing its task performance with that of the human intellect since the process would not terminate. […] The only way a machine can continue to impersonate the open-ended ability of the human intellect is by also being open-ended. […] The human intellect has an open-ended ability because it is capable of learning and the intelligent machine must also be capable of learning.”

This view for which an intelligent machine should learn forever as humans do is clearly reminiscent of Turing’s claim in his seminal paper that the process of making a machine intelligent should follow “the normal teaching of a child” Turing, (1950) .

Much later, an interesting early appearance of open-endedness in a pure AI paper is Meyer and Wilson, (1991) , a text from the founders of the “From animals to animats” community at the time where the corresponding conference emerged. They mention “an open-ended space of network architectures” in a paper dedicated to the artificial evolution of behaviours. The notion of “open-ended evolution” soon became the topic of several papers in the AL community, where the goal was to explain how evolution can generate increasingly complex and capable creatures in a context where most evolutionary algorithms were disappointingly converging to some unsatisfactory fixed-point or cyclic patterns. For instance, Standish, (2003) mentions such discussions as central to the AL conferences in the early 2000’s, and open-ended evolution is still an active topic in the AL community, see for example Taylor et al., (2016) ; Packard et al., (2019) . To extract elements of a definition from this line of research, we retain the definition from Standish, (2003) , where we find:

“The issue of open-ended evolution can be summed up by asking under what conditions will an evolutionary system continue to produce novel forms.”

Importantly, we see that the focus is on the conditions on the environment rather than on the evolution process itself (e.g., see also Soros and Stanley, (2014) , for similar conclusions) and that the expected outcome is the production of novelty.

One perspective on the conditions for open-ended evolution or OEL is thus to consider an rich enough environment which provides an open-ended sequence of problems. A similar idea of generating sequences of problems dates back to Schmidhuber, (2013) where the author defines “Open-Ended PowerPlay”. The goal of the approach is the “Invention of new problems” Srivastava et al., (2012) to continually challenge a learning agent. This is much more closely related to what the machine learning community now means with OEL, but note that the focus is on generating problems in the environment rather than on the necessary mechanisms to solve them. This work is often cited in the machine learning works interested in the automated generation of curricula, and particularly in those which focus on generating a sequence of challenging problems in some environment rather than on solving them. For instance, this is the case of the poet algorithm Wang et al., (2019) ; Wang et al., (2020) where the open-ended invention of challenging environments is conceptualized as separated from the agent:

“How can progress in machine learning and reinforcement learning be automated to generate its own never-ending curriculum of challenges without human intervention? The recent emergence of quality diversity (QD) algorithms offers a glimpse of the potential for such continual open-ended invention. […] The Paired Open-Ended Trailblazer (POET) algorithm introduced in this paper combines these principles to produce a practical approach to generating an endless progression of diverse and increasingly challenging environments while at the same time explicitly optimizing their solutions.”

This perspective about open-ended invention is now widely adopted in the machine learning community, and can be seen as one of the most prevalent understandings of what OEL truly means, for example see Dharna et al., (2020) ; Bontrager and Togelius, (2021) ; Kepes et al., (2022) ; Dharna et al., (2022) . This view has been recently strengthened and further broadened with the proposal of a theoretical perspective claiming that the current challenges of machine learning, including the production of general artificial intelligence, can be addressed shifting focus from learning algorithms to generalized exploration . This allows both reinforcement learning (RL) and supervised learning (SL) algorithms to automatically generated new data Jiang et al., (2023) :

“Importantly, generalized exploration is a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.”

Within this framework, the authors propose that promising approaches involve processes for the open-ended generation of novel data that maximize the agent’s learning potential while also leading it to explore areas of the problem space that are at the same time novel and grounded (i.e., relevant for the use of the system). This approach might for example involve the open-ended automatic generation of novel and relevant tasks in RL, for example based on parameterised environments, and novel and relevant new data in SL, for example based on the generation of synthetic training data.

A different perspective, pioneered by Weng et al., (2000) ; Weng et al., (2001) , shifts the focus of the “openness” from the environment to the agent:

“What is autonomous mental development? With time, a brain-like natural or an artificial embodied system, under the control of its intrinsic developmental program (coded in the genes or artificially designed) develops mental capabilities through autonomous real-time interactions with its environments (including its own internal environment and components) by using its own sensors and effectors. Traditionally, a machine is not autonomous when it develops its skills, but a human is autonomous throughout its lifelong mental development. […] A mental developmental process is also an open-ended cumulative process”

In Weng’s perspective, the autonomous progressive learning of the agent should be guided by an internal developmental program . This program is characterised by various elements that are relevant for OEL: “ Sensor-specific and effector-specific; task-nonspecific; tasks unknown at programming time; generate representation automatically; animal-like online learning; open-ended learning of more new tasks. ” This approach led to the birth of the new field of “Developmental Robotics” and the related “Development and Learning” interdisciplinary community and conference Lungarella et al., (2003) .

In this context, other seminal works identified the key computational ingredients that allow the actual implementation of the developmental program, namely intrinsic motivations Barto et al., (2004) ; Singh et al., (2004) . This is also the case of Prince et al., (2005) who introduces a notion of ongoing emergence which is explicitly related to open-ended learning and intrinsic motivations. These works managed to ignite a research effort directed to systematically draw concepts on different intrinsic motivations from psychology, and translate them into specific machine-learning algorithms Barto et al., (2004) :

“Psychologists distinguish between extrinsic motivation, which means being moved to do something because of some specific rewarding outcome, and intrinsic motivation, which refers to being moved to do something because it is inherently enjoyable. Intrinsic motivation leads organisms to engage in exploration, play, and other behavior driven by curiosity in the absence of explicit reward. […] Although these arguments are compelling, developmental approaches to artificial agent design have been slow to penetrate the mainstream of the machine learning community.”

Intrinsic motivations can thus support the possibly-open autonomous acquisition of knowledge and skills:

“According to this approach, an agent undergoes an extended developmental period during which collections of reusable skills are autonomously learned that will be useful for a wide range of later challenges.”

These works followed previous pioneering works on specific intrinsic motivation mechanisms which initially were overlooked Schmidhuber, (1990) ; Schmidhuber, (1991) . The renewed research effort led to connect educational science and developmental psychology concepts of OEL in children to the notion of intrinsic motivations in machine learning and robots Kaplan and Oudeyer, (2007) . In addition, it led to distinguish and formalise different classes of intrinsic motivations, in particular related to prediction , novelty , and competence Oudeyer and Kaplan, (2007) ; Barto et al., (2013) . Intrinsic motivations can thus form the ‘motivational engine’ guiding autonomous open-ended learning in organisms and robots Baldassarre and Mirolli, (2013) .

Another critical step for the definition of OEL agents was the investigation of the relation between intrinsic motivations and evolution, which we have seen to be two key areas where open-endedness can manifest. In general, evolution is proposed to lead to the emergence of both extrinsic motivations (those serving typical biological needs such as hunger and sex) and intrinsic motivations as both can guide the acquisition of behaviour adapted to the environmental conditions Schembri et al., (2007) ; Singh et al., (2010) . In organisms, both types of motivations are supported by brain mechanisms responding to different general principles Baldassarre, (2011) . In particular, extrinsic motivations support the acquisition of material resources having a direct adaptive advantage, and to this purpose monitor visceral body states. Instead, intrinsic motivations, emerged later in evolution, support the acquisition of knowledge and skills which are only later useful to increase fitness, and thus monitor the brain information gain . For this reason, only intrinsic motivations, if suitably translated into algorithms, have the “ potential to produce open-ended learning machines and robots ” Baldassarre, (2011) . This idea led to the start of the IMOL Workshop series mentioned above.