A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents — part 2/4

Cleaned-text code points [15802, 31593). Bibliography omitted here; source.md includes it.

Aside from intrinsic motivations, research has highlighted that a second key element can foster OEL, namely goals . Goals are internal representations of desired states of the environment that can guide the agent’s action. Goals represent a key element in classic symbolic AI systems, for example to support planning Russell and Norvig, (2016) , but have been less employed in ML systems. A main exception is the RL option framework, where goals might be associated to the termination condition of options Sutton et al., 1999a () , or goal-conditioned policies , where RL action policies are goal indexed Baldassarre, (2001) ; Liu et al., (2022) ; Colas et al., (2022) . Within this context, the potential relevance of goals for OEL was highlighted in Barto et al., (2004) , where the learning of the agent was guided by the autonomous generation of options (hence goals) when a “salient event” was encountered. Later, self-generated goals have been indicated to be a fundamental means usable by OEL agents to autonomously generate curricula driven by competence-based intrinsic motivations Santucci et al., (2012) ; Santucci et al., (2016) . The EU-funded project GOAL-Robots 4 4 4 GOAL-Robots - Goal-based Open-ended leaning Autonomous Robots: https://www.goal-robots.eu/ pivoted on this idea to build GOAL Agents , where ‘GOAL’ stands for ‘Goal-based Open-ended Autonomous Learning’. The capacity of OEL agents to autonomously generate, discover or select goals has been recently called autotelic Colas et al., (2022) , and plays an important role in the framework proposed here.

We summarize the above overview of the genealogy of the notion of open-endedness in the AI/AL literature in Figure 1 . From the historical investigation above, we can retain that the open-endedness property can be bound to the environment or to the agent. In the latter case, open-ended learning relies on intrinsic motivations able to drive the autonomous unbounded learning of increasingly complex and new knowledge and skills. In this case, goals have emerged as a key means to support open-ended learning.

### 2.2 Current perspectives

Though all the perspectives considered in the previous section still correspond to active research lines, we observe a renewal of the questions related to open-endedness in the more recent literature. For instance, though they do not define what they mean by OEL, Fan et al., (2022) have OEL environments, tasks, goals, and task suites, the main idea being to have a “wide variety” of such elements.

A relevant recent paper from the “Open-ended learning group” at DeepMind seems to adopt the above perspective. Though the term “open-ended” hardly appears apart from the title, introduction, and conclusion, we can find in Stooke et al., (2021) the following claim:

“We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours.”

One can see that an OEL process should provide a goal distribution shift, or curriculum, from which the agent can continually learn to achieve new goals. As outlined in Section 4 , this may be confused with continual or lifelong learning. The same is true of a more recent article from the same company Adaptive Agent Team, (2023) .

The work Doncieux et al., (2018) present another perspective that is relevant for our attempt to build a stronger definition of OEL. The authors define an OEL process as a process building appropriate MDPs depending on the current situation and the external reward function. This perspective has two major features. First, it is the only paper we found that attempts to provide an explicit definition of OEL. Second, it departs from all other existing frameworks by considering that each task should come with its own state and action spaces. This is in sharp contrast with all the multitask RL frameworks we know about, where a unique policy or set of policies with the same input-output format are used, implying that the state and action spaces are the same for all tasks. As outlined in a previous work tackling the issue of representations within the RL framework Konidaris, (2019) , having a necessary and sufficient representation specific to each task corresponds to the necessity of state and action abstraction : it is easier to solve a task if we abstract away all sensory information and potential actions that are not relevant for task achievement.

The same idea can be found in a recent trend in computational neuroscience research which focuses on the executive functions that help us determine an adequate representation of the task we want to solve Niv, (2019) ; Rmus et al., (2021) . In Doncieux et al., (2018) , the capability to abstract away adequate state and action spaces for each task is based on a more general representational redescription capability inspired by Karmiloff-Smith, (1994) .

However, the definition of OEL provided in Doncieux et al., (2018) is not entirely satisfactory. A key point is that the framework considers that, when an agent addresses an open-ended sequence of tasks, each task comes with its own externally defined reward function. But if the tasks cannot be anticipated, the corresponding reward functions cannot be provided in advance by the agent designers. Moreover, and most importantly, although it is an interesting condition, it is in no way obvious why abstraction and representational redescription capabilities would be a necessary condition for exhibiting OEL capabilities. Finally, in Doncieux et al., (2018) goals and motivations are built from the reward. Instead, intrinsic motivations frameworks do the contrary by deriving goals from intrinsic motivations and rewards from intrinsic motivations and goal achievement Barto et al., (2004) ; Santucci et al., (2016) ; Colas et al., (2022) .

This final remark can be generalized to many works cited above. Indeed, a key issue with previous attempts at defining OEL is that they all try to bind the concept to a large conjunction of properties, such as being autonomous, being endowed with intrinsic motivations or being capable of state abstraction. Relying on such conjunctions does not help pinpointing what is specific to OEL, which we believe is a key requirement for faster progress in designing agents capable of this property.

The work that best responds to this remark is Romero et al., 2023b () , where the authors put forward the Lifelong Open-Ended Learning Autonomy (LOLA) framework. We think that by isolating OEL from lifelong learning and autonomous learning, the LOLA framework of Romero et al., 2023b () takes a step in the right direction. However, for the OEL component of their framework they refer to Doncieux et al., (2018) without providing a proper definition, though the referred paper does not provide a satisfactory OEL definition.

Thus the key contribution in Section 3 consists in providing such a core definition, on which we hope to build a solid OEL framework. Then in Section 4 , we combine the just defined OEL property with other properties mentioned in the literature. In particular, we show how the property can be combined with lifelong learning and that it removes the need for the continual RL framework of Abel et al., (2023) . In addition, we discuss the fact that an OEL agent might be autotelic or it could be teachable (that is, it receives goals from the environment). Finally, in Section 6 we discuss the limits of our proposal. We summarise all these elements and their relations in Figure 2 , which expresses the relationships between the definitions we propose.

## 3 Open-ended learning: a definition

Our investigation of the literature about the definition of OEL in AI and ALife has revealed that some authors were trying to define OEL problems whereas others were focusing on defining OEL solutions . The coexistence of both perspectives is not surprising as it is unclear whether the conditions for the emergence of OEL are more on the side of the environment – which defines the problem the agent should solve – or on the side of the agent – where a class of agents defines the solution to the OEL problem. In this paper, we want to define OEL problems in the most general way, so we consider both perspectives. To clearly distinguish definitions about the OEL problem from those about OEL solutions, we highlight the former in blue and the latter in red.

To introduce our framework, we first position it with respect to the definition of a continual reinforcement learning problem proposed in Abel et al., (2023) . The authors formalize a class of problems where, to be optimal, an agent should never stop learning. In more details, and without strictly following their terminology, they define a learning agent as a trajectory in policy space generated by a history of interactions with the environment. In their framework, a set of agents corresponds to the set of policies that are generated by the corresponding interaction histories and learning algorithms. A triplet formed by an environment, a performance measure, and a set of such agents defines a continual RL problem if the best performing agents in that set never converge to a single policy. That is, in a continual RL problem the best agents must continually learn. Typical problems that require these types of solutions involve non-stationary scenarios presenting ever changing features to which the agents need to continuously adapt Kauvar et al., (2013) ; Romero et al., 2023a () .

These definitions are helpful, but a few remarks need to be made. First, in the given definition a continual RL agent may switch between a finite set of policies forever without generating anything new. By contrast, in Section 3.1 we propose the idea that a key element of OEL is the never ending learning of new knowledge, building on a notion of open-ended process.

Second, the notion of continual learning given in Abel et al., (2023) does not imply the idea of a growing set of goals that an agent may face. To account for such an idea, in Section 3.3 we plug our notion of open-ended process into the goal-conditioned reinforcement learning (GCRL) framework, where the open-ended process plays the role of generating goals.

### 3.1 Open-ended process

The key building block of our proposal is a general notion of open-ended process . To arrive at this notion we first define a process.

There is no constraint on what a token is and no time limit, and the same token may appear several times. Note that a process may generate several tokens at the same time.

To define an open-ended process , we introduce the idea that the process generates some novelty through time. But we would like to stress that novelty is a property relative to an observer: some observers will consider a token as new if it is different enough from all previous tokens according to their own perspective, but other observers may consider that the same token is too similar to previous tokens to be considered distinct. The notion of observer proposed here is broad as it might be a non-transparent subjective evaluator (e.g., a human user) or alternatively an objective mechanism that incorporates some measure of novelty (e.g., an algorithm). As a consequence, our key definition is the following:

Differently from the definition of continual learning of Abel et al., (2023) , this definition captures the idea that something new must happen at least from time to time. Moreover, the definition implies that an open-ended process works over an infinite time horizon.

A property of open-endedness defined this way is that any open-ended process necessarily generates an infinite number of distinct tokens. As a simple mathematical proof, if | 𝚃𝚘𝚔 t | |{\tt{Tok}}_{t}| denotes the number of distinct tokens generated up to time t t , by definition there exists a time t ′ > t t^{\prime}>t such that | 𝚃𝚘𝚔 t ′ | ≥ | 𝚃𝚘𝚔 t | + 1 |{\tt{Tok}}_{t^{\prime}}|\geq{|\tt{Tok}}_{t}|+1 . This property can be applied an arbitrary number of times, so that for any N = 1 , 2 , … N=1,2,... there exists a t ′ > t t^{\prime}>t such that | 𝚃𝚘𝚔 t ′ | ≥ | 𝚃𝚘𝚔 t | + N {|\tt{Tok}}_{t^{\prime}}|\geq|{\tt{Tok}}_{t}|+N , proving that the number of distinct tokens generated throughout the process is unbounded.

The above definition is very general and can capture many examples of open-endedness. For instance, in open-ended evolution, the tokens are agent phenotypes. In standard RL (i.e., non goal-conditioned), an OEL agent is an agent whose behavior generation process is open-ended. There might be several ways to implement this condition within our definition, for example the tokens might be reward functions, Markov decision processes, policies or trajectories. We now first give a definition of such an OEL framework based on standard RL, and then we focus on GCRL and open-ended goal generation.

### 3.2 Open-ended RL

To define RL problems we borrow the formalism of the problem definitions from Abel et al., (2023) that we introduced in Section 3 . This formalism is used here because it has a broader scope than the more frequently used Markov Decision Processes. However, unlike in Abel et al., (2023) where the distinction between problems and solutions is not considered, our definitions insist on this distinction. As in Abel et al., (2023) , we define an agent-environment “interface” as the ( 𝐎 × 𝐀 ) (\mathbf{O}\times\mathbf{A}) pair where 𝐎 \mathbf{O} is the space of observations and 𝐀 \mathbf{A} is the space of actions resulting from interactions of this agent with this environment. We then define the set of possible histories of the agent-environment interactions as 𝐇 = ⋃ t = 0 ∞ ( 𝐎 × 𝐀 ) t \mathbf{H}=\bigcup_{t=0}^{\infty}(\mathbf{O}\times\mathbf{A})^{t} . A single history of such a set, h t ∈ 𝐇 𝐭 ⊂ 𝐇 h_{t}\in\mathbf{H_{t}}\subset\mathbf{H} , involves a sequence of length t t of observation-action pairs.

From these definitions, we define an RL problem as follows.

To build an open-ended RL problem from the RL problem as defined above, we need to consider at least a sequence of such RL problems. Such problems can be organized purely sequentially, but also hierarchically, when an agent is building complex problem representations on top of simpler representations. In the case of a hierarchical organization, one may call upon the options framework Sutton et al., 1999b () .

Under this hierarchical perspective, the tokens generated by an open-ended process could be RL problems themselves, that is an open-ended RL problem might generate an open-ended sequence of RL subproblems. Based on these elements, we can give the following general and potentially recursive definition of an open-ended RL problem.

### 3.3 Open-ended GCRL

From these definitions, we proceed to define GCRL problems. That is, we introduce into RL problems a notion of goal and goal-conditioned reward function (see Colas et al., (2022) ). We characterize goals as elements of an arbitrary goal space 𝐆 \mathbf{G} but we do not specify where they come from. For example, they could be generated by the environment, another agent or the agent itself.

Similarly, using goals we can apply our general definition of an open-ended process to a goal generation process, and get the following definition:

As noted above, for goal-generation processes whose set of distinct goals for the observer is finite, the problem cannot be open-ended. However, in practice agents generally have a finite lifespan. Thus a process generating a finite set of goals might be seen as giving rise to an open-ended goal generation process even if the infinite horizon condition is not met. We come back to this issue in Section 5 .