A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents — part 3/4

Cleaned-text code points [31593, 47245). Bibliography omitted here; source.md includes it.

The above definitions are very general, and capture a large set of OEL problems, among which many are not satisfactory models of developmental processes. To go further towards modelling open-ended learning from a developmental perspective, we must work on distinguishing trivial cases of open-ended GCRL problems from more interesting ones.

### 3.4 First-order and second-order open-ended GCRL

Given the above definitions, there are two simplistic cases of open-ended GCRL problems.

Case 1: The goal space is discrete, but infinite. A simple example of this case is when the goal is to count from 1 1 to N N , where N N is a natural number. Generating as goals an infinite sequence of growing values of N N is enough for that problem to be open-ended.

Case 2: The goal space is continuous. A simple example of this case is asking a physical agent to travel at a given speed, where the speed is a real number. Targeting higher and higher traveling speeds that converge asymptotically to the highest possible traveling speed of the agent is a problem that already verifies the open-endedness property.

These two simplistic examples qualify as open-ended GCRL problems for naive observers who do not require a strong notion of novelty, but they do not account for the developmental perspective AI authors generally have in mind when they try to define OEL agents. We see two approaches to address this issue.

A first approach to avoid trivial OEL problems consists in putting constraints on what an observer may consider as a “new token” or not. Demanding observers may require new tokens to be “substantially new” or “interestingly new”.

A second approach, which is not orthogonal to the previous one, consists in considering that a developmental process implies that goal tokens are sampled from more and more interesting spaces . With this second approach, we may consider two classes of problems, following a strategy inspired from the work of Etcheverry et al., (2021) . The first class involves problems as the usually considered ones:

By contrast, to define the second class, we first need to define a variety of goal spaces 𝛀 j ∈ 𝛀 \mathbf{\Omega}_{j}\in\mathbf{\Omega} where 𝛀 \mathbf{\Omega} is the set of all possible goal spaces. Then we can introduce the idea of goals generated from different spaces.

Below for simplicity we consider the case of one goal space, and hence 𝐆 \mathbf{G} instead of 𝐆 j \mathbf{G}_{j} , but our definitions could be extended to the case of the generation of multiple goal spaces.

A different way to express the same thing is that in first-order GCRL problems tokens of the open-ended process are goals from a single goal space, whereas in the second-order case tokens of the open-ended processes are goal spaces and goals.

Note that with the above definition, second-order open-ended GCRL problems can still generate a trivial diversity of goal spaces, so again we have to add further conditions so that the generated goals and goal spaces ensure an interesting developmental trajectory.

## 4 Combining open-ended learning with other properties

We have proposed a definition for the elementary property of an OEL problem. Now it is time to ask whether and how we can combine this property with other, complementary ones. In particular, we would like to build an equivalent of the LOLA framework of Romero et al., 2023b () , where LOLA stands for Lifelong Open-Ended Learning Autonomy. Below we do so by first building in Section 4.1 on the fully-formalized notion of continual learning proposed in Abel et al., (2023) . Then we stress the need for accounting for the lifelong learning property. Finally, we also expand in Section 4.3 on the fact that the autonomous learning property is too strict a requirement, as one may consider agents that are both autotelic and teachable, i.e. capable of adopting social partner’s goals.

### 4.1 Goal-conditioned continual reinforcement learning

Given our definitions above, we can now connect our definition of OEL to the definition of continual learning from Abel et al., (2023) , investigating solutions to the continual and open-ended GCRL problem. An RL agent can then be defined as follows:

An goal-conditioned agent can then be defined as follows:

As classically done in the GCRL literature Campos et al., (2020) , we distinguish ‘behavioral goals”, which are used to condition a policy, and “achieved goals”, which are the goals fulfilled by the behavior generated by the policy.

In Abel et al., (2023) the authors define “continual problems” as problems where the optimal agents never stop learning (i.e., they continuously change their policies). A weakness of this definition is that the continual learning RL problem is defined in terms of the type of solutions (agents) to use to solve them. From that perspective, the examples given by the authors in their work all involve non-stationary environments, and thus the non-stationary property should actually be used to define their problems. Given this feature of such environments, it should follow that the optimal agents (solutions) solving them have to keep changing their policies. Furthermore, one should also consider that if the environment continues to change, but after some time it repeats the same configurations, then the best agents might actually learn all the solutions for all environment configurations, and then stop learning. This unless it is assumed that the agents forget previous solutions when the environment changes, an assumption that the authors seem to implicitly make. Only problems that continue to propose novel challenges to the agents, as the true OEL problems considered here, actually require agents that keep changing: these agents have to keep changing not because they forget previous solutions, but because they need to indefinitely acquire new knowledge. This is a signature feature of OEL agents and also of any solution to truly OEL problems.

We can define open-ended RL agents (on the side of solutions) as follows:

This definition is admittedly very general. We can be more specific when defining open-ended GCRL agents:

From this definition, it should be obvious that an open-ended GCRL agent defined as above is the solution to a continual RL problem as defined by Abel et al., (2023) . Indeed, to be optimal, these agents should continuously learn in order to solve the continuously generated novel goals.

By contrast, if we remove the condition on OEL and if the environment is stationary, we may account for convergence to an universally efficient agent that has learned enough to become capable, after some time, to achieve any forthcoming goal in a zero-shot manner.

Given the considerations above, we would like the agents not to (completely) forget the tasks it previously achieved when it addresses new ones. This property corresponds to lifelong learning. Thus we should also include the lifelong learning property into our framework.

### 4.2 Lifelong open-ended learning

Though the names suggest they are similar, the lifelong learning property and the continual learning property are not exactly the same. Indeed, the authors who use the first term generally have in mind the capability to overcome catastrophic forgetting (e.g., see Parisi et al., (2019) ) whereas the authors of the continual learning definition mentioned above do not even mention this aspect Abel et al., (2023) .

The lifelong learning property captures the idea of a growing repertoire of skills, although it mainly focuses on avoiding to forget the previous skills when learning new ones. Thus, characterizing such a property requires being more specific on the learning processes from the side of the agent, hence the definition must be from the side of solutions.

### 4.3 The origin of goals: extrinsic, autotelic, and teachable open-ended learning agents

So far, we have not specified where the goals come from. In this respect, there are three main possibilities.

First, the goals may come from the environment itself. In this case we say they are extrinsic. This corresponds to the GoalEnv category of environments in the standard gym interface for RL agents. In this case, the environment provides the agent with a state, a goal, and a reward. Extrinsic OEL agents are thus as follows.

An important case of extrinsic OEL problem is one where the environment generates an open-ended sequence of goals that maximise the speed and quality of the learning process of the agents Jiang et al., (2023) . Extrinsic OEL agents, however, are not the main focus here.

The second case corresponds to agents who set and learn to pursue their own tasks or goals autonomously. In this case we say the goals generated by the agents are intrinsic. An interesting case of this is when the agents autonomously set their own goals, in which case we have autotelic agents.

To discover new goals, autotelic OEL agents generally expand the set of behavioral goals they consider by sampling from the set of goals they have already achieved Campos et al., (2020) , which defines a curriculum over goals. More precisely, the most efficient curricula consist in sampling at the frontier of the domain of currently achieved goals Seepanomwan et al., (2017) ; Pitis et al., (2020) ; Castanet et al., (2022) . This interaction should ensure that the agent achieves enough new goals to continue expanding its behavioral goal domain.

From a slightly different perspective, for an autotelic agent facing a second-order open-ended GCRL problem where it generates goal spaces from its own experience, the environment has to be rich enough to foster this open-ended goal space generation process.

A third case involves agents that autonomously generate goals but whose goal generator can be influenced by interactions with social partners. By giving demonstrations, descriptions, instructions or feedback, social partners can suggest to these agents to adopt some goals rather than others. Following Sigaud et al., (2023) , focusing on autotelic agents, we say that such agents are teachable.

It should be clear that ensuring the OEL property is more difficult for purely autonomous agents than for teachable agents, as the former must discover new goals or goal spaces without external support, whereas the latter can benefit from social interactions to do so. That is, being teachable may relax lot of the requirements on the goal discovery capabilities of agents.

Note that some agents can be teachable without autonomously generating goals, meaning that they fully depend on external agents for goal generation. Note also that autonomously generating goals is not a sufficient condition for being an OEL agent, as an autonomous agent may always sample from the same limited set of goals.

Also, it is not a sufficient condition to be a continual learner according to the definition of Abel et al., (2023) , as the behavior may become stationary.

This line of thoughts suggests studying in the future the different properties of the types of agents developmental AI researchers have in mind, and the way they interact with each other.

## 5 Evaluating open-ended learning agents

The OEL framework poses two important challenges: (1) How can we make sure that a learning agent is an OEL agent? (2) How can we compare the performance of two OEL agents?

### 5.1 Assessing the open-ended learning property

A key difficulty with OEL agents is that, based on the definition given here, it is not easy to evaluate in practice if an agent is showing open-ended learning or not. Indeed, such an agent should address new goals forever, so we should wait forever to determine if it succeeds in doing so. Of course, this is not doable in practice. A related viewpoint is that the definition does not take into account the fact that concrete agents generally have a finite lifespan.

A first approach to address this problem was proposed by Cartoni and Baldassarre, (2018) , where the authors attempt to define open-ended learning problems in a concrete way. The agent’s “life” is divided into two phases. In a first, very long “intrinsic phase”, the agent is set in a given environment and receives no learning guidance (e.g., reward functions or goals) nor any hardwired knowledge. In a second “extrinsic phase”, the agent is tested with a set of tasks (e.g., goals to achieve) that are “randomly drawn” from the same environment, in the sense that these tasks are representative of all possible tasks that might be generated in that environment. The key idea is that, since in the intrinsic phase the agent does not know the tasks it will have to solve in the extrinsic phase, it will have to possibly rely on intrinsic motivations to maximise its knowledge and skill acquisition and thus have a high performance with tasks in the extrinsic phase. Importantly, this performance represents a measure of the knowledge-gain capacity of the OEL processes in the intrinsic phase. This approach is definitely practical, but we have to be aware that it does not truly evaluate the OEL property in itself.

Another possible approach is more closely related to the definition of OEL given here. It consists in measuring the rate of novel goals that the agent discovers and/or learns to master throughout its life, and then extrapolating from its dynamics if the agent would continue to discover new goals forever if given infinite time. In particular, if the number of seen goals asymptotically converges to a constant, the agent is not an OEL agent. Otherwise, if the curve shows a logarithmic behavior, a linear behavior with positive slope, or even better an increasing slope, then the agent is an OEL agent. The practical application of this approach requires the possibility of marking goals as novel in order to compute the goal-generation rate, an operation which would pose the problems previously discussed.

### 5.2 Comparing open-ended learning agents

Besides, comparing the goal-reaching capabilities of purely autotelic agents is particularly difficult since different agents may follow a completely different curriculum, thus there might be no common ground on which to compare them. However, the method proposed by Cartoni and Baldassarre, (2018) was designed to also address this problem. Indeed, the tasks used in the extrinsic phase to measure the agent performance should be representative and cover the whole space of problems that might be generated in the environment. Thus, the average performance of different agents on these tasks should be representative of the effectiveness of their OEL processes independently of their specific curricula which generated specific task-trajectories across the task space.

In the case of teachable agents, the problem can be mitigated as one may determine a common social interaction policy for an interacting social partner and measure to which extent various teachable agents manage to learn the goals the social partner is trying to suggest.

## 6 Discussion and Conclusion

In this paper we have investigated the genealogy of the notion of open-ended learning and isolated a core property that may be general enough to account for a large number of open-ended learning phenomena. From there, we have focused on a goal-conditioned reinforcement learning perspective, and proposed a framework from which open-ended GCRL problems and agents could be defined.

In addition, we have outlined the need for combining our core property with other properties that would help developmental IA authors to capture the idea of a growing repertoire of capabilities or skills, which is so important in the OEL process of children.