AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence — part 3/8

Cleaned-text code points [31004, 46730). Bibliography omitted here; source.md includes it.

Research on this pillar will involve improving the number and difficulty of tasks that can be solved. We must also identify what ‘materials’ the learner (e.g. the RNN) should be made of. For example, the work in the second meta-learning camp mentioned so far was done with standard RNNs, where the only things that can change within the lifetime of the agent (i.e. during inner-loop training) are the activations of the network. The network weights do not change within its lifetime: they instead change only across ‘generations’ (aka iterations) via the outer-loop optimization steps. We recently introduced an approach called differentiable plasticity that enables meta-learning with a different type of neural network [ 112 ] . The weights in these networks change intralife via Hebbian learning (in a fashion similar to the principle that ‘neurons that fire together, wire together’). In this case, each weight consists of two parameters: one is fixed throughout the lifetime of the agent and is directly tuned by the outer-loop optimization algorithm. The other changes as a function of network dynamics, so the network has the ability to store information in its weights, which can improve performance on many tasks vs. traditional RNN meta-learning approaches [ 112 ] . We showed that on some problems that require remembering lots of information over time, RNNs composed of these Hebbian materials outperform LSTMs composed of traditional neurons. While work on Hebbian learning in neural networks has a long history, our recent introduction of the ability to train such networks via gradient descent offers the possibility to harness this alternative to SGD at scale when meta-learning large, deep neural networks.

Another type of learner network includes neuromodulation , wherein the output of one neuron in a network can change the learning rates of other connections in the network [ 40 , 187 , 125 , 161 , 141 , 160 ] . As with Hebbian learning, there is a long history of neuromodulation work without using gradient descent, but we have recently introduced the ability to train networks with neuromodulation in a fully differentiable, end-to-end fashion [ 113 ] . An example type of problem that could in theory be solved with meta-learning and neuromodulation is catastrophic forgetting, a phenomenon wherein neural networks are unable to continuously learn because they learn each newly encountered task by overwriting their knowledge of how to solve all previously learned tasks [ 47 , 111 , 48 ] . Neuromodulation can help because some neurons in the network can detect which task is currently being performed, and those neurons can turn learning on in the part of the network that performs that task and turn learning off everywhere else in the network. That can enable a neural network to learn new skills without cannibalizing old skills. We have shown that meta-learning with neuromodulation can create networks that perfectly solve catastrophic forgetting, meaning they have learned how to learn without forgetting, albeit in small networks and on simple problems [ 40 , 187 ] . Since the first version of this paper was published, we introduced a method called A Neuromodulated Meta-Learning algorithm (ANML), which meta-learns a solution to catastrophic forgetting for deep neural networks at the unprecedented scale of 600 sequential tasks (over 9000 SGD updates) [ 12 ] .

These are just a few examples of how the type of learner used, including the materials it is made of, can have a meaningful effect on the quality and type of solution that results. The choice of materials is a hyperparameter of the meta-learning algorithm. Of course, they too could (and likely ultimately will) be searched for.

Meta-learning is currently computational expensive. It requires training over a distribution of different tasks and often involves differentiating through the entire learning process. Computational limitations, memory limits, and optimization challenges are major hurdles for this work. They currently limit the complexity of tasks that we can train on, how long those tasks can last (e.g. how many time steps), and how many outer-loop (meta) optimization steps we can perform. Research to improve our ability to train such networks and do so on more and harder tasks will be critical to unlocking the power of meta-learning.

One question raised by meta-learning learning algorithms is what the distribution of tasks should be for meta-learning. How narrow should it be? Should it widen over time? As has been pointed out, switching from normal learning to meta-learning changes the burden on the researcher from designing learning algorithms to designing environments [ 56 ] . But why stop there? I argue that we should instead learn the appropriate task distribution that learning agents should meta-learn on. That is AI-GA’s Third Pillar, which I discuss next.

### 2.3 The Third Pillar: Generating effective learning environments and training data

It is unlikely that we will ever know how to directly program a generally intelligent agent. Instead, we will almost certainly have to train a learning agent on the right training data and/or in the right set of training environments. That raises the question as to what the correct training environments are. Even if we knew what the right set of training environments were, we likely would not get away with sampling from them randomly for training (that would be too inefficient, if it worked at all). Instead, we also have to know the right order in which to present the tasks, which may itself be a function of the current learning capabilities of the learning agent. Just as human educational systems have developed curricula for how to teach human learners a variety of subjects, we likely will need to produce effective curricula in order to produce generally intelligent artificial learning agents.

Manually identifying and creating each training environment required and their order would be an extremely challenging, slow, and expensive undertaking. It would be limited by our time, intelligence, and our ability to understand each AI student enough to design optimal learning environments for them. However, we know from the history of machine learning that data is the rocket fuel of success, and we will therefore need lots of it.

The AI-GA philosophy is that we should have learning algorithms learn to automatically generate effective learning environments (i.e. generate their own training data), including creating an effective curriculum that can train up an initially unintelligent agent (the equivalent of a human newborn) into a generally intelligent agent (the equivalent of a human adult, or something smarter). Part of generating a learning environment is generating the reward function that defines success in that environment. One can imagine the generation of labeled training data (images and their labels), virtual worlds that focus on mastering a particular skill (much as a hockey player practices passing, shooting, and skating), and even entire virtual worlds in which agents learn to communicate, coordinate to perform complex tasks, and learn a variety of increasingly difficult concepts, from math and physics to writing, philosophy, and machine learning. One approach might be to create a set of different training environments and expose a learning agent to them in some order (possibly contingent on their past performance). Another approach might be to create a single training environment, which itself internally represents a curriculum for an agent. As on Earth, the difference might ultimately be semantic: in your lifetime have you experienced a set of different learning environments, or a single complex one?

There has been little research into the Third Pillar. It is the least-studied, least-understood, and likely hardest of the three pillars. That also makes it an interesting new frontier of research. Many fundamental breakthroughs await in this nascent field, making it a ripe orchard in which researchers can hunt for impactful discoveries. I next outline a sketch of how we might make progress on this pillar, including some of our early work in this direction. Of course, the most exciting ideas and directions are the ones we cannot envision yet, making the following sketch more of an exercise to kickstart progress rather than a playbook for future research.

Much work has occurred in the fields of computational evolutionary biology, artificial life, and neuroevolution attempting to investigate how to create an open-ended complexity explosion like the one that occurred with living animals on Earth [ 99 , 172 , 13 , 88 , 138 , 17 , 162 , 2 , 183 , 70 , 86 , 85 , 167 , 26 , 20 ] . The dominant theme is to try to create the conditions within a virtual world in which agents evolve and hope a complexity explosion will occur. In virtually all of that work, the environment itself is manually created by human researchers, including the rules by which agents interact if there are multiple agents. The hope is often that interactions between such agents will kickoff a coevolutionary arms race that will produce open-ended innovation. A specific version of this idea, called coevolution, is to have two populations of learning agents competing or cooperating with each other [ 35 , 43 , 79 , 134 , 194 ] . A related idea called self-play is to have an agent learn by playing itself, or an archive of past versions of itself [ 10 , 157 , 156 ] . Significant gains have been made, most famously in learning to play the games Go and Dota [ 157 , 156 ] , validating both the idea of generating effective learning environments (i.e. generating training data) and having agents play against automatically learned curricula. However, these approaches have failed to create anything resembling an open-ended complexity explosion because the non-agent (abiotic) component of the environment they operate in is fixed. For example, while self-play has automatically produced agents that are capable of kicking soccer balls into a goal and playing the games of Go and Dota, no amount of self-play in virtual soccer, Go, or Dota will produce agents that can drive, write poetry, invent flying machines, and conduct science to understand the world they find themselves in. Another approach is to automatically generate different reward functions within an environment [ 56 ] . While promising, like self-play, ultimately such agents are confined to a predefined environment that limits what they can learn.

In my opinion, a more promising direction is to explicitly optimize environments to be effective for learning, instead of hoping we can create environments that create dynamics that lead to coevolutionary arms races that produce generally intelligent learners. Just thinking about environments as something that can be optimized by learning algorithms is interesting and opens many new research directions. One project in this direction is PowerPlay [ 151 , 163 ] . Another vein of research comes from the community attempting to automatically generate content for video games, including training AI agents [ 80 , 154 , 82 ] . The first paper that made me consider directly optimizing environments as an interesting possibility is Brant and Stanley [17] , where an expanding set of novel mazes were generated, each of which could be solved by at least one agent. We followed up on that idea with the Paired Open-Ended Trailblazer (POET) algorithm in Wang et al. [192] , which is discussed in more detail below. However, the question remains of how to properly conduct such optimization. What should the learning algorithm that is optimizing the environment try to make the environment accomplish? What is the reward function (aka loss function or fitness function) for the environment generator? This is one of the key questions for AI-GA research.

One option is the following: define intelligence, then optimize an environment such that learners exposed to that environment are increasingly generally intelligent. This option seems unrealistic for a variety of reasons. The first two problems are that it requires defining what intelligence is and how to measure it. Those problems have eluded scientists and philosophers for centuries. The second is that, even if we did have a reward function for general intelligence, it would almost certainly be highly sparse and deceptive (non-convex). In the parlance of evolutionary biology, these would be considered fitness landscapes that are perfectly flat in many places and extremely rugged in others. The reward function would be sparse because for much of the space there would be no signal or gradient from the reward function. Stanley and Lehman provide a colorful example in the uselessness of, for example, trying to select for human-level intelligence by starting with bacteria and selecting for those that perform better on IQ tests [ 166 ] . Assuming for a second that IQ tests do have something to say about general intelligence, it is clearly unhelpful to try to guide a search for intelligence from bacteria by optimizing for performance on IQ tests because until you already have something with the intelligence of a human child, the test provides no signal to guide search.

The problem of deception is even more pernicious. It describes the situation where the reward function provides not just zero signal, but the wrong signal, causing search algorithms to get trapped on local optima far lower in performance than the global optimum. When searching for the solutions to extremely ambitious problems, even if you have a way to quantifiably measure what success looks like, optimizing for that score is unlikely to produce the correct path through the search space to find the global optimum, and instead usually results in getting trapped on a local optima [ 166 , 92 , 196 , 123 ] . In other words, despite our best efforts, we will likely create deceptive reward functions that do not provide a gradient towards any satisfactory optimum, and will instead lead the search process to get trapped on low-performing local optima.

All of these problems are why basic science is so important. If we went back a few hundred years and sought to create clean energy, it would have been futile to search for technologies that produce fewer and fewer carbon emissions per unit of energy produced. Instead, the first effective clean energy (nuclear power) was discovered by someone passionately thinking about light beams chasing them on a train moving at the speed of light. Similarly, to invent microwaves, one should not have rewarded engineers for making devices that are increasingly fast at heating food, but instead the correct path was to invest in radar technology. If one looks at most major technological innovations, the path of stepping stones that led to them is nearly always circuitous and would have been unpredictable ahead of time [ 166 ] .

Nevertheless, it is useful to recognize that it is possible to generate effective learning tasks with a particular task in mind, such as performing well on an IQ test, having a robot perform a specific skill, etc. While it is unlikely to be fruitful for very ambitious tasks (where the learner is far from possessing the desired skill), it could be practically useful if the amount to be learned is smaller. I call this approach to generating effective learning environments the target-task approach . While it is helpful to research how to create learning algorithms that can produce learners that solve a predefined task of interest—a subject we are working on [ 129 ] —for ambitious goals like producing general AI we likely will need a different approach.