AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence — part 5/8

Cleaned-text code points [61155, 76186). Bibliography omitted here; source.md includes it.

POET exhibits many desirable properties. First, it creates an expanding set of diverse environmental challenges, each of which can be solved by the current population of agents to some degree. In most cases, agents get better at solving their particular challenge, meaning they are gaining skills. Additionally, agents can goal-switch between challenges. We observe that in many cases the current agent in an environment is stuck on a local optima, but eventually an agent from another environment transfers in, ultimately leading to much higher performance. One can imagine that POET could create wildly different species of solutions within one run, but our initial demonstration domain was too simple to produce tremendous diversity because the environmental encoding only allowed the production of environments with different amounts of landscape ruggedness and different sizes of gaps and obstacles. However, if POET was combined with a flexible way to encode environments, one of its runs could create water worlds, desert worlds, and mountain worlds, each with its own types of agents customized to perform well in those worlds. Such specialization is especially easy to imagine if the bodies of virtual robots are simultaneously optimized, which is a subfield with a long history [ 158 , 21 , 22 , 67 , 6 , 57 ] . Presumably goal-switching would happen much more often within certain types of water worlds, less so amongst more different types of water worlds, and never between water worlds and mountain worlds (at least, not directly). Thus, such algorithms naturally would hedge their bets on the best path to creating any type of interesting solution, including general AI, by simultaneously pursuing a diversity of high-quality solutions. In effect, POET creates multiple, simultaneous, overlapping curricula to learn an ever-expanding set of skills. Many of the curricula might be ineffective dead ends, but as long as some of them are fruitful, the algorithm can succeed.

#### 2.3.4 More expressive environment encodings

One drawback to the POET work is that it assumes a specific type of world, such as obstacle courses in a particular physics simulator, and a specific way of parameterizing that type of world (e.g. a vector with numbers defining the width of the gaps, the height of the stumps, etc.). But ultimately such a strategy is confined to only be able to produce environmental challenges allowable by that parameterization of that physics simulator. For example, our obstacle course simulator does not allow the creation of many types of challenges, such as doors, tire swings, swimming worlds, playing chess, or needing to learn different types of mathematics. It also cannot generate sound, smell, and other sensory modalities. In short, the original POET did not have a sufficiently expressive environmental encoding.

In our newest work on the Third Pillar [ 129 ] , we are creating algorithms that can generate effective learning environments with a fully expressive environment encoding , meaning one that can generate all possible learning environments. Recalling the concept of computers that are Turing Complete, we might call an environmental encoding that can create any possible learning environment Darwin Complete. The name reflects that the encoding enables the creation of all of the environments that made Darwinian evolution successful (and many more).

To create a Darwin Complete environment encoding, we generate environments via a deep neural network (that can optionally be recurrent). We call these DNNs Generative Teaching Networks [ 129 ] because we explicitly train them to be optimal teachers for another student deep neural network that learns on the data (or in the environment) the GTNs create. Because recurrent neural networks are Turing Complete [ 155 ] ), GTNs can produce any type of data, and thus could in theory create everything from image classification tasks to entire virtual 3D worlds complete with sound, touch, and smell. They could also create opponent agents to play against, and thus are strictly more expressive than self-play techniques (although they are likely much harder to optimize). That said, it may be easier to have trained agents interact within GTN-produced worlds, rather than forcing the GTN to learn to produce the policies of an arbitrary number of agents. In that option, there would be an entire population of agents in each generated environment. That would potentially make it easier to create multi-agent interactions, including the emergence of language and culture (including the catalyzing force of agents learning via cultural transmission and the cultural ratchet, meaning the amassing of increasing amounts of knowledge over time that each agent can learn from [ 184 ] ).

Because DNNs are differentiable, we can have GTNs produce training data or environments for a learner DNN, then test the trained learner on a target task, and differentiate back through the entire learning process to update the parameters of the GTN to improve its ability to produce effective training data. Because the GTN is trying to make the learner good on a predefined task, this use case is an example of the “target-task” version of generating effective learning environments. In our experiments so far, we have shown that a GTN can be trained to produce data that enables a student network to learn to classify MNIST. Interestingly, the GTN is not constrained to produce realistic-looking training data (e.g. images a human would recognize as handwritten digits). In fact, we found that the data it generates look completely unrecognizable and alien, yet still the student network learns to recognize real handwritten digits. Moreover, the student network learns four times faster than when training on real MNIST training data! The result that unrecognizable images are meaningful to DNNs is reminiscent of the realization that deep neural networks are easily fooled and will declare that unrecognizable images are everyday objects (e.g. guitars and starfish) with near certainty [ 120 ] . It is an interesting, open question as to whether natural brains, including those of humans, could be rapidly trained to perform any skill via such alien data (a la the novel Snow Crash). Interestingly, researchers recently showed that they could generate fooling images (‘super-stimuli,’ in the parlance of biologists) for the neurons of real monkeys [ 133 ] ; they did so by generating images via the DGN-AM technique [ 121 ] to synthetically generate data that activate neurons in a live monkey’s brain. Many of the synthesized images were unrecognizable, yet activated the neurons in the brains of monkeys more than any of real images from the natural world.

As mentioned above, there are two options for generating effective learning environments. The previous paragraph describes how we have already experimented with GTNs in the target-task paradigm. Intriguingly, GTNs (or any sufficiently expressive environment generator) can also be used in the open-ended paradigm for generating effective learning environments. This is a promising path to making progress on AI-GA’s Third Pillar. The idea is to harness GTNs to produce an expanding set of learning challenges for agents. For example, one could pair the GTN encoding with POET to create an expanding set of GTNs that each specify an environment. Alternately, a single, powerful GTN could be created that is conditioned on a noise vector (and possibly an agent’s past experience and learning progress) to endlessly produce new, effective learning challenges. Of course, just because the GTN can do that in theory does not mean it will be easy to make it work in practice, and much work remains to accomplish that lofty goal. Additionally, GTNs are just one approach to generating learning environments. They may prove too expressive and thus make searching their vast search space intractable. We might want to bake in more prior knowledge by constraining all environments to be in a physics simulator (based on our laws of physics), which both narrows the search space considerably and increases the chance that the skills learned will be relevant to our world and more comprehensible to us. Many other approaches are viable and research into the best ways to generate effective learning environments, including how to encode them, will be essential for progress.

A wide open question in this line of research is what the reward function for the environment generator should be in the open-ended version of generating effective learning environments. This is a deep, fascinating, hard question that could be a key to unlocking significant progress in machine learning research. Finding the answer to this question could finally enable us to solve the longstanding grand challenge of producing open-ended search algorithms [ 172 ] and producing general AI, potentially solving two grand challenges in one stroke.

I do not have an answer to what this environment-generator reward function should be, but I have some ideas that researchers could begin experimenting with and improving on. The question relates to abstracting what environments were for in natural evolution or, analogously, what problems were for in the history of scientific and technological innovation. What role did environments serve in producing the complexity explosion on Earth, including creating human intelligence? What role did problems play in scientific and technological innovation within our culture?

What I consider the most promising direction for potential reward functions for environmental generators is to define environments as useful (and thus reward their creation) if the environments are such that agents transferring in from other, previously generated environments (1) perform well in the new environments (after some learning) faster than if they were trained on them from scratch, and (2) learn in the new environments (i.e. the environments are not too easy and not too hard, such that the agent experiences ‘learning progress’ [ 149 ] ). The first condition encourages shared structure between the problems (e.g. similar laws of physics, or mathematical rules), such that having learned in some subset of problems makes agents be able to transfer that knowledge to other environments. That could prevent the creation of arbitrarily different problems that are challenging, but in uninteresting ways. The second condition prevents the creation of worlds that do not require and encourage learning. It also forces environments to be new in some way. This idea needs to be developed, improved, and experimentally investigated, but it gives a hint of how we can begin to make progress on producing very general principles for preferring the creation of some environments versus others. Another idea that could help is an explicit pressure to produce generalists, perhaps by incentivizing agents to be able to solve as many niches as possible. We have other ideas for how to reward the generation of interesting environments, but I cannot share them because we are actively investigating them. It is likely that none of these ideas will just work. Instead, we need lots of research by the community into these questions to make progress on this front.

A major open question that remains is how we can constrain the generation of environments to be those we find interesting and/or that produce intelligence that helps us solve real-world problems. In other words, how do we ground the environment generator to make things relevant to us and our universe? For example, one might argue that such a system could produce intelligence that is alien to us and that we cannot communicate with. However, if it is truly general intelligence, presumably through its learning efforts and our own we could learn to communicate with it. Additionally, creating alien forms of intelligence would be fascinating as it would teach us about the limits and possibilities within the space of intelligent beings.

#### 2.3.5 The viability of the Third Pillar

The hypothesis behind the Third Pillar is that coming up with general principles for how to create effective learning environments is easier than creating a curriculum of training environments by hand. As with the overall AI-GA philosophy, the bet is that with sufficient compute, letting learning algorithms solve the problem will ultimately be easier than trying to solve it ourselves. Of course, this may require orders of magnitude more compute than we have at present, and whether we will have sufficient compute to see this pillar succeed before general AI is produced by the manual path is an open question. However, I predict that we will abstract the role environments play in the creation of complexity and intelligence, and that doing so will shave orders of magnitude off the total amount of compute required, such that we will not need the computation that was required on Earth to produce humans in order to create general AI via an AI-GA.

One thing in favor of the Third Pillar, and the AI-GA approach in general, is that we know it can work. Darwinian evolution on Earth is the only existence proof we have of how to build general intelligence, and AI-GAs are modeled off of that phenomenon. It is a fascinating research grand-challenge to figure out whether we can extract the principles that made Earth’s intelligence-generating algorithm so successful in a way that enables us to create an AI-GA with the computation available to us.

Note that the AI-GA grand challenge (creating an algorithm that generates general AI) is related to, but different from, the grand challenge of creating open-ended search algorithms [ 172 ] . One could create an AI-GA without it being open-ended. For example, one could launch an AI-GA with the target task being to pass the Turing Test, and the algorithm would then halt when it produces general AI because it would have no other goal to optimize for. One could also create an open-ended algorithm that would never produce an AI-GA (e.g. one that generates endless music innovations). That said, these two grand challenges are deeply related and will catalyze the progress of each other.

## 3 Discussion

The AI-GA approach raises many questions. This section attempts to quickly address a variety of those different questions.

Traditional machine learning involves hand-designing an environment, hand-selecting an architecture, and then hand-designing a learning algorithm that learns to solve that environment. Meta-learning represents a step towards a more fully automated pipeline. One area of research in meta-learning focuses on learning architectures. Another area of meta-learning is learning the learning algorithm. But in both, a researcher is still required to design the distribution of tasks for meta-training. AI-GAs take the next step in this natural progression by automatically learning all three things: the architecture, the learning algorithm, and the training environments.