AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence — part 1/8

Cleaned-text code points [0, 15874). Bibliography omitted here; source.md includes it.

# AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence

###### Abstract

Perhaps the most ambitious scientific quest in human history is the creation of general artificial intelligence, which roughly means AI that is as smart or smarter than humans. The dominant approach in the machine learning community is to attempt to discover each of the pieces that might be required for intelligence, with the implicit assumption that at some point in the future some group will complete the Herculean task of figuring out how to combine all of those pieces into an extremely complex machine. I call this the “manual AI approach.” This paper describes another exciting path that ultimately may be more successful at producing general AI. It is based on the clear trend from the history of machine learning that hand-designed solutions eventually are replaced by more effective, learned solutions. The idea is to create an AI-generating algorithm (AI-GA), which itself automatically learns how to produce general AI. Three Pillars are essential for the approach: (1) meta-learning architectures, (2) meta-learning the learning algorithms themselves, and (3) generating effective learning environments. While work has begun on the first two pillars, little has been done on the third. Here I argue that either the manual or AI-GA approach could be the first to lead to general AI, and that both are worthwhile scientific endeavors irrespective of which is the fastest path. Because both approaches are roughly equally promising, and because the machine learning community is mostly committed to the engineered AI approach currently, I argue that our community should shift a substantial amount of its research investment to the AI-GA approach. To encourage such research, I describe promising work in each of the Three Pillars. I also discuss the safety and ethical considerations unique to the AI-GA approach. Because it may be the fastest path to general AI and because it is inherently scientifically interesting to understand the conditions in which a simple algorithm can produce general intelligence (as happened on Earth where Darwinian evolution produced human intelligence), I argue that the pursuit of AI-GAs should be considered a new grand challenge of computer science research.

## 1 Two approaches to producing general AI: the manual approach vs. AI-generating algorithms

Arguably the most ambitious scientific quest in human history is the creation of general artificial intelligence, which roughly means AI as smart or smarter than humans 1 1 1 This paper will not attempt to define general intelligence aside from saying it roughly means AI as smart or smarter than humans. Nor does this paper engage in the debate about to what extent such a thing exists. Such terrain is well-trodden without resolution, and is not the focus of this paper. . The creation of general AI would transform every aspect of society and economic sector. It would also catalyze scientific discovery, leading to unpredictable advances, including further advances in AI itself (the potential ethical consequences of which I discuss in Section 4 ).

### 1.1 The manual AI approach

One approach to producing general AI is via a two-phase process. In Phase One, we discover each of the pieces that might be required for intelligence. In Phase Two, we then figure out how to put all of those pieces together into an extremely complex machine. This path has been followed since the earliest days of AI, such as initial research into systems intended to capture logical thought or process language [ 142 ] . This is also the path implicitly being taken by the vast majority of the machine learning research community. Most papers introduce or refine a single building block of intelligence, such as convolution [ 91 ] , recurrent gated cells [ 65 , 23 , 24 ] , skip connections [ 61 , 164 ] , attention-mechanisms [ 197 , 36 ] , activation functions [ 52 , 58 , 118 , 105 , 25 ] , external memory [ 53 , 193 ] , good initializations [ 50 , 52 ] , normalization schemes [ 74 , 98 ] , hierarchical methods [ 34 , 11 , 60 , 188 ] , capsules [ 144 , 64 ] , unsupervised learning techniques [ 52 , 135 , 89 , 51 ] , value or advantage functions [ 179 ] , intrinsic motivation [ 174 , 128 , 126 , 19 ] , trust regions [ 153 ] , auxiliary tasks [ 76 ] , solutions to catastrophic forgetting [ 83 , 40 , 202 , 187 ] , and many, many more. Table 1 lists many example building blocks, and this list is far from exhaustive. Seeing the length of this incomplete list raises the following questions: How long will it take to identify the right variant of each building block in this list? How many more essential building blocks exist that we have yet to invent? How long will it take us to discover them all?

Even if we were able to create effective versions of each of the key building blocks to general AI in a reasonable amount of time, our work would remain unfinished. What is rarely explicitly mentioned is that collecting key building blocks of intelligence is only useful in terms of building general AI if at some point some group takes on the Herculean scientific challenge of figuring out how to combine of all of those pieces into an intelligent machine. Combining multiple, different building blocks is extraordinarily difficult due to (1) the many different ways they can be combined, (2) non-linear interaction effects between modules (e.g. a version of a module can work well when combined with some building blocks, but not others), and (3) the scientific and engineering challenges inherent in working with complex systems of dozens or hundreds of interacting parts, each of which is a complex, likely black-box, machine learning module (e.g. imagine trying to debug such a machine or identifying which piece needs to be tweaked if things are not working). There have been notable, valiant, successful efforts to combine multiple (e.g. around six to thirteen) AI building blocks [ 78 , 62 ] , but such attempts still feature a low number of building blocks vs. the dozens or hundreds that may be required. Such efforts are also rare and understandably are unable to assess the contribution of each separate building block, let alone different combinations and/or configurations of each building block. Such knowledge may be necessary, however, for us to engineer extreme levels of intelligence into machines.

A final challenge with combining dozens or hundreds of separate building blocks is that doing so does not work well within our scientific culture. Current scientific incentives, organizations, and traditions usually reward small teams of scientists to produce a few papers per year. Combining all of the required AI building blocks together instead would likely require an extremely large, dedicated team to work for years or decades in something akin to an Apollo program. While some excellent institutions like DeepMind and OpenAI are large, well-funded, and focused on creating general AI, they still tend to feature smaller teams focused on separate projects and publishing regularly, instead of an organization where all hands on deck are committed to one project, which may be required to engineer AI. Of course, their structures could evolve with time once the building blocks are thought to have been identified.

To be clear, I believe the manual AI path (1) has a chance of being the fastest path to AI (Section 1.6 ) and (2) is worthwhile to pursue even if it is not (Section 1.5 ). That said, it is important to make its assumptions explicit—that it is a manual engineering approach involving the two phases described above—and to acknowledge how daunting of a challenge it is, for scientific, engineering, and sociological reasons. Making those points is one of the objectives of this manuscript.

• convolution

• pooling

• activation functions

• gradient-friendly architectures (e.g. skip connections)

• gated recurrent units (e.g. LSTMs & GRUs)

• structural organization (regularity, modularity, hierarchy)

• spatial tranformers

• capsules

• batch/layer norm

• experience replay buffers

• writeable memory (e.g. memory networks, neural Turing machines)

• recurrent layers

• multi-modal fusion

⋄ \diamond learned losses (e.g. evolved policy gradients)

⋄ \diamond hierarchical RL, options

⋄ \diamond intelligent exploration (e.g. via intrinsic motivation)

⋄ \diamond good initializations (Xavier, MAML, etc.)

⋄ \diamond universal value function approximators

⋄ \diamond auxiliary tasks (predicting states, autoencoding, predicting rewards, etc.)

⋄ \diamond catastrophic forgetting solutions

⋄ \diamond efficient continual learning

⋄ \diamond Dyna

⋄ \diamond variance reduction techniques

⋄ \diamond good hyperparameters

⋄ \diamond value functions, action-value functions, advantage functions

⋄ \diamond models of the world

⋄ \diamond planning algorithms

⋄ \diamond backprop through time

⋄ \diamond causal reasoning

⋄ \diamond trust regions

⋄ \diamond Bayesian learning

⋄ \diamond neuromodulation

⋄ \diamond Hebbian learning

⋄ \diamond active learning

⋄ \diamond probabilistic models

⋄ \diamond generative adversarial networks

⋄ \diamond time-contrastive, object-contrastive losses

⋄ \diamond audio-visual correspondence

⋄ \diamond domain confusion

⋄ \diamond entropy bonuses

⋄ \diamond regularization

⋄ \diamond distance metrics

⋄ \diamond hindsight experience replay

⋄ \diamond attention mechanisms

⋄ \diamond complex optimizers (Adam, RMSprop, etc.)

⋄ \diamond learning rates & schedules

⋄ \diamond discount factors

⋄ \diamond eligibility traces

⋄ \diamond theory of mind

⋄ \diamond self-modeling

⋄ \diamond unsupervised learning

† \dagger co-evolution / self-play

† \dagger staging / shaping

† \dagger curriculum learning

† \dagger communication / language

† \dagger multi-agent interaction

† \dagger diverse environmental challenges

† \dagger environmental hyperparameters (e.g. episode length)

### 1.2 A different approach: AI-generating algorithms (AI-GAs)

There is another exciting path that ultimately may be more successful at producing general AI: the idea is to learn as much as possible in an automated fashion, involving an AI-generating algorithm (AI-GA) that bootstraps itself up from scratch to produce general AI. As argued below, this approach also could prove to be the fastest path to AI, and is interesting to pursue even if it is not.

One motivation for a learn-as-much-as-possible approach is the history of machine learning. There is a repeating theme in machine learning (ML) and AI research. When we want to create an intelligent system, we as a community often first try to hand-design it, meaning we attempt to program it ourselves. Once we realize that is too hard, we then try to hand-code some components of the system and have machine learning figure out how to best use those hand-designed components to solve a task. Ultimately, we realize that with sufficient data and computation, we can learn the entire system. The fully learned system often performs better. It also is more easily applied to new challenges (i.e. it is a more general solution).

The clearest example of this trend is in computer vision. First we tried to hand-code computational vision systems in the early decades of AI research and found that they did not work well [ 182 ] . We then switched to a hybrid approach wherein we manually designed features (e.g. edge-, texture-, object-, and other feature-detectors, including the famous HOG [ 33 ] and SIFT [ 104 ] feature sets) and then had machine learning algorithms automatically learn the best way to use those hand-coded features to solve a particular problem [ 182 ] . Starting around 2012 [ 87 ] , we realized that we had sufficient computation and data to learn entire vision systems without injecting hand-designed features. Doing so has improved performance and such systems are very general: they can be applied to a wide variety of vision tasks and learn appropriate features for each. This same phenomenon has occurred in other modes of machine learning, such as sound (e.g. speech-to-text systems) [ 63 ] , touch [ 49 ] , natural language understanding and translation [ 36 , 8 , 177 ] , and many more [ 52 ] .

This trend also has occurred in other areas of machine learning. For decades, but with especially intense effort since 2012, we as a community have been hand-designing increasingly powerful neural network architectures [ 61 , 68 , 181 , 52 ] . Increasingly, however, researchers are searching for deep neural network architectures automatically with machine learning algorithms [ 140 , 137 , 114 , 203 ] . Despite all of the considerable effort that has gone into designing architectures for image classification manually, an architecture search algorithm produced the state of the art on the CIFAR benchmark at the time of its publication [ 139 ] . The state of the art on the extremely popular and over-optimized ImageNET benchmark also belongs not to a human-designed architecture, but to one produced by an architecture search algorithm [ 139 ] . Additionally, while most hyperparameters have been found by hand throughout the history of machine learning, they are increasingly learned [ 159 , 77 ] . Additionally, there is a recent increase in meta-learning the learning algorithms themselves [ 37 , 191 , 40 , 187 , 161 ] (discussed more in Section 2.2 ).

I am amongst those that believe that this trend of learned pipelines replacing hand-designed ones will continue, but increasingly it will apply to the machinery of machine learning itself. That opens the tantalizing prospect that we might be able to create an algorithm that learns everything required to produce general AI, and thus produce an AI-GA. That would obviate the need to discover, refine, and combine the individual building blocks of an intelligent machine (e.g. those in Table 1 and the many yet to be discovered).

A second motivation for a learn-as-much-as-possible approach is that it is the only approach that we are certain can work. That is because it already has. On Earth, a remarkably unintelligent algorithm in Darwinian evolution was paired with the correct conditions and bootstrapped itself up from the simplest replicators to ultimately producing the human mind. Darwinian evolution thus is the first general-intelligence-generating algorithm, providing an existence proof that the concept of general-intelligence-generating algorithms can work. Natural evolution required unfathomable amounts of computation, however. An open scientific question is how we can create abstractions of what drove the success of natural evolution in order to produce AI-GAs that can work within the computation that we will have in the coming years.

### 1.3 The Three Pillars required to produce an AI-GA

There are Three Pillars that, if we could get them to work well, should produce general AI. Thus, research into AI-GAs should focus on improving and combining them. The Three Pillars are (1) meta-learning architectures, (2) meta-learning the learning algorithms themselves (often just called meta-learning), and (3) automatically generating effective learning environments. Section 2 describes each pillar and what research into it might look like. Briefly, research into the First and Second Pillars is decades old, but both have seen a surge of recent interest and activity. There has been very little research into the Third Pillar, although we recently published our first step in this direction [ 192 ] . Thus the Third Pillar is the least-studied and least-understood. I consider it the hardest. I predict that more history-making discoveries await in this pillar than the other two, making it an exciting, fruitful area of research.

### 1.4 AI-GAs: A new grand challenge for AI research