AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence — part 7/8

Cleaned-text code points [91484, 106761). Bibliography omitted here; source.md includes it.

In my view, the largest ethical concern unique to the AI-GA path is that it is, by definition, attempting to create a runaway process that leads to the creation of intelligence superior to our own. Many AI researchers have stated that they do not believe that AI will suddenly appear, but instead that progress will be predictable and slow. However, it is possible in the AI-GA approach that at some point a set of key building blocks will be put together and paired with sufficient computation. It could be the case that the same amount of computation had previously been insufficient to do much of interest, yet suddenly the combination of such building blocks finally unleashes an open-ended process. I consider it unlikely to happen any time soon, and I also think there will be signs of much progress before such a moment. That said, I also think it is possible that a large step-change occurs such that prior to it we did not think that an AI-GA was in sight. Thus, the stories of science fiction of a scientist starting an experiment, going to sleep, and awakening to discover they have created sentient life are far more conceivable in the AI-GA research paradigm than in the manual path. As mentioned above, no amount of compute on training a computer to recognize images, play Go, or generate text will suddenly become sentient. However, an AI-GA research project with the right ingredients might, and the first scientist to create an AI-GA may not know they have finally stumbled upon the key ingredients until afterwards. That makes AI-GA research more dangerous.

Relatedly, a major concern with the AI-GA path is that the values of an AI produced by the system are less likely to be aligned with our own. One has less control when one is creating AI-GAs than when one is manually building an AI machine piece by piece. Worse, one can imagine that some ways of configuring AI-GAs (i.e. ways of incentivizing progress) that would make AI-GAs more likely to succeed in producing general AI also make their value systems more dangerous. For example, some researchers might try to replicate a basic principle of Darwinian evolution: that it is ‘red in tooth and claw.’ If a researcher tried to catalyze the creation of an AI-GA by creating conditions similar to those on Earth, the results might be similar. We might thus produce an AI with human vices, such as violence, hatred, jealousy, deception, cunning, or worse, simply because those attributes make an AI more likely to survive and succeed in a particular type of competitive simulated world. Note that one might create such an unsavory AI unintentionally by not realizing that the incentive structure they defined encourages such behavior. In fact, it is routine for researchers to be surprised by the strategies AI comes up with to optimize the objectives it is given, and often the strategies and behaviors it creates are not at all what the researcher intended [ 97 ] . That phenomenon will only increase as AI-GA systems become more powerful, complex, and are able to make significant advances on their own. If a system based on red-in-tooth-and-claw competition is a faster path to a successful AI-GA, or even if it is thought to be, then some organizations or individuals on Earth may be incentivized to create such systems despite their potential risks.

Additionally, it is likely safer to create AI when one knows how to make it piece by piece. To paraphrase Feynman again, one better understands something when one can build it. Via the manual approach, we would likely understand relatively more about what the system is learning in each module and why. The AI-GA system is more likely to produce a very large black box that will be difficult to understand. That said, even current neural networks, which are tiny and simple compared to those that will likely be required for AGI, are inscrutable black boxes that are very difficult to understand the inner workings of [ 120 , 122 , 124 , 121 , 200 , 102 , 199 , 180 , 107 , 201 , 100 , 54 ] . Once these networks are larger and have more complex, interacting pieces, the result might be sufficiently inscrutable that it does not end up mattering whether the inscrutability is even higher with AI-GAs. While ultimately we likely will learn much about how these complex brains work, that might take many years. From the AI safety perspective, however, what is likely most critical is our ability to understand the AI we are creating right around the time that we are finally producing very powerful AI.

For all these reasons, it is essential to invest in AI-GA-specific AI safety research. AI-GA researchers need to be in constant communication with AI safety researchers to help inform them. Ideally, AI-GA researchers should conduct safety research themselves in addition to making AI-GA advances. Each AI-GA scientist must take precautions to try to ensure that AI-GA research is safe and, should it succeed, that it produces AIs whose values are aligned with our own.

It is fair to ask why should I write this paper if I think AI-GA research is more dangerous, as I am attempting to inform people about it potentially being a faster path to general AI and advocating that more people work on this path. One reason is I believe that, on balance, technological advances produce more benefit than harm. That said, this technology is very different and could prove an exception to the rule. A second reason is because I think society is better off knowing about this path and its potential, including its risks and downsides. We might therefore be better prepared to maximize the positive consequences of the technology while working hard to minimize the risks and negative outcomes. Additionally, I find it hard to imagine that, if this is the fastest path to AI, then society will not pursue it. I struggle to think of powerful technologies humanity has not invented soon after it had the capability to do so. Thus, if it is inevitable, then we should be aware of the risks and begin organizing ourselves in a way to minimize those risks. Very intelligent people disagree with my conclusion to make knowledge of this technology public. I respect their opinions and have discussed this issue with them at length. It was not an easy decision for me to make. But ultimately I feel that it is a service to society to make these issues public rather than keep them the secret knowledge of a few experts.

There is another ethical concern, although many will find it incredible and dismiss it as the realm of fantasy or science fiction. We do not know how physical matter such as atoms can produce feelings and sensations like pain, pleasure, or the taste of chocolate, which philosophers call qualia. While some disagree, I think we have no good reason to believe that qualia will not emerge at some point in artificially intelligent agents once they are complex enough. A simple thought experiment makes the point: imagine if the mimic path enabled us to simulate an entire human brain and body, down to each subatomic particle. It seems likely to me that such a simulation would feel the same sensations as its real-world counterpart.

Recognizing if and when artificial agents are feeling pain, pleasure, and other qualia that are worthy of our ethical considerations is an important subject that we will have to come to terms with in the future. However, that issue is not specific to the method in which AI is produced, and therefore is not unique to the AI-GA path. There is an AI-GA-specific consideration on this front, however. On Earth, there has been untold amounts of suffering produced in animals en route to the production of general AI. Is it ethical to create algorithms in which such suffering occurs if it is essential, or helpful, to produce AI? Should we ban research into algorithms that create such suffering in order to focus energy on creating AI-GAs that do not involve suffering? How do we balance the benefits to humans and the planet of having general AI vs. the suffering of virtual agents? These are all questions we will have to deal with as research progresses on AI-GAs. They are related to the general question of ethics for artificial agents, but have unique dimensions worthy of specific consideration.

Some of these ideas will seem fantastical to many researchers. In fact, it is risky for my career to raise them. However, I feel obligated to let society and our community know that I consider some of these seemingly fantastical outcomes possible enough to merit consideration. For example, even if there is a small chance that we create dangerous AI or untold suffering, the costs are so great that we should discuss that possibility. As an analogy, if there were a 1% chance that a civilization-ending asteroid could hit Earth in a decade or ten, we would be foolish not to begin discussing how to track it and prevent that catastrophe.

We should keep in mind the grandeur of the task we are discussing, which is nothing short than the creation of an artificial intelligence smarter than humans. If we succeed, we arguably have also created life itself, by some definitions. We do not know if that intelligence will feel. We do not know what its values might be. We do not know what its intentions towards us may be. We might have an educated guess, but any student of history would recognize that it would be the height of hubris to assume we know with certainty exactly what general AI will be like. Thus, it is important to encourage, instead of silence, a discussion of the risks and ethical implications of creating general artificial intelligence.

## 5 Conclusions

In this essay I described three paths to producing general artificial intelligence. The ‘mimic path’ builds as many biological details of human brains as possible into computational models, and is pursued largely by neuroscientists, computational neuroscientists, and cognitive scientists. The mimic path is unlikely to be the fastest path to general AI because it attempts to simulate all of the detail of biological brains irrespective of whether they can be ignored or abstracted by different, more efficient, machinery.

The ‘manual path’ is what most of the machine learning community is currently committed to. It involves two phases. In Phase 1, we identify each of the building blocks necessary to create a complex thinking machine. This is the phase we are currently in: most papers either introduce new candidate building blocks or improvements to previously proposed building blocks. It is unclear how long it might take to identify all of the correct building blocks, including the right variants of each one. The manual path implicitly assumes a Phase 2, where we will undertake the Herculean task of figuring out how to combine all of the correct variants of these building blocks into a complex thinking machine. One of the goals of this essay is simply to make explicit the path most of machine learning is committed to and that it implies a second phase that is rarely discussed. Of course, these phases will in practice overlap. There will be teams that increasingly try to combine existing building blocks while other teams continue to create new building blocks and improve existing ones. I discussed the scientific, engineering, and sociological pros and cons of this manual path to creating general AI.

I also described an alternative path to AI: creating general AI-generating algorithms, or AI-GAs. This path involves Three Pillars: meta-learning architectures, meta-learning algorithms, and automatically generating effective learning environments. As with the other paths, there are advantages and disadvantages to this approach. A major con is that AI-GAs will require a lot of computation, and therefore may not be practical in time to be the first path to produce general AI. However, AI-GA’s ability to benefit more readily from exponential improvements in the availability of compute may mean that it surpasses the manual path before the manual path succeeds. A reason to believe that the AI-GA path may be the fastest to produce general AI is in line with the longstanding trend in machine learning that hand-coded solutions are ultimately surpassed by learning-based solutions as the availability of computation and data increase over time. Additionally, the AI-GA path may win because it does not require the Herculean Phase 2 of the manual path and all of its scientific, engineering, and sociological challenges. Additional benefits of AI-GA research are that fewer people are working on it, making it an exciting, unexplored research frontier.

All three paths are worthwhile scientific grand challenges. That said, society should increase its investment in the AI-GA path. There are entire fields and billions of dollars devoted to the mimic path. Similarly, most of the machine learning community is pursuing the manual path, including billions of dollars in government and industry funding. Relative to these levels of investment, there is little research and investment in the AI-GA path. While still small relative to the manual path, there has been a recent surge of interest in Pillar 1 (meta-learning architectures) and Pillar 2 (meta-learning algorithms). However, there is little work on Pillar 3, and no work to date on attempting to combine the Three Pillars. Since the AI-GA path might be the fastest path to producing general AI, then society should substantially increase its investment in AI-GA research. Even if one believes the AI-GA path has a 1%-5% of being the first to produce general AI, then we should allocate corresponding resources into the field to catalyze its progress. That, of course, assumes we conclude that the benefits of potentially producing general AI faster outweigh the risks of producing it via AI-GAs, which I ultimately do. At a minimum, I hope this paper motivates a discussion on these questions. While there is great uncertainty about which path will ultimately produce general AI first, I think there is little uncertainty that we are underinvesting in a promising area of machine learning research.

Finally, this essay has discussed many of the interesting consequences of building general AI that are unique to producing general AI via AI-GAs. One benefit is being able to produce a large diversity of different types of intelligent beings, and thus accelerating our ability to understand intelligence in general and all its potential manifestations. Doing so may also better help us understand our own single instance of intelligence, much as traveling the world is necessary to truly understand one’s hometown. Each different intelligence produced by an AI-GA could also create entire alien histories and cultures from which we can learn from. Downsides unique to AI-GAs were also discussed, including that it might make the sudden, unanticipated production of AI more likely, that it might make producing dangerous forms of AI more likely, and that it may create untold suffering in virtual agents. While I offered my own views on these issues and how I weigh the positives and negatives of this technology for the purpose of deciding whether we should pursue it, a main goal of mine is to motivate others to discuss these important issues.