Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning — part 4/4

Cleaned-text code points [46638, 51895). Bibliography omitted here; source.md includes it.

For these experiments we ran 5 independent, randomly initialized, runs and report the median of those runs. During the elite selection routine we did not reevaluate offspring 30 times like on the Atari experiments. That is because we ran these experiments before the Atari experiments, and we improved our evaluation methods after these experiments were completed. We did not have the computational resources to re-run these experiments with the changed protocol, but we do not believe this change would qualitatively alter our results. We also used the normalized columns initialization routine of Salimans et al. (2017) instead of Xavier initialization, but we found them to perform qualitatively similarly. When determining the fitness of each agent we evaluate the mean over 5 independent episodes. After each generation, for plotting purposes only, we evaluate the elite 30 times.

The architecture has two 256-unit hidden layers with tanh activation functions. This architecture is the one in the configuration file included in the source code released by Salimans et al. (2017) . The architecture described in their paper is similar, but smaller, having 64 neurons per layer ( Salimans et al., 2017 ) . Although relatively shallow by deep learning standards, and much smaller than the Atari DNNs, this architecture still contains ∼ \sim 167k parameters, which is orders of magnitude greater than the largest neural networks evolved for robotics tasks that we are aware of, which contained 1,560 ( Huizinga et al., 2016 ) and before that 800 parameters ( Clune et al., 2011 ) . Many assumed evolution would fail at larger scales (e.g. networks with hundreds of thousands or millions of weights, as in this paper).

Previous work has called the Humanoid-v1 problem solved with a score of ∼ \sim 6,000 ( Salimans et al., 2017 ) . The GA achieves a median above that level after ∼ \sim 1,500 generations. However, it requires far more computation than ES to do so (ES requires ∼ \sim 100 generations for median performance to surpass the 6,000 threshold). It is not clear why the GA requires so much more computation, especially given how quickly the GA found high-performing policies in the Atari domain. It is also surprising that the GA does not excel at this domain, given that GAs have performed well in the past on robot control tasks ( Clune et al., 2011 ) . While the GA needs far more computation in this domain, it is interesting nevertheless that it does eventually solve it by producing an agent that can walk and score over 6,000. Considering its very fast discovery of high-performing solutions in Atari, clearly the GA’s advantage versus other methods depends on the domain, and understanding this dependence is an important target for future research.

### 7.6 The meaning of “frames”

Many papers, including ours, report the number of “frames” used during training. However, it is a bit unclear in the literature what is meant exactly by this term. We hope to introduce some terminology that can lend clarity to this confusing issue, which will improve reproducibility and our ability to compare algorithms fairly. Imagine if the Atari-emulator emitted 4B frames during training. We suggest calling these “game frames.” One could sub-sample every 4th frame (indeed, due to “frame skip”, most Atari papers do exactly this, and repeat the previous action for each skipped frame), resulting in 1B frames. We suggest calling these 1B frames “training frames”, as these are the frames the algorithm is trained on. In our paper we report the game frames used by each algorithm. Via personal communication with scientists at OpenAI and DeepMind, we confirmed that we are accurately reporting the number of frames (and that they are game frames, not training frames) used by DQN, A3C, and ES in Mnih et al. (2015) , Mnih et al. (2016) , and Salimans et al. (2017) , respectively.

There is one additional clarification. In all of the papers just mentioned and for the GA in this paper, the input to the network for Atari is the current frame t and three previous frames. These three previous frames are from the training frame set , meaning that if t t counts each game frame then the input to the network is the following: game frame t , frame t-4 , frame t-8 , and frame t-12 .

## Instructions for reporting errors

We are continuing to improve HTML versions of papers, and your feedback helps enhance accessibility and mobile support. To report errors in the HTML that will help us improve conversion and rendering, choose any of the methods listed below:

Click the "Report Issue" ( ) button, located in the page header.

Tip: You can select the relevant text first, to include it in your report.

Our team has already identified the following issues . We appreciate your time reviewing and reporting rendering errors we may not have found yet. Your efforts will help us improve the HTML versions for all readers, because disability should not be a barrier to accessing research. Thank you for your continued support in championing open access for all.

Have a free development cycle? Help support accessibility at arXiv! Our collaborators at LaTeXML maintain a list of packages that need conversion , and welcome developer contributions .