Correspondence between neuroevolution and gradient descent — part 3/3

Cleaned-text code points [31733, 43135). Bibliography omitted here; source.md includes it.

Our results are also relevant to population-based genetic algorithms in which members of the population are periodically reset to the identities of the individuals with lowest loss values 11 , 12 , 13 . For instance, when correspondence holds, individuals in the neuroevolution populations considered in this paper have an averaged loss equal to that of the corresponding gradient descent algorithm. Therefore, some individuals must have loss less than that of the corresponding gradient descent algorithm (see e.g. Fig. 1 (a), and Fig. 3 (b) for the case λ = 1/10). This observation indicates the potential for such methods to be competitive with gradient-descent algorithms.

The neuroevolution-gradient descent correspondence we have identified follows from that between the overdamped Langevin dynamics and Metropolis Monte Carlo dynamics of a particle in an external potential 19 , 20 . Our work therefore adds to the existing set of connections between machine learning and statistical mechanics 42 , 43 , and continues a trend in machine learning of making use of old results: the stochastic and deterministic algorithms considered here come from the 1950s 9 , 10 and 1970s 1 , 2 , 3 , 4 , 5 , 6 , and are connected by ideas developed in the 1990s 19 , 20 .

## Methods

### Derivation of the neuroevolution-gradient descent correspondence

We start by considering the quantity P ( x , t ), the probability that a neural network has the set of parameters x at time t under a given stochastic protocol. The time evolution of this quantity is governed by the master equation 44 , 45 , which in generic form reads

The two terms in ( 19 ) describe, respectively, gain and loss of the probability P ( x , t ) (note that the probability to have some set of parameters is conserved, i.e., ∫d x P ( x , t ) = 1). The symbol \({W}_{{{{\bf{x}}}}^{\prime} -{{{\bf{x}}}}}({{{\bf{x}}}})\) (sometimes written \({W}(({{{\bf{x}}}} \to {{{\bf{x}}}}^{\prime}))\) ) quantifies the probability of moving from the set of parameters x to the set of parameters \({{{\bf{x}}}}+({{{\bf{x}}}}^{\prime} -{{{\bf{x}}}})={{{\bf{x}}}}^{\prime} \) , and encodes the details of the stochastic protocol. For the neuroevolution procedure defined in section “Neuroevolution” we write ( 18 ) as

Here, ϵ denotes the set of random numbers (the “mutation”) by which the neural-network parameters are changed; the integral \(\int {{{\rm{d}}}}{{{\boldsymbol{\epsilon }}}}=\int\nolimits_{-\infty }^{\infty }{{{\rm{d}}}}{\epsilon }_{i}\cdots \int\nolimits_{-\infty }^{\infty }{{{\rm{d}}}}{\epsilon }_{N}\) runs over all possible choices of mutations; and

is the rate for going from the set of parameters x to the set of parameters x + ϵ . Eq. ( 20 ) contains two factors. The first,

quantifies the probability of proposing a set of Gaussian random numbers ϵ . The second factor, the Metropolis “min” function in ( 20 ), quantifies the probability of accepting the proposed move from x to x + ϵ ; recall that U ( x ) is the loss function.

We can pass from the master equation ( 19 ) to a Fokker-Planck equation by assuming a small mutation scale σ , and expanding the terms in ( 19 ) to second order in σ 44 , 45 . Thus

and

where \({{{\boldsymbol{\epsilon }}}}\cdot \nabla =\mathop{\sum }\nolimits_{i = 1}^{N}{\epsilon }_{i}{\partial }_{i}\) (note that \({\partial }_{i}\equiv \frac{\partial }{\partial {x}_{i}}\) ). Collecting terms resulting from the expansion gives

Taking the integrals inside the sums, ( 24 ) reads

where

and

What remains is to calculate ( 26 ) and ( 27 ), which we do in different ways depending on the value of the evolutionary reciprocal temperature β .

#### The case of finite β

First we consider finite β , in which case we can evaluate ( 26 ) and ( 27 ) using the results of Refs. 19 , 20 (making small changes in order to account for differences in proposal rates between those papers and ours).

Eq. ( 26 ) can be evaluated as follows (writing U ( x ) = U for brevity):

In these expressions Θ( x ) = 1 if x ≥ 0 and is zero otherwise. In going from ( 28 ) to ( 29 ) we have assumed that β ϵ ⋅ ∇ U ( x ) is small. This condition cannot be met for β = ∞ ; that case is treated later. In going from ( 30 ) to ( 31 ) we have used the result Θ( x ) + Θ( − x ) = 1; the first integral in ( 31 ) then vanishes by symmetry. The second integral, shown in ( 32 ), can be turned into ( 33 ) using the symmetry arguments given in Ref. 20 , which we motivate as follows. Upon a change of sign of the integration variables, ϵ → − ϵ , the value of the integral in ( 32 ) is unchanged and it is brought to a form that is identical except for a change of sign of the argument of the Θ function. Adding the two forms of the integral removes the Θ functions, giving the form shown in ( 33 ), and dividing by 2 restores the value of the original integral. ( 33 ) can be evaluated using standard results of Gaussian integrals.

Eq. ( 27 ) can be evaluated in a similar way:

The ≈ sign in ( 38 ) indicates that we have omitted terms of order σ 3 .

Inserting ( 34 ) and ( 39 ) into ( 25 ) gives us, to second order in σ , the Fokker-Planck equation

This equation is equivalent (the diffusion term is independent of x , and so the choice of stochastic calculus is unimportant) to the N Langevin equations 44 , 45

where ξ is a Gaussian white noise with zero mean and variance σ 2 :

Eq. ( 41 ) describes an evolution of the neural-network parameters x i that is equivalent to gradient descent with learning rate α = β σ 2 /2, plus Gaussian white noise. Averaging over independent stochastic trajectories of the learning process (starting from identical initial conditions) gives

which is equivalent to simple gradient descent on the loss function.

#### The case β = ∞

When β = ∞ we only accept mutations that do not increase the loss function. To treat this case we return to ( 26 ) and take the limit β → ∞ :

We can make progress by introducing the integral form of the Θ function (see e.g., ref. 46 ),

Then ( 44 ) reads

where the symbols

are standard Gaussian integrals. Upon evaluating them as

and

( 46 ) reads

The form ( 51 ) is similar to ( 34 ) in that it involves the derivative of the loss function U ( x ) with respect to x i , but contains an additional normalization term, ∣ ∇ U ∣. This term is sometimes introduced as a form of regularization in gradient-based methods 25 . Here, the form emerges naturally from the acceptance criterion that sees any move accepted if the move does not increase the loss function: as a result, the length of the step taken does not depend strongly on the size of the gradient.

In the limit β → ∞ , ( 27 ) reads

upon applying the symmetry arguments used to evaluate ( 32 ). Eq. ( 54 ) is half the value of the corresponding term for the case β ≠ ∞ , Eq. ( 39 ), because one term corresponds to Brownian motion in unrestricted space, the other to Brownian motion on a half-space.

Inserting ( 51 ) and ( 54 ) into ( 25 ) gives a Fokker–Planck equation equivalent to the N Langevin equations

where η is a Gaussian white noise with zero mean and variance σ 2 /2:

As an aside, we briefly consider the case of non-isotropic mutations, for which the Gaussian random update applied to parameter i has its own variance \({\sigma }_{i}^{2}\) , i.e., \({\epsilon }_{i} \sim {{{\mathcal{N}}}}(0,{\sigma }_{i}^{2})\) in step 2 of the procedure described in section “Neuroevolution”. In this case the derivation above is modified to have σ replaced by σ i in ( 21 ). In the case of finite β the equations ( 41 ) and ( 42 ) retain their form with the replacement σ → σ i . In the case of infinite β , ( 56 ) retains its form with the replacement σ → σ i and ( 55 ) reads

with \({\tilde{\partial }}_{i}\equiv {\sigma }_{i}\frac{\partial }{\partial {x}_{i}}\) and \(| \tilde{\nabla }U| \equiv \sqrt{\mathop{\sum }\nolimits_{j = 1}^{N}{\left({\tilde{\partial }}_{j}U\right)}^{2}}\) . Non-isotropic mutations are used in covariance matrix adaptation strategies 47 . Those schemes also evolve the step size parameter dynamically, and to model this more general case one must make σ a dynamical variable of the master equation, with update rules appropriate to the algorithm of interest.

## Data availability

Data can be generated using the source code at https://github.com/reproducible-science/MC-GD-correspondence .

## Code availability

Source code is available at https://github.com/reproducible-science/MC-GD-correspondence .

## Supplementary information

### Supplementary Information (download PDF )

## Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

## About this article

### Cite this article

Whitelam, S., Selin, V., Park, SW. et al. Correspondence between neuroevolution and gradient descent. Nat Commun 12 , 6317 (2021). https://doi.org/10.1038/s41467-021-26568-2

Download citation

Received : 26 April 2021

Accepted : 04 October 2021

Published : 02 November 2021

Version of record : 02 November 2021

DOI : https://doi.org/10.1038/s41467-021-26568-2

### Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Advertisement

## Search

### Quick links

Explore articles by subject

Find a job

Guide to authors

Editorial policies

Nature Communications ( Nat Commun )

ISSN 2041-1723 (online)

## nature.com footer links

### About Nature Portfolio

About us

Press releases

Press office

Contact us

### Discover content

Journals A-Z

Articles by subject

protocols.io

Nature Index

### Publishing policies

Nature portfolio policies

Open access

### Author & Researcher services

Reprints & permissions

Research data

Language editing

Scientific editing

Nature Masterclasses

Research Solutions

### Libraries & institutions

Librarian service & tools

Librarian portal

Open research

Recommend to library

### Advertising & partnerships

Advertising

Partnerships & Services

Media kits

Branded content

### Professional development

Nature Awards

Nature Careers

Nature Conferences

### Regional websites

Nature Africa

Nature China

Nature India

Nature Japan

Nature Middle East

Privacy Policy

Use of cookies

Your privacy choices/Manage cookies

Legal notice

Accessibility statement

Terms & Conditions

Your US state privacy rights

© 2026 Springer Nature Limited

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.