A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 12/12

Cleaned-text code points [156539, 165992). Bibliography omitted here; source.md includes it.

Non-deterministic functions do not have a dedicated symbol, but must be represented
as a combination of deterministic functions, energy modules, and latent variables. An
example is shown on the right of Figure 18. A non-deterministic function is represented
by a deterministic function ˜y = g(x,z ) whose output depends on a latent variable z. The
latent variable is fed to a regularizing energy term R(z). We first defineZh as the level set
of z, i.e. the set of z for which R(z) is less than a threshold h:
Zh ={z/R(z)<h} (21)
As z varies overZh, the output will vary over the set:
Yh ={y|y =g(x,z ),∀z∈Zh} (22)
In some cases, this setup may be used to represent probability distributions. Forst,
the energy term is transformed into a probability distribution using a Gibbs-Boltzmann
formula:
P (z) = exp(−R(z))∫
z′ exp(−R(z′)) (23)
Drawing the latent variable from this distribution implies a distribution over y:
P (y|x) =
∫
z
δ(y−g(x,z ))P (z) (24)
where δ() is the Dirac delta function.
58

Figure 19: Amortized Inference with an EBM. An encoder is trained to produce ˜z = Enc(sx,y )
to approximate the value of the latent that minimizes the energy ˇz = argminz∈ZEw(x,y,z ). The
regularizerR(z) plays the essential role of limiting the information that z contains about y. This
is particularly important here because the system has access to y and can “cheat” by carrying the
complete information about y through the encoder.
Appendix: Amortized Inference for Latent Variables
Inference in latent variable models consists in performing the optimization
ˇz = argminz∈ZEw(x,y,z ). When z is continuous, this may be best performed through
a gradient-based optimization that involves backpropagating gradients through the model
dowen to z for multiple iterations. In generative architectures, this may be expensive,
requiring to back-propagate through the decoder and the predictor. One way to reduce the
cost of inference is to use amortized inference. The idea is to train a neural net to predict
an approximate solution to the inference optimization problem.
The architecture is depicted in Figure 20. An encoder ˜ z = Enc(sx,y ) is trained to
minimize a divergence measure H(ˇz, ˜z) between the encoder output and the optimal latent
variable ˇz = argmin z∈ZEw(x,y,z ). Once trained, the prediction ˜ z may be use as an
estimate of ˇz or as an initial value for the inference optimization.
59

Figure 20: Amortized Inference with a Regularized Generative Latent-Variable EBM architecture.
An encoder is trained to produce ˜z = Enc(sx,y ) to approximate the value ˇz that minimizes the
energy. The regularizer R(z) plays the essential role of limiting the information that z contains
about y. This is particularly important here because the system has access to y and can “cheat” by
carrying the complete information about y through the encoder.
The regularizer R(z) is even more important than in the regular inference case because
the prediction pathway has access toy and can “cheat” by carrying the complete information
abouty through the encoder. Without an information-limiting regularizer This would cause
a collapse of the energy function, since it would allow any y to be reconstructed perfectly.
The regularizer is there to minimize the information that ˇz may contains about y.
Variational Auto-Encoders, and LISTA-style sparse Auto-Encoders belong to the family
of Regularized GLVEBM with amortized inference. Most of those models are unconditional
and do not possess an x nor an Enc(x) module.
60

Appendix: Loss functions for Contrastive Training of EBM
Much can be said about contrastive methods. Table 1 lists a few examples of contrastive
methods, together with their strategy for picking contrastive samples ˆy and their loss func-
tional.
Rows 1-2 in the table are exact maximum likelihood methods. They assume that the
gradient of the log partition function can be computed exactly. Rows 2-4 are approxi-
mate maximum likelihood methods. Rows 5-10 are not interpretable within a probabilistic
framework.
Row 1: Maximum Conditional Likelihood for discrete y is used whenever the
energy needs to be turned into a probability distribution. through a Gibbs formulaP (y|x) =
exp(−Fw(x,y ))/∑
y′∈Yexp(−Fw(x,y′). The loss is the negative log conditional likelihood.
This is the prevalent approach when y is a discrete variable within a finite set (e.g. for
classification).
Row 2 and 3: Maximum Conditional Likelihood is used for any model that should
produce probability estimates. Row 2 only applies to tractable models in which the integral
in the contrastive term (or its gradient) can be computed analytically. Row 3 applies to
situations where the integral is intractable and its gradient must be approximated by Monte
Carlo sampling methods. It then comes down to devising good methods to sample ˆy values
from the model’s Gibbs distribution: Pw(y|x) = exp(−βFw(x,y )/
∫
y′ exp(−βFw(x,y′).
Row 4: Contrastive Divergence . MCMC sampling methods for Row 3 may take a
long time to mix. One may start from a training sample and let the Markov chain evolve
for a short time, and then accept or reject the resulting sample so as to satisfy detailed
balance (Carreira-Perpi˜ n´ an and Hinton, 2005).
Row 5: Pairwise hinge , also known as triplet loss, drives the energy of the correct
output to be lower than the energy of the contrastive output by at least a margin m(y, ˆy)
which may grow with a measure of divergence between y and ˆy. The difficulty is to find
suitable contrastive samples whose energy is low and “threatening”, a task sometimes called
“hard negative mining”.
Rows 6-8: Min-hinge, Square-hinge, Square-exp . Assumes that the energy has
a lower bound. minimizes the energy of the correct output and pushes the energy of con-
trastive outputs above a margin equal to m(y, ˆy) for rows 6 and 7, and infinite for row
8.
Row 8: Logistic. As with the pairwise hinge, the logistic loss maximizes the difference
between the energies of the contrastive output and the correct output. Unlike the pairwise
hinge, the difference is pushed to infinity, but with a force that decreases quickly.
Row 9: GAN . A GAN differs from other contrastive methods in the way contrastive
samples are generated. The contrastive samples are produced by a generator network that
is trained preferentially generate samples that have low energy according to the model. In
principle, any loss function can be used, as long as it increases with the energy of the correct
output, and decreases with the energy of the contrastive sample.
Row 10: Denoising Auto-Encoder . A denoising AE produces contrastive samples
by corrupting outputs from training samples. The corruption can be performed by adding
noise or by masking parts of the output. The energy function is the reconstruction error,
61

hence the energy is trained to be zero on the data manifold, and to grow with D(y, ˆy) as ˆy
moves away fromy on the data manifold.
Method Energy ˆy Generation Loss
1 Max Likelihood discrete y exhaustive Fw(x,y ) + log∑
y′∈Yexp(−Fw(x,y′))
2 Max Likelihood tractable exhaustive Fw(x,y ) + log
∫
y′∈Yexp(−Fw(x,y′))
3 Max likelihood any MC or MCMC Fw(x,y )−Fw(x, ˆy)
4 Contr. Divergence any trunc’d MCMC Fw(x,y )−Fw(x, ˆy)
5 Pairwise Hinge any most offending [Fw(x,y )−Fw(x, ˆy) +m(y, ˆy)]+
6 Min-Hinge positive most offending Fw(x,y ) + [m(y, ˆy)−Fw(x, ˆy)]+
6 Square-Hinge divergence most offending Fw(x,y )2 +
(
[m(y, ˆy)−Fw(x, ˆy)]+)2
7 Square-Exp any most offending Fw(x,y )2 + exp(−βFw(x, ˆy))
8 Logistic any most offending log(1 + exp(Fw(x,y )−Fw(x, ˆy))
9 GAN any ˆy =gu(z) H(Fw(x,y ),Fw(x, ˆy),m (y, ˆy))
10 Denoising AE D(y,gw(y)) ˆy =N(y) D(y,gw(ˆy)
Table 1: A of list of contrastive methods and loss functions to train energy-based models. They
are all use loss functions with two terms, one that pushes down on the energy of a training sample,
and one the pulls up the energies of one or several contrastive samples.
They differ by the strategy they employ to generate contrastive samples, and by the precise form of
the loss function.
Exact or approximate Maximum Likelihood methods (rows 1-4) are used whenever the model needs to
produce probability estimates. When the second term is intractable, its gradient may be approximated
through Monte-Carlo methods, which can be seen as particular ways to produce ˆy. Many contrastive
self-supervised methods for joint embedding architectures (Siamese nets) use Row 1 (InfoNCE).
A number of contrastive methods (Rows 5-8) are based on finding a ˆy that is “highly offending”,
meaning different from the desired y, yet given a low energy by the model. Pairs of energies for y
and ˆy are fed to a loss function that pushes the former to low values and the latter to higher values.
This can be done with a variety of losses including hinge loss.
GANs (row 9) are contrastive methods in which the contrastive samples are produced by a generator
network whose input is a random vector. The generator is trained to produce samples to which the
model currently attributes a low energy, but should attribute a high energy.
Denoising Auto-Encoders (row 10) apply a corruption process to training samples to produce con-
trastive samples ˆy = N(y). The energy function is the reconstruction error Fw(y) = D(y,gw(y))
whereD() is a symmetric divergence measure and gw(y) a parameterized function. By training gw()
to map ˆy to y, the energy for ˆy is trained to be equal to D(ˆy,y ), while the energy of y is trained to
be zero.
62