A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 4/12

Cleaned-text code points [42877, 57677). Bibliography omitted here; source.md includes it.

4.1 Self-Supervised Learning
Self-Supervised Learning (SSL) is a paradigm in which a learning system is trained to
capture the mutual dependencies between its inputs. Concretely, this often comes down to
training a system to tell us if various parts of its input are consistent with each other.
For example, in a video prediction scenario, the system is given two video clips, and
must tell us to what degree the second video clip is a plausible continuation of the ﬁrst one.
In a pattern completion scenario, the system is given part of an input (image, text, audio
signal) together with a proposal for the rest of the input, and tells us whether the proposal
is a plausible completion of the ﬁrst part. In the following, we will denote the observed part
of the input by x and the possibly-unobserved part by y.
Importantly, we do not impose that the model be able to predicty from x. The reason
is that there may be an inﬁnite number of y that are compatible with a given x. In a video
prediction setting, there is an inﬁnite number of video clips that are plausible continuations
of a given clip. It may be diﬃcult, or intractable, to explicitly represent the set of plausible
predictions. But it seems less inconvenient to merely ask the system to tell us if a proposed
y is compatible with a given x.
A general formulation can be done with the framework of Energy-Based Models (EBM).
The system is a scalar-valued function F (x,y ) that produces low energy values when x and
y are compatible and higher values when they are not. The concept is depicted in Figure 8.
Data points are black dots. The energy function produces low energy values around the data
points, and higher energies away from the regions of high data density, as symbolized by the
contour lines of the energy landscape. The EBM implicit function formulation enables the
system to represent multi-modal dependencies in which multiple values of y are compatible
with a given x. The set of y compatible with a given x may be a single point, multiple
discrete points, a manifold, or a collection of points and manifolds.
To enable Mode-2 planning, a predictive world model should be trained to capture the
dependencies between past and future percepts. It should be able to predict representations
of the future from representations of the past and present . The general learning principle
is as follows: given two inputs x and y, learn two functions that compute representations
sx =gx(x) and sy =gy(y) such that (1) sx and sy are maximally informative about x and
y and (2) sy can easily be predicted from sx. This principle ensures a trade-oﬀ between
making the evolution of the world predictable in the representation space, and capturing as
much information as possible about the world state in the representation.
What concepts could such an SSL system learn by being trained on video? Our hypoth-
esis is that a hierarchy of abstract concepts about how the world works could be acquired.
Learning a representation of a small image region such that it is predictable from neigh-
boring regions surrounding it in space and time would cause the system to extract local
edges and contours in images, and to detect moving contours in videos. Learning a repre-
sentation of images such that the representation of a scene from one viewpoint is predictable
from the representation of the same scene from a slightly diﬀerent viewpoint would cause
the system to implicitly represent a depth map. A depth map is the simplest way to explain
how a view of a scene changes when the camera moves slightly. Once the notion of depth has
been learned, it would become simple for the system to identify occlusion edges, as well as
the collective motion of regions belonging to a rigid object. An implicit representation of 3D
17

time or space →
Energy
Function
x
F(x,y)
y
x
y
Figure 8: Self-Supervised Learning (SSL) and Energy-Based Models (EBM). SSL is a learning
paradigm in which a learning system is trained to “ﬁll in the blanks”, or more precisely to capture the
dependencies between observed parts of the input and possibly unobserved parts of the input. Part of
the input signal is observed and denoted x (in pink), and part of the input signal is either observed
or unobserved and denoted y (in blue). In a temporal prediction scenario, x represents past and
present observations, and y represent future observations. In a general pattern completion scenario,
various parts of the input may be observed or unobserved at various times. The learning system is
trained to capture the dependencies between x and y through a scalar-valued energy function F (x,y )
that takes low values when x and y are consistent or compatible, and higher values if x and y are
inconsistent or incompatible. In a video prediction scenario, the system would produce a low energy
value if a video clip y is a plausible continuation of the video clip x. This energy-based model (EBM)
formulation enables the system to represent multi-modal dependencies in which multiple values of y
(perhaps an inﬁnite set) may be compatible with a given x. In the right panel, an energy landscape
is represented in which dark discs represent data points, and closed lines represents contours (level
sets) of the energy function.
objects may spontaneously emerge. Once the notion of object emerges in the representation,
concepts like object permanence may become easy to learn: objects that disappear behind
others due to parallax motion will invariably reappear. The distinction between inanimate
and animate object would follow: inanimate object are those whose trajectories are easily
predictable. Intuitive physics concepts such as stability, gravity, momentum, may follow by
training the system to perform longer-term predictions at the object representation level.
One may imagine that through predictions at increasingly abstract levels of representation
and increasingly long time scales, more and more complex concepts about how the world
works may be acquired in a hierarchical fashion.
The idea that abstract concepts can be learned through prediction is an old one, for-
mulated in various way by many authors in cognitive science, neuroscience, and AI over
several decades. The question is how to do it, precisely.
18

Figure 9: Latent-Variable Energy-Based Model (LVEBM).
To evaluate the degree of compatibility betweenx andy, an EBM may need the help of a latent variable
z. The latent variable can be seen as parameterizing the set of possible relationships between an x
and a set of compatible y. Latent variables represent information about y that cannot be extracted
from x. For example, if x is a view of an object, and y another view of the same object, z may
parameterize the camera displacement between the two views. Inference consists in ﬁnding the latent
that minimizes the energy ˇz = argminz∈ZEw(x,y,z ). The resulting energy Fw(x,y ) = Ew(x,y, ˇz)
only depends on x and y. In the dual view example, inference ﬁnds the camera motion that best
explains how x could be transformed into y.
4.2 Handling Uncertainty with Latent Variables
As was pointed out above, one of the main issues is enabling the model to represent mul-
tiple predictions. This may require the use of a latent variable. A latent variable is an
input variable whose value is not observed but inferred. A latent variable can be seen as
parameterizing the set of possible relationships between an x and a set of compatible y.
Latent variables are used to represent information about y that cannot be extracted from
x.
Imagine a scenario in which x is a photo of a scene, and y a photo of the same scene
from a slightly diﬀerent viewpoint. To tell whether x and y are indeed views from the
same scene, one may need to infer the displacement of the camera between the two views.
Similarly, if x is a picture of a car coming to a fork in the road, and y is a picture of the
same car a few seconds later on one of the branches of the fork, the compatibility between
x and y depends on a binary latent variable that can be inferred: did the car turn left or
right.
In a temporal prediction scenario, the latent variable represents what cannot be pre-
dicted about y (the future) solely from x and from past observations (the past). It should
contain all information that would be useful for the prediction, but is not observable, or
not knowable. I may not know whether the driver in front of me will turn left or right,
accelerate or brake, but I can represent those options by a latent variable.
A latent-variable EBM (LVEBM) is a parameterized energy function that depends on
x, y, and z: Ew(x,y,z ). When presented with a pair ( x,y ) the inference procedure of the
EBM ﬁnds a value of the latent variable z that minimizes the energy
ˇz = argmin
z∈Z
Ew(x,y,z ) (4)
19

This latent-variable inference by minimization allows us to eliminate z from the energy
function:
Fw(x,y ) = min
z∈Z
Ew(x,y,z ) =Ew(x,y, ˇz) (5)
Technically,Fw(x,y ) should be called a zero-temperature free energy, but we will continue
to call it the energy.
4.3 Training Energy-Based Models
Before we discuss EBM training, it is important to note that the deﬁnition of EBM does not
make any reference to probabilistic modeling . Although many EBMs can easily be turned
into probabilistic models, e.g. through a Gibbs distribution, this is not at all a necessity.
Hence the energy function is viewed as the fundamental object and is not assumed to
implicitly represent the unnormalized logarithm of a probability distribution.
Training an EBM consists in constructing an architecture (e.g. a deep neural network)
to compute the energy function Fw(x,y ) parameterized with a parameter vector w. The
training process must seek a w vector that gives the right shape to the energy function.
For a given x from the training set, a well-trained Fw(x,y ) will produce lower energies for
values ofy that are associated withx in the training set, and higher energies to other values
of y.
Given a training sample (x,y ), training an EBM comes down to devising a suitable loss
functionalL(x,y,F w(x,y )), which can be expressed directly as a function of the parameter
vector L(x,y,w ), and such that minimizing this loss will make the energy of the training
sample Fw(x,y ) lower than the energies Fw(x, ˆy) of any ˆy diﬀerent from y.
Making the energy of the training sample low is easy: it is suﬃcient for the loss to be
an increasing function of the energy, and for the energy to have a lower bound.
The diﬃcult question is how to ensure that the energies of ˆy diﬀerent from y are higher
than the energy of y. Without a speciﬁc provision to ensure that Fw(x,y′) > Fw(x,y )
whenever ˆy⁄=y the energy landscape may suﬀer a collapse: given an x the energy landscape
could become “ﬂat”, giving essentially the same energy to all values of y.
What EBM architectures are susceptible to collapse? Whether an EBM may be
susceptible to collapse depends on its architecture. Figure 10 shows a number of standard
architectures and indicates whether they can be subject to collapse.
A regular predictive or deterministic-generative architecture (Figure 10(a)) cannot col-
lapse. For anyx, a single ˜y is produced. The energy is zero whenever y = ˜y. Any y diﬀerent
from ˜y will have a higher energy, as long as D(y, ˜y) is strictly larger than zero whenever y
is diﬀerent from ˜y.
A generative latent-variable architecture (non-deterministic generative) (Figure 10(b))
can collapse when the latent variable has too much information capacity. When the latent
variablez varies over the setZ, the prediction ˜y varies over a set Pred(sx,Z), which must
match the set of y that are compatible with x. If Z is too “large” then the region of low-
energy y may be larger than the region of high data density. If z has the same dimension
as y, the system could very well give zero energy to the entire y space.
An auto-encoder (AE) (Figure 10(c)) can collapse when the representation sy has too
much information capacity. For example, if the dimension of sy is equal or higher than that
20

d) Joint Embedding Architecture
CAN COLLAPSE
b) Generative latent-variable Architecture
CAN COLLAPSE
a) Prediction / regression
NO COLLAPSE
c) Auto-Encoder
CAN COLLAPSE
Figure 10: A few standard architectures and their capacity for collapse.
(a) Deterministic generative architecture: cannot collapse because it can only produce a single out-
put. For a given x, only one value of y may have zero energy: y = ˜y. Other values of y will have
higher energy if D(u, ˜y) is larger than zero when y⁄= ˜y.
(b) Non-deterministic generative architecture: can collapse when the latent variable has excessive
information capacity. If for a given x and for all y there exists a z that produces zero prediction
energy (e.g. if z has the same or higher dimension as y), the entire y space will have low energy.
The information capacity of z should be just enough so that varying z over its set will produce all
the plausible ˜y for a given x.
(c) Auto-encoder: can collapse if the system learns the identity function or if it can correctly recon-
struct a region of y space that is much larger than the region of high data density, thereby giving low
energy to an overly large region.
(d) Simple joint embedding architecture: can collapse if the encoders ignore the inputs and produce
representations that remain constant and equal, or if the encoders are invariant over overly broad
regions of the space.
of y, the AE could learn the identity function, producing a reconstruction error equal to
zero over the entire y space.
Lastly, a Joint Embedding Architecture (JEA) (Figure 10(d)) can collapse when the
information carried by sx and/or sy are insuﬃcient. If the encoders ignore the inputs, and
produce constant and equal codes sx =sy, the entire space will have zero energy.
These are but a few examples of architectures.
How do we design the loss to prevent collapse? There are two approaches:
contrastive methods and regularized methods. In the following, I will argue that contrastive
methods have ﬂaws and that regularized (non contrastive) methods are much more likely
to be preferable in the long run.
Contrastive methods consist in using a loss functional whose minimization has the
eﬀect of pushing down on the energies of training samples ( x,y ), and pulling up on the
energies of suitably-hallucinated “contrastive” samples ( x, ˆy). The contrastive sample ˆ y
should be picked in such a way as to ensure that the EBM assigns higher energies to points
outside the regions of high data density. This translates into designing a loss that is an
increasing function of Fw(x,y ) and a decreasing function of Fw(x, ˆy), at least whenever
Fw(x, ˆy) is not suﬃciently higher than Fw(x,y ). There are many such contrastive loss
21