A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 6/12

Cleaned-text code points [70956, 84889). Bibliography omitted here; source.md includes it.

type of collapse can be understood with the following thought experiment. Imagine that
z has the same dimension as sy. Assume that the predictor is a parameterized function
(e.g. a neural net) that can choose to ignore sx and to simply copy z on its output ˜sy =z.
For any sy it is possible to set ˇz = sy, which would make the energy D(sy, ˜sy) zero. This
corresponds to a totally flat and collapsed energy surface.
How do we prevent this collapse from happening?
By limiting or minimizing the information content of the latent variable.
How can this be done?
By making z discrete, low-dimensional, sparse, or noisy, among other methods.
A few concrete examples may help build an intuitive understanding of the phenomenon.
Suppose that D(sy, ˜sy) =||sy−˜sy||2 and that z is discrete with K possible integer values
[0,K−1]. For a given x, there can be only K possible values of ˜sy:
Pred(sx, 0), Pred(sx, 1),..., Pred(sx,K−1).
Hence, these can be the only values of sy with zero energy, and there are only K of them.
Consider a point sy that starts from Pred(sx, 0) and moves towards Pred(sx, 1). Its energy
will start from zero, increase quadratically as sy moves away from Pred( sx, 0), until sy.
Whensy becomes closer to Pred(sx, 1) than to Pred(sx, 0), the energy will decrease, reaching
zero when sy reaches Pred(sx, 1). In representation space, the energy will be the minimum
of K quadratic energy wells.
Similarly, imagine that z is a vector whose dimension d is lower than that of ˜sy. Then,
assuming that Pred(sx,z ) is a smooth function of z, the set of possible predictions will be
at most a d-dimensional manifold in the space of sy.
More to the point, imagine that the energy function is augmented by a regularization
term onz of the formR(z) =α∑d
i=1|zi|, i.e. theL1 norm ofz. This will drive ˇz to be sparse.
As with classical sparse coding, this will cause the region of low energy to be approximated
by a union of low-dimensional manifolds (a union of low-dimensional linear subspaces if
Pred(sx,z ) is linear in z), whose dimension will be minimized by the L1 regularizer.
Making z a stochastic sample from a distribution whose entropy is maximized will also
have a proper regularization effect. This is the basis of Variational Auto-Encoders and
similar models.
A more complete discussion of regularizers that can minimize the information content
of latent variables is beyond the scope of this paper. For now, we can mention four classes
of methods: discretization/quantification (e.g. as in VQ-VAE (Walker et al., 2021), dimen-
sionality/rank minimization (e.g. as in Implicit Rank-Minimizing AE (Jing et al., 2020),
sparsification (as in linear sparse modeling (Olshausen and Field, 1996), LISTA (Gregor
and LeCun, 2010b), and non-linear sparse modeling (Evtimova and LeCun, 2022)), and
fuzzyfication (as in noisy AE (Doi et al., 2007), VAE (Kingma and Welling, 2013), and
variants used in control problems (Henaff et al., 2019)).
The ability of the JEPA to predict in representation space makes it considerably prefer-
able to generative models that directly produce a prediction of y. In a video prediction
scenario, it is essentially impossible to predict every pixel value of every future frame. The
details of the texture on a carpet, the leaves of a tree moving in the wind, or the ripples on
a pond, cannot be predicted accurately, at least not over long time periods and not without
consuming enormous resources. A considerable advantage of JEPA is that it can choose to
27

Figure 14: Training a JEPA with VICReg.
VICReg is a non sample-contrastive method for training embeddings. The information content of the
representationssx and sy is maximized by first mapping them to higher-dimensional embeddings vx
and vy through an expander (e.g. a trainable neural net with a few layers). The loss function drives
the covariance matrix of the embeddings towards the identity (e.g. computed over a batch).
VICReg can be seen as a dimension-contrastive method as opposed to sample-contrastive methods.
ignore details of the inputs that are not easily predictable . Yet, Criteria 1 and 2 will ensure
that the information content of the ignored details are kept to a minimum.
How can we implement Criteria 1 and 2?
In other words, given a parameterized deterministic encoding function sy = Encw(y), how
do we maximize the information content of sy?
If Encw(y) is invertible, sy contains all the information about y, but that may be sub-
optimal for Criterion 3, as sy will contain many irrelevant or hard-to-predict details about
y. More precisely, sy is maximally informative about y if the function Encw(y) is minimally
surjective, i.e. if the volume of sets of y that map to the same sy is minimal. The same
reasoning applies to the x encoder. To turn this criterion into a differentiable loss, we need
to make some assumptions.
4.5.1 VICReg
The VICReg method (Bardes et al., 2021) makes a few assumptions about the distributions
ofsx andsy. A graphical representation is shown in Figure 14. To maximize the information
content ofsx, VICReg uses the following two sub-criteria: (1) the components ofsx must not
be constant, (2) the components ofsx must be as independent of each other as possible. This
is approximated by first non-linearly mapping sx andsy to higher-dimensional embeddings
vx and vy through a trainable expander module (e.g. a neural net with a few layers), and
using a loss function with two differentiable loss terms computed over a batch of samples:
28

1. Variance: a hinge loss that maintains the standard deviation of each component of
sy and vy above a threshold over a batch.
2. Covariance: a covariance loss in which the covariance between pairs of different
components of vy are pushed towards zero. This has the effect of decorrelating the
components of vy, which will in turn make the components of sy somewhat indepen-
dent.
The same criteria are applied to sx and vx separately.
The third criterion of VICReg is the representation prediction error D(sy, ˜sy). In the
simplest implementations of VICReg, the predictor is constant (equal to the identity func-
tion), making the representations invariant to the transformation that turns x into y. In
more sophisticated versions, the predictor may have no latent variable, or may depend on
a latent variable that is either discrete, low dimensional, or stochastic.
The fourth criterion is necessary when the predictor uses a latent variable whose infor-
mation content must be minimized, for example a vector whose dimension approaches or
surpasses that of ˜sy.
A simple instantiation of VICReg to learn invariant representations consists in making
x andy be different views (or distorted versions) of the same content, setting the predictor
to the identity function, and defining D(sy, ˜sy) =D(sy,sx) =||sy−sx||2.
Inferring the latent variable through gradient-based methods may be onerous. But the
computational cost can be greatly reduced by using amortized inference, as explained in
Appendix 8.3.3.
While contrastive methods ensure that representations of different inputs in a batch
are different, VICReg ensures that different components of representations over a batch are
different. VICReg is contrastive over components, while traditional contrastive methods
are contrastive over vectors, which requires a large number of contrastive samples.
But the most promising aspect of JEPA trained with VICReg and similar non-contrastive
methods is for learning hierarchical predictive world models, as we examine in the next sec-
tion.
4.5.2 Biasing a JEPA towards learning “useful” representations
With the training criteria listed above, the JEPA finds a trade-off between the completeness
and the predictability of the representations. What is predictable and what does not get
represented is determined implicitly by the architectures of the encoders and predictor.
They determine a inductive bias that defines what information is predictable or not.
But it would be useful to have a way to bias the system towards representations that
contain information relevant to a class of tasks. This can be done by adding prediction
heads that take ˜sy as input and are trained to predict variables that are easily derived from
the data and known to be relevant to the task.
4.6 Hierarchical JEPA (H-JEPA)
JEPA models trained non-contrastively may constitute our best tool for learning world
models that are able to learn relevant abstractions. When trained with VICReg and similar
29

x y
JEPA-1
JEPA-2
Figure 15: Hierarchical JEPA (H-JEPA)
The ability of the JEPA to learn abstract representations in which accurate prediction can be per-
formed allows hierarchical stacking. In this diagram JEPA-1 extracts low-level representations and
performs short-term predictions. JEPA-2 takes the representations extracted by JEPA-1 as inputs
and extracts higher-level representations with which longer-term predictions can be performed. More
abstract representations ignore details of the inputs that are difficult to predict in the long term,
enabling them to perform longer-term predictions with coarser descriptions of the world state.
criteria, a JEPA can choose to train its encoders to eliminate irrelevant details of the inputs
so as to make the representations more predictable. In other words, a JEPA will learn
abstract representations that make the world predictable. Unpredictable details will be
eliminated by the invariance properties of the encoder, or will be pushed into the predictor’s
latent variable. The amount of information thereby ignored will be minimized by the
training criteria and by the latent variable regularizer.
It is important to note that generative latent-variable models are not capable of elimi-
nating irrelevant details, other than by pushing them into a latent variable. This is because
they do not produce abstract (and invariant) representations of y. This is why we advocate
against the use of generative architectures.
The capacity of JEPA to learn abstractions suggests an extension of the architecture to
handle prediction at multiple time scales and multiple levels of abstraction. Intuitively, low-
level representations contain a lot of details about the input, and can be used to predict in
the short term. But it may be difficult to produce accurate long-term predictions with the
same level of details. Conversely high-level, abstract representation may enable long-term
predictions, but at the cost of eliminating a lot of details.
30

Let’s take a concrete example. When driving a car, given a proposed sequence of actions
on the steering wheel and pedals over the next several seconds, drivers can accurately
predict the trajectory of their car over the same period. The details of the trajectory over
longer periods are harder to predict because they may depend on other cars, traffic lights,
pedestrians, and other external events that are somewhat unpredictable. But the driver
can still make accurate predictions at a higher level of abstraction: ignoring the details of
trajectories, other cars, traffic signals, etc, the car will probably arrive at its destination
within a predictable time frame. The detailed trajectory will be absent from this level of
description. But the approximate trajectory, as drawn on a map, is represented. A discrete
latent variable may be used to represent multiple alternative routes.
Figure 15 shows a possible architecture for multilevel, multi-scale world state prediction.
Variablesx0,x 1,x 2 represent a sequence of observations. The first-level network, denoted
JEPA-1 performs short-term predictions using low-level representations. The second-level
network JEPA-2 performs longer-term predictions using higher-level representations. One
can envision architectures of this type with many levels, possibly using convolutional and
other modules, and using temporal pooling between levels to coarse-grain the representation
and perform longer-term predictions. Training can be performed level-wise or globally, using
any non-contrastive method for JEPA.
I submit that the ability to represent sequences of world states at several levels of
abstraction is essential to intelligent behavior. With multi-level representations of world
states and actions, a complex task can be decomposed into successively more detailed sub-
tasks, instantiated into actions sequences when informed by local conditions. For example,
planning a complex task, like commuting to work, can be decomposed into driving to the
train station, catching a train, etc. Driving to the train station can be decomposed into
walking out of the house, starting the car, and driving. Getting out of the house requires
standing up, walking to the door, opening the door, etc. This decomposition descends all
the way down to millisecond-by-millisecond muscle controls, which can only be instantiated
when the relevant environmental conditions are perceived (obstacles, traffic lights, moving
objects, etc).
4.7 Hierarchical Planning
If our world model can perform predictions hierarchically, can it be used to perform Mode-2
reasoning and planning hierarchically?
Hierarchical planning is a difficult topic with few solutions, most of which require that
the intermediate vocabulary of actions be predefined. But if one abides by the deep learning
philosophy, those intermediate representations of action plans should also be learned .
Figure 16 shows a possible architecture for hierarchical Mode-2 planning that can exploit
the hierarchical nature of a multi-scale world model.
A percept is encoded into representations at multiple levels of abstractions by a cascade
of encoders:
s[0] = Enc1(x); s2[0] = Enc2(s[0]); ... (14)
Prediction takes place at all levels. Higher levels perform longer-term prediction, while lower
levels perform shorter-term predictions. The overall task is defined by a high-level objective,
depicted as C(s2[4]) in the diagram. The top level infers a sequence of high-level actions
31