A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 5/12

Cleaned-text code points [57677, 70956). Bibliography omitted here; source.md includes it.

Contrastive
Method
Regularized
Method
Low energy
region
Training
samples
Contrastive
samples
x
x
x
y
y
y
Figure 11: Contrastive and regularized methods for EBM training. A conceptual diagram of an
energy landscape is shown on the left. Training samples are blue dots. The region of low energy is
shown in orange (a level set of the energy function).
Contrastive methods (top right) push down on the energy of training samples (blue dots) and
pulls up on the energies of suitably-placed contrastive samples (green dots).
regularized methods (bottom right) push down on the energy of training samples and use a reg-
ularizer term that minimizes the volume of low-energy regions. This regularization has the effect of
“shrink-wrapping” the regions of high data density within the low-energy regions, to the extent that
the flexibility of the energy function permits it.
One issue with contrastive methods is that the energy will only be pulled up wherever contrastive
samples have been placed. One must devise methods that preferentially place contrastive samples in
regions of low energy, which is what Monte-Carlo and MCMC methods do. However, a disadvantage
of contrastive methods is that the number of contrastive samples necessary to make an energy surface
adopt a good shape may grow exponentially with the dimension of y space.
functions, some of them taking a single triplet ( x,y, ˆy), others requiring a batch of positive
and contrastive values of y.
A simple example of contrastive loss functions is as follows:
L(w,x,y, ˆy) =H(Fw(x,y ),Fw(x, ˆy),m (y, ˆy)) (6)
whereH is an increasing function ofFw(x,y ), and a decreasing function ofFw(x, ˆy) whenever
the latter is less than the former plus a positive margin function m(y, ˆy). A simple instance
of such loss is the distance-dependent hinge loss:
L(w,x,y, ˆy) =
[
Fw(x,y )−Fw(x, ˆy) +µ||y−ˆy||2
]+
(7)
where [a]+ is the identity when a is positive and zero otherwise. This makes the energy
grow at least quadratically with the distance to the data manifold. Other contrastive loss
22

functionals take multiple contrastive samples into consideration:
L(w,x,y, ˆy[1],..., ˆy[K]) =H(Fw(x,y ),Fw(x, ˆy[1]),...,Fw(x, ˆy[K])) (8)
Which must be an increasing function of the first argument, and a decreasing function of
all other arguments. An example of such loss is the popular InfoNCE:
L(w,x,y, ˆy[1],..., ˆy[K]) =Fw(x,y ) + log
[
exp(−Fw(x,y )) +
K∑
k=1
exp(−Fw(x, ˆy[k]))
]
(9)
Contrastive methods are very popular, particularly for Siamese network architectures
trained with pairs wherex is a distorted or corrupted version ofy and ˆy another random (or
suitably chosen) training sample. This includes such methods as the original Siamese net,
as well as more recent methods including DrLIM, PIRL, MoCO, SimCLR, CPT, and others.
Contrastive methods also include such classical methods as probabilistic models trained with
maximum likelihood that are not automatically normalized. Contrastive samples ˆy are often
produced using Monte Carlo methods, Markov-Chain Monte Carlo methods, or approximate
versions thereof, such as Contrastive Divergence. Generative Adversarial Networks can also
be seen as contrastive methods in which the ˆ y are produced by the trainable generator
network. Denoising Auto-Encoders and their special case, Masked Auto-Encoders, are also
examples of contrastive training methods in which the ˆy is generated by corrupting the clean
y. A more detailed discussion of various contrastive methods is given in appendix 8.3.3.
But there are two main issues with contrastive methods. First, one has to design a
scheme to generate or pick suitable ˆy. Second, when y is in a high-dimensional space, and if
the EBM is flexible, it may require a very large number of contrastive samples to ensure that
the energy is higher in all dimensions unoccupied by the local data distribution. Because of
the curse of dimensionality, in the worst case, the number of contrastive samples may grow
exponentially with the dimension of the representation. This is the main reason why I will
argue against contrastive methods.
Regularized methods for EBM training are much more promising in the long run
than contrastive methods because they can eschew the curse of dimensionality that plagues
contrastive methods. They consist in constructing a loss functional that has the effect
of pushing down on the energies of training samples, and simultaneously minimizing the
volume of y space to which the model associates a low energy . The volume of the low-energy
region is measured by a regularization term in the energy and/or in the loss. By minimizing
this regularization term while pushing down on the energies of data points, the regions of
low energy will “shrink-wrap” the regions of high data density. The main advantage of
non-contrastive regularized methods is that they are less likely than contrastive methods
to fall victim to the curse of dimensionality. The main question is precisely how to design
such volume-minimizing regularizers. The answer depends greatly on the architecture of
the model, which is discussed in the next sections. However, non-contrastive methods have
existed for a long time. Examples include sparse modeling, sparse auto-encoders, and auto-
encoders with noisy latent variables, such as VAE.
It is important to note that contrastive and regularized methods are not incompatible
with each other, and can be used simultaneously on the same model.
How would regularized methods apply to the architectures of Figure 10(b-d)?
23

In the latent-variable generative architecture, restricting the information capacity of z
will restrict the volume of y space that can take low energy. If z is discrete with possible k
values, at most k points in y space will have zero energy. if Z is a manifold of dimension d
then the region of y space with zero energy will have at most d dimensions.
Similarly, in the auto-encoder architecture, restricting the information capacity of sy
will restrict the volume of y space that can be reconstructed with low energy.
Lastly, in the Joint Embedding Architecture, Maximizing the information that sx con-
tains about x and sy contains about y will minimize the volume of y space that can take
low energy.
In the following, we will focus on an architecture for SSL the Joint Embedding Predictive
Architectures (JEPA) which can seen as a combination of the Joint Embedding Architecture
and the Latent-Variable Generative Architecture. The JEPA is non-generative in that it
does not actually predict y, but predicts the representation of y, sy from that of x, sx.
4.4 Joint Embedding Predictive Architecture (JEPA)
The centerpiece of this paper is the Joint Embedding Predictive Architecture (JEPA). JEPA
is not generative in the sense that it cannot easily be used to predict y from x. It merely
capture the dependencies between x and y without explicitly generating predictions of y.
A generic JEPA is shown in Figure 12. The two variablesx andy are fed to two encoders
producing two presentationssx andsy. These two encoders may be different. They are not
required to possess the same architecture nor are they required to share their parameters.
This allows x and y to be different in nature (e.g. video and audio). A predictor module
predicts the representation of y from the representation of x. The predictor may depend
on a latent variable z. The energy is simply the prediction error in representation space:
Ew(x,y,z ) =D(sy, Pred(sx,z )) (10)
The overall energy is obtained by minimizing over z:
ˇz = argmin
z∈Z
Ew(x,y,z ) = argmin
z∈Z
D(sy, Pred(sx,z )) (11)
Fw(x,y ) = min
z∈Z
Ew(x,y,z ) =D(sy, Pred(sx, ˇz)) (12)
(13)
The main advantage of JEPA is that it performs predictions in representation space ,
eschewing the need to predict every detail ofy. This is enabled by the fact that the encoder
of y may choose to produce an abstract representation from which irrelevant details have
been eliminated.
But there are two ways a JEPA may represent the multiplicity of values ofy compatible
with x. The first one is invariance properties of the y encoder, the second one is the latent
variablez, as explained below.
multi-modality through encoder invariance : The encoder function sy = Enc(y)
may have invariance properties. If all the y’s in a set map to the same value of sy, all those
y’s will have identical energies. With JEPA, we lose the ability to generate outputs, but we
gain a powerful way to represent multi-modal dependencies between inputs and outputs.
24

Figure 12: The Joint-Embedding Predictive Architecture (JEPA) consists of two encoding branches.
The first branch computes sx, a representation of x and the second branch sy a representation of y.
The encoders do not need to be identical. A predictor module predicts sy from sx with the possible
help of a latent variable z. The energy is the prediction error. Simple variations of the JEPA may
use no predictor, forcing the two representations to be equal, or may use a fixed predictor with no
latent, or may use simple latents such as discrete variables.
The main advantage of JEPA is that it performs predictions in representation space, eschewing the
need to predict every detail of y, and enabling the elimination of irrelevant details by the encoders.
More precisely, the main advantage of this architecture for representing multi-modal dependencies is
twofold: (1) the encoder function sy = Enc(y) may possess invariance properties that will make it
produce the samesy for a set of different y. This makes the energy constant over this set and allows
the model to capture complex multi-modal dependencies; (2) The latent variable z, when varied over
a setZ, can produce a set of plausible predictions Pred(sx,Z) ={˜sy = Pred(sx,z )∀z∈Z}
If x is a video clip of a car approaching a fork in the road, sx and sy may represent the position,
orientation, velocity and other characteristics of the car before and after the fork, respectively, ig-
noring irrelevant details such as the trees bordering the road or the texture of the sidewalk. z may
represent whether the car takes the left branch or the right branch of the road.
multi-modality through latent variable predictor: The predictor may use a latent
variablez to capture the information necessary to predictsy that is not present insx. When
z is varied over a setZ, the predictor produces a set of plausible predictions Pred( sx,Z) =
{˜sy = Pred(sx,z )∀z∈Z}. For example, if x is a video clip of a car approaching a fork
in the road, sx and sy may represent the past and future positions, orientations, velocities
and other characteristics of the car, ignoring irrelevant details such as the trees bordering
the road or the texture of the sidewalk. The latent z may be a binary variable indicating
whether the car takes the left branch ( z = 0) or the right branch (z = 1 if the road. If the
car takes the left branch, the value z = 0 will produce a lower energy D(sy, ˜sy) than z = 1.
4.5 Training a JEPA
Like any EBM, a JEPA can be trained with contrastive methods. But, as pointed out
above, contrastive methods tend to become very inefficient in high dimension. The relevant
25

Maximize
Information
Content
Maximize
Information
Content
Minimize
Information
Content
Minimize
Prediction
Error
Figure 13: Non-contrastive training of JEPA.
The main attraction of JEPAs is that they can be trained with non-contrastive methods. The basic
principle of such training is that (1) sx should be maximally informative about x; (2) sy should
be maximally informative about y; (3) sy should be easily predictable from sx; and (4) z should
have minimal information content. Criteria 1, 2, and 4 collectively prevent a collapse of the energy
function.
Examples of such non-contrastive criteria for JEPA training include VICReg and Barlow Twins.
As with every EBM, JEPAs can also be trained with contrastive methods. But doing so runs into
the curse of dimensionality and limits the practical dimension of sy.
dimension here is that of sy, which may be considerably smaller than y, but still too high
for efficient training.
What makes JEPAs particularly interesting is that we can devise non-contrastive meth-
ods to train them. As explained in section 4.3, non-contrastive methods use regularizers
that measure the volume of space that can take low energy values. In the case of the JEPA,
this can be done through four criteria, as depicted in Figure 13:
1. maximize the information content of sx about x
2. maximize the information content of sy about y
3. make sy easily predictable from sx
4. minimize the information content of the latent variable z used in the prediction.
Criteria 1 and 2 prevent the energy surface from becoming flat by informational col-
lapse. They ensure that sx andsy carry as much information as possible about their inputs.
Without these criteria the system could choose to make sx and sy constant, or weakly in-
formative, which would make the energy constant over large swaths of the input space.
Criterion 3 is enforced by the energy termD(sy, ˜sy) and ensures thaty is predictable from
x in representation space.
Criterion 4 prevents the system from falling victim to another type of informational col-
lapse by forcing the model to predict sy with as little help from the latent as possible. This
26