A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 8/12

Cleaned-text code points [98593, 111715). Bibliography omitted here; source.md includes it.

and exploration into corners of the state space in which the world model’s prediction are
currently inexact or uncertain.
The main open question is how much can be learned using passive observation (modes
1, 2, 4), how much requires egomotion (mode 3), and how much requires full agency (mode
5).
5 Designing and Training the Actor
The role of the actor module is threefold:
1. inferring optimal action sequences that minimize the cost, given the predictions pro-
duced by the world model for Mode-2 actions.
2. producing multiple conﬁgurations of latent variables that represent the portion of the
world state the agent does not know.
3. training policy networks for producing Mode-1 actions.
There is no conceptual diﬀerence between an action and a latent variable. The conﬁgurations
of both sets of variables must be explored by the actor. For latent variables, conﬁgurations
must be explored to plan under uncertainty. For action variables conﬁgurations must be
explored to produce an optimal one that minimizes the cost. In adversarial scenarios (such
as games), the latent conﬁgurations must be explored that maximize the cost. In eﬀect, the
actor plays the role of an optimizer and explorer.
When the world model and the cost are well-behaved, the actor module can use a
gradient-based optimization process to infer an optimal action sequence. To do so, it receives
estimates of the gradient of the cost computed by backpropagating gradients through the
cost and the unfolded world model. It uses those estimates to update the action sequence.
When the world model or the cost are not so well-behaved, a gradient-based search
for an optimal action sequence may fail. In this case another search/planning method
may be applied. If the action space is discrete or can be discretized, one can use dynamic
programming methods or approximate dynamic programming methods such as beam search
or Monte-Carlo tree search. In eﬀect, any planning method developed in the context of
optimal control, robotic, or “classical” AI may be used in this context.
Once an optimal action sequence is obtained through the planning / inference / opti-
mization process, one can use the actions as targets to train a policy network. The policy
network may subsequently be used to act quickly, or merely to initialize the proposed action
sequence to a good starting point before the optimization phase. Multiple policy networks
can be trained for multiple tasks.
The actor also produces conﬁgurations of latent variables. These latent variables repre-
sent the portion of the world state that the agent does not know. Ideally, the actor would
systematically explore likely conﬁgurations of the latents. Ideally, the regularizer for the
latents,R1 and R2 in Figure 17, would represent log-priors from which the latent could be
sampled. But in a similar way as the policy network, one may devise a latent amortized
inference module that learns distributions of latent variables. Good distributions would
produce predictions that are plausible. The distribution primate’s may depend on all the
variables available at that time.
37

6 Designing the Conﬁgurator
The conﬁgurator is the main controller of the agent. It takes input from all other modules
and modulates their parameters and connection graphs. The modulation can route signals,
activate sub-networks, focus attention, etc. In a scenario in which the predictor and the
upper layers of the perception encoder are transformer blocks, the conﬁgurator outputs
may constitute extra input tokens to these transformer blocks, thereby modulating their
connection graphs and functions.
The conﬁgurator module is necessary for two reasons: hardware reuse, and knowledge
sharing. There is an obvious advantage to be able to reuse the same circuit for multiple
tasks, particularly if the tasks can be accomplished sequentially, and if the amount of
resources (e.g. parameter memory) is limited. But there is another advantage: knowledge
reuse. A reasonable hypothesis is that a world model trained for a given environment can be
used for a range of diﬀerent tasks with minor changes. One can imagine a “generic” world
model for the environment with a small portion of the parameters being modulated by the
conﬁgurator for the task at hand. This would be more data eﬃcient and computationally
eﬃcient than having separate world models for each skill. The disadvantage is that the
agent can only accomplish one task at a time.
The conﬁgurator may prime the perception module for a particular task by modulating
the parameters at various levels. The human perceptual system can be primed for a par-
ticular task, such as detecting an item in a cluttered drawer, detecting fruits or preys in a
forest, reading, counting certain events, assembling two parts, etc. For tasks that require a
rapid detection of simple motifs, the conﬁgurator may modulate the weights of low-level lay-
ers in a convolutional architecture. For tasks that involve satisfying relationships between
objects (such as assembling two parts with screws) the conﬁguration may be performed by
modulating tokens in high-level transformer modules.
The predictor part of the world model must be able to perform a wide range of functions
depending on the task at hand. For predictors performing short-term predictions at a
low level of abstraction, conﬁguration may mean dynamic signal routing. In a low-level
retinotopic feature array representation, prediction may be reduced to local displacements
of individual feature vectors, accompanied with small transformations of those vectors.
This may be advantageously implemented with local gating/routing circuits. For longer-
term prediction at higher-levels of abstraction, it may be preferable to use a transformer
architecture. Transformer blocks are particularly appropriate for object-based reasoning in
which objects interact. The reason is that the function of transformer blocks is equivariant
to permutation. Thanks to that property, one does not need to worry about which object
is assigned to which input token: the result will be identical and consistent with the input
assignment. Recent work in model-based robotics have proposed to use a transformer
operating at the level of an entire trajectory, imposing constraints on the attention circuits
to conﬁgure the predictor for causal prediction or other tasks (Janner et al., 2021).
Conveniently, the function of a transformer block is easy to conﬁgure by adding extra
input tokens. Those extra inputs have the eﬀect of modulating the connection graph used
by the rest of the network, thereby allowing the speciﬁcation of a wide range of input-output
functions.
38

Perhaps the most important function of the conﬁgurator is to set subgoals for the agent
and to conﬁgure the cost module for this subgoal. As mentioned in Section 3.2, a simple
way to make the cost conﬁgurable is by modulating the weights of a linear combination of
elementary cost sub-modules. This may be appropriate for the immutable Intrinsic Cost
submodule: allowing for a complex modulation of the Intrinsic Cost may make the basic
drives of the agent diﬃcult to control, including cost terms that implement safety guardrails.
In contrast, one can imagine more sophisticated architectures allowing the Trainable Critic
part of the cost to be ﬂexibly modulated. As with the predictor, if the high-level cost is
formulated as a set of desired relationships between objects (“is the nut set on the screw?”)
one may use a transformer architecture trained to measure to what extent the state of the
world diverges from the condition to be satisﬁed. As with the predictor, extra token inputs
can be used to modulate the function.
One question that is left unanswered is how the conﬁgurator can learn to decompose
a complex task into a sequence of subgoals that can individually be accomplished by the
agent. I shall leave this question open for future investigation.
7 Related Work
Most of the ideas presented in the paper are not new, and have been discussed at length in
various forms in cognitive science, neuroscience, optimal control, robotics, AI, and machine
learning, particularly in reinforcement learning.
Perhaps the main original contributions of the paper reside in
• an overall cognitive architecture in which all modules are diﬀerentiable and many of
them are trainable.
• H-JEPA: a non-generative hierarchical architecture for predictive world models that
learn representations at multiple levels of abstraction and multiple time scales.
• a family of non-contrastive self-supervised learning paradigm that produces represen-
tations that are simultaneously informative and predictable.
• A way to use H-JEPA as the basis of predictive world models for hierarchical planning
under uncertainty.
Below is an attempt to connect the present proposal with relevant prior work. Given
the scope of the proposal, the references cannot possibly be exhaustive.
7.1 Trained World Models, Model-Predictive Control, Hierarchical PLan-
ning
The use of models in optimal control goes back to the early days with the Kelley-Bryson
method (see (Bryson and Ho, 1969) and reference therein, or the review (Morari and Lee,
1997)). Some methods allowed for online system identiﬁcation (Richalet et al., 1978).
Using neural networks to learn models for control is an old idea, going back to the early
1990s (Jordan and Rumelhart, 1992; Narendra and Parthasarathy, 1990; Miller et al., 1995).
39

In the context of optimal control, learning a Mode-1 like policy network is known as direct
inverse control.
The idea of Mode-2 style inference over actions using a predictive model in the context
of reinforcement learning is also an old idea with, for example, Sutton’s Dyna architec-
ture (Sutton, 1991). See (Bertsekas, 2019) for an extensive review.
The idea of learnable models has recently enjoyed a renewal of interest in various con-
texts (Ha and Schmidhuber, 2018b; Ha and Schmidhuber, 2018a; Hafner et al., 2018; Hafner
et al., 2020) (see (Moerland et al., 2020) for a recent survey of model-based reinforcement
learning).
Learning world models is particularly important in the context of robotics, especially
for grasping and manipulation where sample eﬃciency is paramount and simulation is often
inaccurate. In fact, because classical reinforcement learning approaches require too many
trials for real-world applications, interesting advances in learned model for control have
emerged from ML-based robotics research (Agrawal et al., 2016; Finn and Levine, 2017;
Chua et al., 2018; Srinivas et al., 2018; Yu et al., 2020; Yarats et al., 2021). For a recent
review, see (Levine, 2021) and references therein.
A diﬃcult setting is one in which the main input is visual, and a world model must be
learned from video. Early attempts to train predictive models without latent variables from
simple video produced blurry predictions (Lerer et al., 2016). To handle uncertainty in the
predictions, one can use various ﬂavors of latent variable models such as generative adver-
sarial networks (GAN) (Goodfellow et al., 2014), variational auto-encoders (VAE) (Kingma
and Welling, 2013) vector-quantized VAE (VQ-VAE) (van den Oord et al., 2017). Vari-
ations of these methods have been applied to video prediction and help represent multi-
modal outputs and reduce blurriness using GAN (Mathieu et al., 2015; Luc et al., 2020),
VAE (Babaeizadeh et al., 2017; Denton and Fergus, 2018; Henaﬀ et al., 2019), or VQ-
VAE (Walker et al., 2021). Although many of these methods have not been applied to
control problems, some have been applied to vehicle trajectory prediction for autonomous
driving (Henaﬀ et al., 2019; Mercat et al., 2020), or various robot control tasks (Oh et al.,
2015; Fragkiadaki et al., 2015; Agrawal et al., 2016; Finn et al., 2016; Nagabandi et al.,
2017; Babaeizadeh et al., 2017; Srinivas et al., 2018). Unlike the proposed JEPA, these
models are generative. The key issue of how to represent uncertainty in the prediction
remains.
The alternative to regularized latent-variable models is constrative methods, such as
Contrastive Predictive Coding (CPC) (H´ enaﬀ et al., 2019), which has been applied to
learning visual representations through video prediction (van den Oord et al., 2018).
To solve the multi-modality/blurriness problem, other works have proposed to perform
video prediction in representations spaces. In some works, the representation space is
obtained from a vision pipeline that has been trained in supervised mode, for example to
perform semantic segmentation (Luc et al., 2017; Luc et al., 2018). Unfortunately, the
requirement for a pre-trained vision pipeline reduces the general usability of these methods
for learning world models by observation.
In the same spirit as JEPA, there have been proposals for automatically learning rep-
resentations of video frames so they can be easily predicted. These proposals are generally
limited to learning low-level features, and often use reconstruction through a decoder as a
way to prevent collapse (Goroshin et al., 2015a; Srivastava et al., 2015). Some authors have
40