A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 3/12

Cleaned-text code points [29440, 42877). Bibliography omitted here; source.md includes it.

Pred(s,a) Pred(s,a) Pred(s,a)
C(s[T])
s[t] s[t+1]
action
s[0]
a[t] a[T-1]a[0]
Actor
 A(s[0])  A(s[t])   A(s[t+1])D D
C(s[t]) C(s[t+1])C(s[0])
Figure 5: Training a reactive policy module from the result of Mode-2 reasoning. Using Mode-2 is
onerous, because it mobilizes all the resources of the agent for the task at hand. It involves running
the world model for multiple time steps repeatedly. This diagram depicts how to train a policy module
A(s[t]) to approximate the action that results from Mode-2 optimization. The system first operates
in Mode-2 and produces an optimal sequence of actions (ˇa[0],..., ˇa[T ]). Then the parameters of the
policy module are adjusted to minimize a divergence D(ˇa[t]),A (s[t])) between the optimal action and
the output of the policy module. This results in a policy module that performs amortized inference,
and produces an approximation for a good action sequence. The policy module can then be used to
produce actions reactively in Mode-1, or to initialize the action sequence prior to Mode-2 inference
and thereby accelerate the optimization.
3.1.3 From Mode-2 to Mode-1: Learning New Skills
Using Mode-2 is onerous. The agent only possesses one world model “engine”. It is config-
urable by the configurator for the task at hand, but it can only be used for a single task
at a time. Hence, similarly to humans, the agent can only focus on one complex task at a
time.
Mode-1 is considerably less onerous, since it only requires a single pass through a policy
module. The agent may possess multiple policy modules working simultaneously, each
specialized for a particular set of tasks.
The process described in Figure 5 shows how a policy module A(s[t]) can be trained
to produce approximations of the optimal actions resulting from Mode-2 reasoning. The
system is run on Mode-2, producing an optimal action sequence (ˇ a[0],..., ˇa[t],..., ˇa[T ]).
Then, the parameters of the policy module A(s[t]) are updated to minimize a divergence
measure between its output and the optimal action at that time D(ˇa[t],A (s[t])). Once
properly trained, the policy module can be used to directly produce an action in Mode-1
˜a[0] =A(s[0]). It can also be used to recursively compute an initial action sequence proposal
before Mode-2 optimization:
s[t + 1] = Pred(s[t],a [t]) ; ˜a[t + 1] =A(s[t + 1])
The policy module can be seen as performing a form of amortized inference.
This process allows the agent to use the full power of its world model and reasoning
capabilities to acquire new skills that are then “compiled” into a reactive policy module
that no longer requires careful planning.
3.1.4 Reasoning as Energy Minimization
The process of elaborating a suitable action sequence in Mode-2 can be seen as a form
of reasoning. This form of reasoning is based on simulation using the world model, and
12

TC1(s)
 s
 IC1(s)  IC2(s)  ICk(s)... TC2(s) TCl(s)...
Intrinsic Cost (IC) Trainable Cost / Critic (TC)
Figure 6: Architecture of the cost module. The cost module comprises the intrinsic cost module
which is immutable ICi(s) (left) and the critic or Trainable Cost TCj(s) (right), which is trainable.
Both IC and TC are composed of multiple submodules whose output energies are linearly combined.
Each submodule imparts a particular behavioral drive in the agent. The weights in the linear com-
bination, ui and vj, are determined by the configurator module and allow the agent to focus on
different subgoals at different times.
optimization of the energy with respect to action sequences. More generally, the “actions”
can be seen as latent variables representing abstract transformations from one state to the
next. This type of planning though simulation and optimization may constitute the kind
of reasoning that is most frequent in natural intelligence.
Many classical forms of reasoning in AI can actually be formulated as optimization
problems (or constraint satisfaction problems). It is certainly the case for the kind of
probabilistic inference performed with factor graphs and probabilistic graphical models.
The proposed architecture is, in fact, a factor graph in which the cost modules are log
factors. But the kind of reasoning that the proposed architecture enables goes beyond
traditional logical and probabilistic reasoning. It allows reasoning by simulation and by
analogy.
3.2 The Cost Module as the Driver of Behavior
The overall architecture of the cost module is shown in Figure 6. It is composed of the
intrinsic cost module which is immutable IC i(s) and the critic or Trainable Cost TC j(s),
which is trainable. Both IC and TC are composed of multiple submodules whose output
energies are linearly combined
C(s) = IC( s) + TC(s) (1)
IC(s) =
k∑
i=1
uiICi(s) (2)
13

TC(s) =
l∑
j=1
vjTCj(s) (3)
Each submodule imparts a particular behavioral drive to the agent. The weights in the
linear combination, ui and vj, are modulated by the configurator module and allow the
agent to focus on different subgoals at different times.
The intrinsic cost module (IC) is where the basic behavioral nature of the agent is
defined. It is where basic behaviors can be indirectly specified.
For a robot, these terms would include obvious proprioceptive measurements corre-
sponding to “pain”, “hunger”, and “instinctive fears”, measuring such things as external
force overloads, dangerous electrical, chemical, or thermal environments, excessive power
consumption, low levels of energy reserves in the power source, etc.
They may also include basic drives to help the agent learn basic skills or accomplish its
missions. For example, a legged robot may comprise an intrinsic cost to drive it to stand
up and walk. This may also include social drives such as seeking the company of humans,
finding interactions with humans and praises from them rewarding, and finding their pain
unpleasant (akin to empathy in social animals). Other intrinsic behavioral drives, such as
curiosity, or taking actions that have an observable impact, may be included to maximize
the diversity of situations with which the world model is trained (Gottlieb et al., 2013)
The IC can be seen as playing a role similar to that of the amygdala in the mammalian
brain and similar structures in other vertebrates.
To prevent a kind of behavioral collapse or an uncontrolled drift towards bad behaviors,
the IC must be immutable and not subject to learning (nor to external modifications).
The role of the critic (TC) is twofold: (1) to anticipate long-term outcomes with minimal
use of the onerous world model, and (2) to allow the configurator to make the agent focus
on accomplishing subgoals with a learned cost.
In general, the behavioral nature of an AI agent can be specified in four ways:
1. by explicitly programming a specific behavior activated when specific conditions are
met
2. by defining an objective function in such a way that the desired behavior is executed
by the agent as a result of finding action sequences that minimize the objective.
3. by training the agent to behave a certain way through direct supervision. The agent
observes the actions of an expert teacher, and trains a Mode-1 policy module to
reproduce it.
4. by training the agent through imitation learning. The agent observes expert teachers,
and infers an objective function that their behavior appears to be optimizing when
they act. This produces a critic submodule for Mode-2 behavior. This process is
sometimes called inverse reinforcement learning.
The second method is considerably simpler to engineer than the first one, because it
merely requires to design an objective, and not design a complete behavior. The second
method is also more robust: a preordained behavior may be invalidated by unexpected
conditions or a changing environment. With an objective, the agent may adapt its behavior
14

... ...
 ... ...
write write
read read
Loss
write
Short-T erm
Associative
Memory
Intrinsic Cost
Critic
State Sequence
Figure 7: Training the critic. During planning episodes, the intrinsic cost module stores triplets
(time, state, intrinsic energy): (τ,sτ,IC (sτ)) into the associative short-term memory. During critic
training episodes, the critic retrieves a past state vectorsτ, together with an intrinsic energy at a later
time IC(sτ+δ). In the simplest scenario, the critic adjusts its parameters to minimize a divergence
measure between the target IC(stau+δ) and the predicted energy C(sτ). In more complex schemes,
it may use combinations of future intrinsic energies as targets. Note that the state sequence may
contain information about the actions planned or taken by the agent.
to satisfy the objective despite unexpected conditions and changes in the environment. The
second method exploits the learning and inference abilities of the agent to minimize the
amount of priors hard-wired by the designer that are likely to be brittle.
3.3 Training the Critic
An essential question is how to train the critic.
The principal role of the critic is to predict future values of the intrinsic energy. To do
so, it uses the short-term memory module. This module is an associative memory in which
the intrinsic cost module stores triplets (time, state, intrinsic energy): ( τ,sτ,IC (sτ)). The
stored states and corresponding intrinsic energies may correspond to a perceived state or to
a state imagined by the world model during a Mode-2 episode. The memory may retrieve
a state sτgiven a time τ, and may retrieve an energy IC (sτ) given a time τor a state sτ.
With a suitable memory architecture, the retrieval may involve interpolations of keys and
retrieved values. The process is shown in Figure 7
The critic can be trained to predict future intrinsic energy values by retrieving a past
state vector sτtogether with an intrinsic energy at a later time IC( sτ+δ). The parameters
of the critic can then be optimized to minimize a prediction loss, for example ||IC (sτ+δ)−
TC (sτ)||2. This is a simple scenario. More complex schemes can be devised to predict
expectations of discounted future energies, or distributions thereof. Note that the state
vectors may contain information about the actions taken or imagined by the actor.
At a general level, this is similar to critic training methods used in such reinforcement
learning approaches as A2C.
15

The short-term memory can be implemented as the memory module in a key-value
memory network ??: a query vector is compared to a number of key vectors, producing
a vector of scores. The scores are normalized and used as coefficients to output a linear
combination of the stored values. It can be seen as a “soft” associative memory capable of
interpolation. One advantage of it is that, with a proper allocation scheme of new key/value
slots, it is capable of one-shot learning, yet can interpolate between keys and is end-to-end
differentiable.
4 Designing and Training the World Model
Arguably, designing architectures and training paradigms for the world model constitute
the main obstacles towards real progress in AI over the next decades. One of the main
contributions of the present proposal is precisely a hierarchical architecture and a training
procedure for world models that can represent multiple outcomes in their predictions.
Training the world model is a prototypical example of Self-Supervised Learning (SSL),
whose basic idea is pattern completion. The prediction of future inputs (or temporarily
unobserved inputs) is a special case of pattern completion. In this work, the primary
purpose of the world model is seen as predicting future representations of the state of the
world.
There are three main issues to address. First, quite evidently, the quality of the world
model will greatly depend on the diversity of state sequences, or triplets of (state, action,
resulting state) it is able to observe while training. Second, because the world is not entirely
predictable, there may be multiple plausible world state representations that follow a given
world state representation and an action from the agent. The world model must be able to
meaningfully represent this possibly-infinite collection of plausible predictions. Third, the
world model must be able to make predictions at different time scales and different levels
of abstraction.
The first issue touches on one of the main questions surrounding learning for sequential
decision processes: the diversity of the “training set” depends on the actions taken. The
issue is discussed in Section 4.10 below.
The second issue is even more dire: the world is not entirely predictable. Hence, the
world model should be able to represent multiple plausible outcomes from a given state and
(optionally) an action. This may constitute one of the most difficult challenges to which
the present proposal brings a solution. This issue is discussed in Section 4.8 below.
The third issue relates to the problem of long-term prediction and planning. Humans
plan complex goals at an abstract level and use high-level descriptions of the world states
and actions to make predictions. High-level goals are then decomposed into sequences of
more elementary sequences of subgoals, using shorter-term prediction from the world model
to produce lower-level actions. This decomposition process is repeated all the way down
to millisecond-by-millisecond muscle control, informed by local conditions. The question of
how world models could represent action plans at multiple time scales and multiple levels
of abstraction is discussed in Section 4.6
16