A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 2/12

Cleaned-text code points [13454, 29440). Bibliography omitted here; source.md includes it.
3 A Model Architecture for Autonomous Intelligence
The proposed architecture for autonomous intelligent agents is depicted in Figure 2.
It is composed of a number of modules whose functions are described below. Some
of the modules are conﬁgurable on the ﬂy, i.e. their precise function is determined by the
conﬁgurator module. The role of the conﬁgurator is executive control: given a task to be
executed, it pre-conﬁgures the perception, the world model, the cost and the actor for the
task at hand. The conﬁgurator modulates the parameters of the modules it feeds into.
The conﬁgurator module takes input from all other modules and conﬁgures them for
the task at hand by modulating their parameters and their attention circuits. In particular,
the conﬁgurator may prime the perception, world model, and cost modules to fulﬁll a
particular goal.
The perception module receives signals from sensors and estimates the current state
of the world. For a given task, only a small subset of the perceived state of the world
is relevant and useful. The perception module may represent the state of the world in
a hierarchical fashion, with multiple levels of abstraction. The conﬁgurator primes the
perception system to extract the relevant information from the percept for the task at
hand.
The world model module constitutes the most complex piece of the architecture. Its
role is twofold: (1) estimate missing information about the state of the world not provided
by perception, (2) predict plausible future states of the world. The world model may predict
natural evolutions of the world, or may predict future world states resulting from a sequence
of actions proposed by the actor module. The world model may predict multiple plausible
world states, parameterized by latent variables that represent the uncertainty about the
world state. The world model is a kind of “simulator” of the relevant aspects of world.
What aspects of the world state is relevant depends on the task at hand. The conﬁgurator
conﬁgures the world model to handle the situation at hand. The predictions are performed
within an abstract representation space that contains information relevant to the task at
hand. Ideally, the world model would manipulate representations of the world state at
multiple levels of abstraction, allowing it to predict over multiple time scales.
A key issue is that the world model must be able to represent multiple possible predic-
tions of the world state. The natural world is not completely predictable. This is particularly
true if it contains other intelligent agents that are potentially adversarial. But it is often
true even when the world only contains inanimate objects whose behavior is chaotic, or
whose state is not fully observable.
There are two essential questions to answer when building the proposed architectures:
(1) How to allow the world model to make multiple plausible prediction and represent
uncertainty in the predictions, and (2) how to train the world model.
The cost module measures the level of “discomfort” of the agent, in the form of a
scalar quantity called the energy. The energy is the sum of two energy terms computed by
two sub-modules: the Intrinsic Cost module and the Trainable Critic module. The overall
objective of the agent is to take actions so as to remain in states that minimize the average
energy.
The Intrinsic Cost module is hard-wired (immutable, non trainable) and computes a
single scalar, the intrinsic energy that measures the instantaneous “discomfort” of the agent
7

– think pain (high intrinsic energy), pleasure (low or negative intrinsic energy), hunger, etc.
The input to the module is the current state of the world, produced by the perception
module, or potential future states predicted by the world model. The ultimate goal of the
agent is minimize the intrinsic cost over the long run . This is where basic behavioral drives
and intrinsic motivations reside. The design of the intrinsic cost module determines the
nature of the agent’s behavior. Basic drives can be hard-wired in this module. This may
include feeling “good” (low energy) when standing up to motivate a legged robot to walk,
when inﬂuencing the state of the world to motivate agency, when interacting with humans
to motivate social behavior, when perceiving joy in nearby humans to motivate empathy,
when having a full energy supplies (hunger/satiety), when experiencing a new situation to
motivate curiosity and exploration, when fulﬁlling a particular program, etc. Conversely, the
energy would be high when facing a painful situation or an easily-recognizable dangerous
situation (proximity to extreme heat, ﬁre, etc), or when wielding dangerous tools. The
intrinsic cost module may be modulated by the conﬁgurator, to drive diﬀerent behavior at
diﬀerent times.
The Trainable Critic module predicts an estimate of future intrinsic energies. Like the
intrinsic cost, its input is either the current state of the world or possible states predicted by
the world model. For training, the critic retrieves past states and subsequent intrinsic costs
stored in the associative memory module, and trains itself to predict the latter from the
former. The function of the critic module can be dynamically conﬁgured by the conﬁgurator
to direct the system towards a particular sub-goal, as part of a bigger task.
Because both sub-modules of the cost module are diﬀerentiable, the gradient of the
energy can be back-propagated through the other modules, particularly the world model,
the actor and the perception, for planning, reasoning, and learning.
The short-term memory modulestores relevant information about the past, current,
and future states of the world, as well as the corresponding value of the intrinsic cost.
The world model accesses and updates the short-term memory while temporally predicting
future (or past) states of the world, and while spatially completing missing information or
correcting inconsistent information about the current world state. The world model can
send queries to the short-term memory and receive retrieved values, or store new values of
states. The critic module can be trained by retrieving past states and associated intrinsic
costs from the memory. The architecture may be similar to that of Key-Value Memory
Networks (Miller et al., 2016) This module can be seen as playing some of same roles as the
hippocampus in vertebrates.
The actor module computes proposals for sequences of actions and outputs actions to
the eﬀectors. The actor proposes a sequence of actions to the world model. The world model
predicts future world state sequences from the action sequence, and feeds it to the cost.
Given a goal deﬁned by the cost (as conﬁgured by the conﬁgurator), the cost computes the
estimated future energy associated with the proposed action sequence. Since the actor has
access to the gradient of the estimated cost with respect to the proposed action sequence, it
can compute an optimal action sequence that minimizes the estimated cost using gradient-
based methods. If the action space is discrete, dynamic programming may be used to
ﬁnd an optimal action sequence. Once the optimization is completed, the actor outputs
the ﬁrst action (or a short sequence of actions) to the eﬀectors. This process is akin to
model-predictive control in optimal control (Bryson and Ho, 1969).
8

Pred(s,a)
C(s[1])
s[1]
action
s[0]
Actor
A(s)
a[0]
C(s[0])
Figure 3: Mode-1 perception-action episode. The perception module estimates the state of the
world s[0] = Enc(x). The actor directly computes an action, or a short sequence of actions, through
a policy module a[0] =A(s[0]).
This reactive process does not make use of the world model nor of the cost. The cost module computes
the energy of the initial statef[0] = C(s[0]) and stores the pairs (s[0],f [0]) in the short-term memory.
Optionally, it may also predict the next state using the world model s[1] = Pred(s[0],a [0]), and the
associated energy f[0] = C(s[0]) so that the world model can be adjusted once the next observation
resulting from the action taken becomes available.
The actor may comprise two components: (1) a policy module that directly produces
an action from the world state estimate produced by the perception and retrieved from the
short-term memory, and (2) the action optimizer, as described above, for model-predictive
control. The ﬁrst mode is similar to Daniel Kahneman’s “System 1”, while the second mode
is similar to “System 2” (Kahneman, 2011)
In the following, we will use speciﬁc symbols to represent various components in archi-
tectural diagrams. An brief explanation is given in Appendix 8.3.3.
3.1 Typical Perception-Action Loops
There are two possible modes that the model can employ for a perception-action episode.
The ﬁrst one involves no complex reasoning, and produces an action directly from the output
of the perception and a possible short-term memory access. We will call it “Mode-1”, by
analogy with Kahneman’s “System 1”. The second mode involves reasoning and planning
through the world model and the cost. It is akin to model-predictive control (MPC), a
classical planning and reasoning paradigm in optimal control and robotics. We will call
it “Mode-2” by analogy to Kahneman’s “System 2”. We use the term “reasoning” in a
broad sense here to mean constraint satisfaction (or energy minimization). Many types of
reasoning can be viewed as forms of energy minimization.
3.1.1 Mode-1: Reactive behavior
A perception-action episode for Mode-1 is depicted in Figure 3.
The perception module, through an encoder module, extracts a representation of the
state of the world s[0] = Enc(x) containing relevant information for the task at hand. A
9

policy module, a component of the actor, produces an action as a function of the state
a[0] =A(s[0]). The resulting action is sent to the eﬀectors.
The function of the policy module is modulated by the conﬁgurator, which conﬁgures
it for the task at hand.
The policy module implements a purely reactive policy that does not involve delib-
erate planning nor prediction through the world model. Yet, its structure can be quite
sophisticated. For example, in addition to the state s[0], the policy module may access the
short-term memory to acquire a more complete information about previous world states. It
may use the short-term memory for the associative retrieval of an action given the current
state.
While the cost module is diﬀerentiable, its output f[0] =C(s[0]) is indirectly inﬂuenced
by previous actions through the external world. Since the world is not diﬀerentiable, one
cannot back-propagate gradients from the cost through the chain cost ←perception←
world←action. In this mode, gradients of the cost f[0] with respect to actions can only
be estimated by polling the world with multiple perturbed actions, but that is slow and
potentially dangerous. This process would correspond to classical policy gradient methods
in reinforcement learning.
During Mode-1, the system can optionally adjust the world model. It runs the world
model for one step, predicting the next states[1], then it waits for the next percept resulting
from the action taken, and uses the observed world state as a target for the predictor.
With the use of a world model, the agent can imagine courses of actions and predict
their eﬀect and outcome, lessening the need to perform an expensive and dangerous search
for good actions and policies by trying multiple actions in the external world and measuring
the result.
3.1.2 Mode-2: reasoning and planning using the world model
A typical perception-action episode for Mode 2 is depicted in Figure 4.
1. perception: the perception system extract a representation of the current state of
the world s[0] = P (x). The cost module computes and stores the immediate cost
associated with that state.
2. action proposal : the actor proposes an initial sequence of actions to be fed to the
world model for evaluation (a[0],...,a [t],...,a [T ]).
3. simulation: the world model predicts one or several likely sequence of world state
representations resulting from the proposed action sequence ( s[1],...,s [t],...,s [T ]).
4. evaluation: the cost module estimates a total cost from the predicted state sequence,
generally as a sum over time steps F (x) =∑T
t=1C(s[t])
5. planning: the actor proposes a new action sequence with lower cost. This can be done
through a gradient-based procedure in which gradients of the cost are back-propagated
through the compute graph to the action variables. The resulting minimum-cost
action sequence is denoted (ˇa[0],..., ˇa[T ]). Full optimization may require iterating
steps 2-5.
10

Pred(s,a)
C(s[t])
 Pred(s,a)
C(s[t+1])
 Pred(s,a)
C(s[T-1])
 Pred(s,a)
C(s[T])
  
s[t] s[t+1]
action
s[0]
a[0]
Actor
C(s[0])
s[T-1]
a[t] a[t+1] a[T-1]
Figure 4: Mode-2 perception-action episode. The perception module estimates the state of the world
s[0]. The actor proposes a sequence of actions a[0],a [1],...,a [t],a [t + 1],...,a [T ]. The world model
recursively predicts an estimate of the world state sequence using s[t + 1] = Pred(s[t],a [t]). The cost
C(s[t]) computes an energy for each predicted state in the sequence, the total energy being the sum
of them. Through an optimization or search procedure, the actor infers a sequence of actions that
minimizes the total energy. It then sends the ﬁrst action in the sequence (or the ﬁrst few actions)
to the eﬀectors. This is, in eﬀect, an instance of classical model-predictive control with receding-
horizon planning. Since the cost and the model are diﬀerentiable, gradient-based methods can be
used to search for optimal action sequences as in classical optimal control. Since the total energy
is additive over time, dynamic programming can also be used, particularly when the action space is
small and discretized. Pairs of states (computed by the encoder or predicted by the predictor) and
corresponding energies from the intrinsic cost and the trainable critic are stored in the short-term
memory for subsequent training of the critic.
6. acting: after converging on a low-cost action sequence, the actor sends the ﬁrst action
(or ﬁrst few actions) in the low-cost sequence to the eﬀectors. The entire process is
repeated for the next perception-action episode.
7. memory: after every action, the states and associated costs from the intrinsic cost
and the critic are stored in the short-term memory. These pairs can be used later to
train or adapt the critic.
This procedure is essentially what is known as Model-Predictive Control (MPC) with
receding horizon in the optimal control literature. The diﬀerence with classical optimal
control is that the world model and the cost function are learned.
In principle, any form of optimization strategy can be used, for step 5. While gradient-
based optimization methods can be eﬃcient when the world model and cost are well-
behaved, situations in which the action-cost mapping has discontinuities may require to
use other optimization strategies, particularly if the state and/or action spaces can be
discretized. These strategies include dynamic programming, combinatorial optimization,
simulate annealing and other gradient-free methods, heuristic search techniques (e.g. tree
search with pruning), etc.
To simplify, the process was described in the deterministic case, i.e. when there is no
need to handle the possibility of multiple predictions for s[t + 1] resulting from a given
initial state s[t] and action a[t]. In real situations, the world is likely to be somewhat
unpredictable. Multiple states may result from a single initial state and action due to
the fact that the world is intrinsically stochastic (aleatoric uncertainty), or that the state
representation s[t] contains incomplete information about the true world state (epistemic
uncertainty), or that the world model’s prediction accuracy is imperfect due to limited
training data, representational power, or computational constraints.
11