A Path Towards Autonomous Machine Intelligence (LeCun, 2022) — part 10/12

Cleaned-text code points [125279, 140812). Bibliography omitted here; source.md includes it.

elation may be the result of anticipation of outcome by brain structures whose function is
similar to the Trainable Critic.
The presence of a cost module that drives the behavior of the agent by searching for
optimal actions suggests that autonomous intelligent agents of the type proposed here will
inevitably possess the equivalent of emotions. In an analogous way to animal and humans,
machine emotions will be the product of an intrinsic cost, or the anticipation of outcomes
from a trainable critic.
8.2.2 Could this be a Path towards Machine Common Sense?
It is a widely-held opinion that none of the current AI systems possess any level of common
sense, even at the level that can be observed in a house cat. Animals seem to be able to
acquire enough background knowledge about how the world works to exhibit some level
of common sense. By contrast, AI systems, even when (pre-)trained with self-supervised
mode (e.g. from text) seem to exhibit very limited levels of common sense, making them
somewhat brittle.
For example, Large language models (LLMs) seem to possess a surprisingly large amount
of background knowledge extracted from written text. But much of human common-sense
knowledge is not represented in any text and results from our interaction with the physical
world. Because LLMs have no direct experience with an underlying reality, the type of
common-sense knowledge they exhibit is very shallow and can be disconnected from reality.
A possible characterization of common sense is the ability to use models of the world
to ﬁll in blanks, for example predicting the future, or more generally ﬁlling in information
about the world that is unavailable from perception or from memory. With this deﬁnition,
common sense is an ability that emerges from a collection of models of the world or from
a single model engine conﬁgurable to handle the situation at hand. This view of common
sense sits squarely in the camp of “grounded intelligence”: common sense is a collection of
models from low-levels of abstraction to high levels, all the way up to knowledge acquired
through language.
Could SSL applied to conﬁgurable H-JEPA constitute the substrate of machine common
sense? Could a properly-trained and conﬁgured H-JEPA embed enough predictive knowl-
edge and capture enough dependencies about the world to exhibit some level of common
sense?
I speculate that common sense may emerge from learning world models that capture the
self-consistency and mutual dependencies of observations in the world, allowing an agent to
ﬁll in missing information and detect violations of its world model.
8.3 Is it all about scaling? Is reward really enough?
The section reviews a few potential paths towards human-level intelligence that have been
proposed in recent years. The surprising power of large transformer architectures trained
to predict text and other modalities have led some to claim that we merely need to scale up
those models (Brown et al., 2020; Brown et al., 2020). The surprising power of reinforcement
learning for games and other simple environments have led other to claim that reward is
enough (Silver et al., 2021). ﬁnally, the limitations of current deep-learning systems when it
45

comes to reasoning have led some to claim that deep learning systems need to be augmented
by hard-wired circuitry to enable symbol manipulation (Marcus and Davis, 2019)
8.3.1 Scaling is not enough
Large Language Models (LLMs), and more generally, large-scale transformer architectures
trained with a form of generative self-supervised learning, have been astonishingly successful
at capturing knowledge present in text. This has led to a debate in the AI community as
to whether human-level AI can be attained by scaling up these architectures. My position
in this debate is that I do not believe that scaling is enough for two main reasons.
First, current models operate on “tokenized” data and are generative. Every input
modality must be turned into a sequence (or a collection) of “tokens” encoded as vectors.
While this works well for text, which is already a sequence of discrete tokens, it is less
suitable for continuous, high dimensional signals such as video. The type of SSL training
used for LLM-style models can be seen as a particular kind of latent-free generative model
trained with a particular kind of contrastive method called denoising auto-encoder (Vincent
et al., 2010), or in this case masked auto-encoder (Devlin et al., 2018). Hence, they are
subject to the limitations of generative models, latent-variable free models, and contrastive
methods. Generative models have diﬃculty representing complex uncertainties in contin-
uous spaces. LLMs simplify the representation of uncertainty in the prediction by only
dealing with discrete objects from a ﬁnite collection (e.g. words from a dictionary). Repre-
senting uncertainty about a word being predicted comes down to producing a vector whose
components are scores or probabilities for each word (or discrete token) in the dictionary.
But this approach doesn’t work for high-dimensional continuous modalities, such as video.
To represent such data, it is necessary to eliminate irrelevant information about the variable
to be modeled through an encoder, as in the JEPA. Furthermore, the high-dimensionality
of the signal precludes the representation of uncertainty through a normalized distribution.
Second, current models are only capable of very limited forms of reasoning. The absence
of abstract latent variables in these models precludes the exploration of multiple interpre-
tations of a percept and the search for optimal courses of action to achieve a goal. In fact,
dynamically specifying a goal in such models is essentially impossible.
8.3.2 Reward is not enough
The proposed architecture is designed to minimize the number of actions a system needs
to take in the real world to learn a task. It does so by learning a world model that capture
as much knowledge about the world as possible without taking actions in the world. It uses
intrinsic costs that are diﬀerentiable functions of measured or predicted world states. This
makes the proposal more similar to optimal control than to reinforcement learning. In the
proposed model, much of learning takes place at the level of the world model (perceptual
encoder and predictor). In this context, what is the role of reinforcement learning (RL)?
In most RL settings the reward (or the cost, which is a negative reward) is fed to the
agent by the environment. In other words, Intrinsic Cost module is the environment itself,
and is therefore an unknown function. The value of the function can be probed by observing
the state of the world, taking an action, and observing the resulting reward. The gradient
of the reward with respect to the action or the state is unknown and must be estimated by
46

multiple action trials as in policy gradient methods. In Actor-Critic methods, the reward
function is approximated by a critic module that is trained to approximate expected future
values of the reward. The critic provides a diﬀerentiable approximation of the reward
function.
But model-free RL is extremely sample-ineﬃcient, at least when compared with human
and animal learning, requiring very large numbers of trials to learn a skill. Scalar rewards
provide low-information feedback to a learning system. As a consequence, a pure RL system
requires a very large number of trials to learn even relatively simple tasks. Model-based
RL clearly has the potential of being considerably more sample eﬃcient. But the question
becomes how to train the world model: is it trained from taking actions and getting rewards,
or is it trained by predicting the world state? In the latter case,reward is clearly not enough:
most of the parameters in the systems are trained to predict large amounts of observations
in the world. Contrary to the title of a recent position paper by Silver et al. (Silver et al.,
2021), the reward plays a relatively minor role in this scenario.
8.3.3 Do We Need Symbols for Reasoning?
In the proposed architecture, reasoning comes down to energy minimization or constraint
satisfaction by the actor using various search methods to ﬁnd a suitable combination of
actions and latent variables, as stated in Section 3.1.4.
If the actions and latent variables are continuous, and if the predictor and the cost
modules are diﬀerentiable and relatively well behaved, one can use gradient-based methods
to perform the search. But there may be situations where the predictor output changes
quickly as a function of the action, and where the action space is essentially discontinuous.
This is likely to occur at high levels of abstractions where choices are more likely to be
qualitative. A high-level decision for a self-driving car may correspond to “turning left or
right at the fork”, while the low-level version would be a sequence of wheel angles.
If the action space is discrete with low cardinality, the actor may use exhaustive search
methods. If the action set cardinality, and hence the branching factor, are too large, the
actor may have to resort to heuristic search methods, including Monte-Carlo Tree Search,
or other gradient-free methods. If the cost function satisﬁed Bellman’s equations, one may
use dynamic programming.
But the eﬃciency advantage of gradient-based search methods over gradient-free search
methods motivates us to ﬁnd ways for the world-model training procedure to ﬁnd hierar-
chical representations with which the planning/reasoning problem constitutes a continuous
relaxation of an otherwise discrete problem.
A remain question is whether the type of reasoning proposed here can encompass all
forms of reasoning that humans and animals are capable of.
Acknowledgments
Ideas in this paper are a distillation of years of interactions with many people. It is impos-
sible to list them all here.
I will list some of the more salient ones: L´ eon Bottou, Yoshua Bengio, Geoﬀrey Hinton,
Rob Fergus, and Jitendra Malick; Emmanuel Dupoux and Stanislas Dehaene; Mikael Henaﬀ,
47

Alfredo Canziani, and Nicolas Carion at NYU; Li Jing, Yubei Chen, Randall Balestriero,
and St´ ephane Deny at FAIR; Vlad Sobal, Jiachen Zhu, and Katrina Evtimova at NYU.
I am thankful to the following colleagues for their comments on the manuscript: Olivier
Delalleau, Gus Xia, Yoshua Bengio, and Emmanuel Dupoux.
48

References
Agrawal, P., Nair, A., Abbeel, P., Malik, J., and Levine, S. (2016). Learning to poke by
poking: Experiential learning of intuitive physics. CoRR, abs/1606.07419.
Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S. (2017). Stochastic
variational video prediction. CoRR, abs/1710.11252.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for
self-supervised learning of speech representations. In Larochelle, H., Ranzato, M., Had-
sell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing
Systems, volume 33, pages 12449–12460. Curran Associates, Inc.
Bardes, A., Ponce, J., and LeCun, Y. (2021). Vicreg: Variance-invariance-covariance regu-
larization for self-supervised learning. In International Conference on Learning Repre-
sentations (ICLR 2022) . arXiv preprint arXiv:2105.04906.
Battaglia, P., Pascanu, R., Lai, M., Jimenez Rezende, D., et al. (2016). Interaction net-
works for learning about objects, relations and physics. Advances in neural information
processing systems, 29.
Becker, S. and Hinton, G. E. (1992). Self-organizing neural network that discovers surfaces
in random-dot stereograms. Nature, 355(6356):161–163.
Bertsekas, D. (2019). Reinforcement learning and optimal control. Athena Scientiﬁc.
Bordes, A., Usunier, N., Chopra, S., and Weston, J. (2015). Large-scale simple question
answering with memory networks. arXiv:1506.02075.
Bromley, J., Guyon, I., LeCun, Y., Sackinger, E., and Shah, R. (1994). Signature veriﬁcation
using a “siamese” time delay neural network. In NeurIPS.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan,
A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G.,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen,
M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish,
S., Radford, A., Sutskever, I., and Amodei, D. (2020). Language models are few-
shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H.,
editors, Advances in Neural Information Processing Systems , volume 33, pages 1877–
1901. Curran Associates, Inc.
Bryson, A. and Ho, Y. (1969). Applied optimal control. Blaisdell, Waltham, MA.
Carey, S. (2009). The Origin of Concepts . Oxford University Press, New York, New York,
USA.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020).
End-to-end object detection with transformers. In 16th European Conference, Glasgow,
UK (ECCV 2020) , page 213–229.
49

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. (2020). Unsu-
pervised learning of visual features by contrasting cluster assignments. In Advances in
Neural Information Processing Systems.
Carreira-Perpi˜ n´ an, M. A. and Hinton, G. (2005). On contrastive divergence learning. In
Cowell, R. G. and Ghahramani, Z., editors, Proceedings of the Tenth International
Workshop on Artiﬁcial Intelligence and Statistics, volume R5 ofProceedings of Machine
Learning Research, pages 33–40. PMLR. Reissued by PMLR on 30 March 2021.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. (2020a). Big self-
supervised models are strong semi-supervised learners. In NeurIPS.
Chen, X., Fan, H., Girshick, R., and He, K. (2020b). Improved baselines with momentum
contrastive learning. arXiv preprint arXiv:2003.04297 .
Chopra, S., Hadsell, R., and LeCun, Y. (2005). Learning a similarity metric discriminatively,
with application to face veriﬁcation. In 2005 IEEE Computer Society Conference on
Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 539–546. IEEE.
Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning
in a handful of trials using probabilistic dynamics models. CoRR, abs/1805.12114.
Craik, K. J. W. (1943). The nature of explanation . University Press, Macmillan.
Dai, X., Tong, S., Li, M., Wu, Z., Psenka, M., Chan, K. H. R., Zhai, P., Yu, Y., Yuan,
X., Shum, H.-Y., and Ma, Y. (2022). Ctrl: Closed-loop transcription to an ldr via
minimaxing rate reduction. Entropy, 24(4):456.
Dehaene, S., Lau, H., and Kouider, S. (2021). What is consciousness, and could machines
have it? Robotics, AI, and Humanity , pages 43–56.
Denton, E. and Fergus, R. (2018). Stochastic video generation with a learned prior. arXiv
preprint arXiv 1802.07687 .
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training
of deep bidirectional transformers for language understanding. arXiv preprint
arXiv:1810.04805.
Doi, E., Balcan, D. C., and Lewicki, M. S. (2007). Robust coding over noisy overcomplete
channels. IEEE Transactions on Image Processing , 16(2):442–452.
Ermolov, A., Siarohin, A., Sangineto, E., and Sebe, N. (2021). Whitening for self-supervised
representation learning.
Evtimova, K. and LeCun, Y. (2022). Sparse coding with multi-layer decoders using variance
regularization. arXiv:2112.09214.
Finn, C., Goodfellow, I. J., and Levine, S. (2016). Unsupervised learning for physical
interaction through video prediction. CoRR, abs/1605.07157.
50