Could a Large Language Model be Conscious? — part 2/4

Cleaned-text code points [14585, 29724). Bibliography omitted here; source.md includes it.

1960s, users treated Joseph Weizenbaum’s simple dialog system ELIZA as if it were conscious.  
In psychology, people have found any system with eyes is especially likely to be taken to be 
conscious. So I don’t think this reaction is strong evidence. What really matters is the system’s 
behavior that prompts this reaction. This leads to a third candidate for X. 
X = Conversational Ability 
Language models display remarkable conversational abilities. Many current systems are 
optimized for dialogue, and often give the appearance of coherent thinking and reasoning. 
They’re especially good at giving reasons and explanations, a capacity often regarded as a 
hallmark of intelligence.  
In his famous test, Alan Turing highlighted conversational ability as a hallmark of thinking.7 Of 
course even LLMs that are optimized for conversation don’t currently pass the Turing test. There 
are too many glitches and giveaways for that for that. But they’re not so far away. Their 
performance often seems on a par at least with that of a sophisticated child. And these systems 
are developing fast. 
That said, conversation is not the fundamental thing here. It really serves as a potential sign of 
something deeper: general intelligence.  
X = General Intelligence 
Before LLMs, almost all AI systems were specialist systems. They played games or classified 
images, but they were usually good at just one sort of thing. By contrast, current LLMs can do 
many things.  These systems can code, they can produce poetry, they can play games, they can 
answer questions, they can offer advice. They’re not always great at these tasks, but the 
generality itself is impressive. Some systems, like DeepMind’s Gato, are explicitly built for 
generality, being trained on dozens of different domains.8 But even basic language models like 
GPT-3 already show significant signs of generality without this special training.9  
Among people who think about consciousness, domain-general use of information is often 
regarded as one of the central signs of consciousness. So the fact that we are seeing increasing 
generality in these language models may suggest a move in the direction of consciousness. Of 
course this generality is not yet at the level of human intelligence. But as many people have 
observed, two decades ago, if we’d seen a system behaving as LLMs do without knowing how it 
 
7 Alan Turing, Computing Machinery and Intelligence. Mind 49: 433-460, 1950. 
8 Scott Reed et al, A generalist agent. Transactions on Machine Learning Research, 2022. arXiv:2205.06175. 
9 Tom Brown et al, Language models are few shot learners. arXiv:2005.14165, 2020.

worked, we’d have taken this behavior as fairly strong evidence for intelligence and 
consciousness. 
Now, maybe that evidence can be defeated by something else. Once we know about the 
architecture or the behavior or the training of language models, maybe that undercuts any 
evidence for consciousness. Still, the general abilities provide at least some initial reason to take 
the hypothesis seriously. 
Overall, I don’t think there’s strong evidence that current large language models are conscious. 
Still, their impressive general abilities give at least some limited reason to take the hypothesis 
seriously. That’s enough to lead us to considering the strongest reasons against consciousness in 
LLMs. 
3. Evidence against consciousness in large language models? 
 
What are the best reasons for thinking language models aren’t or can’t be conscious? I see this as 
the core of my discussion. One person’s barrage of objections is another person’s research 
program. Overcoming the challenges could help show a path to consciousness in LLMs or 
LLM+s. 
I’ll put my request for evidence against LLM consciousness in the same regimented form as 
before. If you think large language models aren’t conscious, articulate a feature X such that (i) 
these models lack X, (ii) if a system lacks X, it probably isn’t conscious, and give good reasons 
for (i) and (ii). 
There’s no shortage of candidates for X. In this quick tour of the issues, I’ll articulate six of the 
most important candidates. Those six include: 
X = Biology 
The first objection, which I’ll mention very quickly, is the idea that consciousness requires 
carbon-based biology.10 Language models lack carbon-based biology, so they are not conscious. 
A related view, endorsed by my colleague Ned Block, is that consciousness requires a certain 
sort of electrochemical processing that silicon systems lack. Views like these would rule out all 
silicon-based AI consciousness if correct. 
In earlier work, I’ve argued that these views involve a sort of biological chauvinism and should 
be rejected.  In my view, silicon is just as apt as carbon as a substrate for consciousness. What 
matters is how neurons or silicon chips are hooked up to each other, not what they are made of.  
 
10 Ned Block. Comparing the major theories of consciousness. In (M. Gazzaniga, ed.) The Cognitive 
Neurosciences IV. MIT Press, 2009.

Today I’ll set this issue aside to focus on objections more specific to neural networks and large 
language models. I’ll revisit the question of biology at the end.  
X = Senses and Embodiment 
Many people have observed that large language models have no sensory processing, so they 
can’t sense. Likewise they have no bodies, so they can’t perform bodily actions. That suggests, at 
the very least, that they have no sensory consciousness and no bodily consciousness. 
Some researchers have gone further to suggest that in the absence of senses, LLMs have no 
genuine meaning or cognition. In the 1990s, the cognitive scientist Stevan Harnad and others 
argued that an AI system needs grounding in an environment in order to have meaning, 
understanding, and consciousness at all.11 In recent years, a number of researchers have argued 
that sensory grounding is required for robust understanding in LLMs.12  
I’m somewhat skeptical that senses and embodiment are required for consciousness and for 
understanding. In other work (“Does Thinking Require Sensory Grounding?”). I’ve argued that 
in principle, a disembodied thinker with no senses could still have conscious thought, even if its 
consciousness was limited. 13  For example, an AI system without senses could reason about 
mathematics, about its own existence, and maybe even about the world. The system might lack 
sensory consciousness and bodily consciousness, but it could still have a form of cognitive 
consciousness. 
On top of this, LLMs have a huge amount of training on text input which derives from sources in 
the world. One could argue that this connection to the world serves as a sort of grounding. The 
computational linguist Ellie Pavlick and colleagues have research suggesting that text training 
sometimes produces representations of color and space that are isomorphic to those produced by 
sensory training.14 Perhaps the reason is that training materials include descriptions of color and 
space that were originally grounded in sensory processes. 
 
11 Stevan Harnad, The symbol-grounding problem. Physica D 42:335-346, 1990. Note that grounding has quite 
different meanings among AI researchers (roughly, processing caused by sensory inputs and the environment) and 
among philosophers (roughly, constitution of the less fundamental by the more fundamental). 
12 Emily M. Bender and Alexander Koller, Climbing towards NLU: On meaning, form, and understanding in the age 
of data. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5185–5198, 
2020. Brenden Lake and Greg Murphy, Word meaning in minds and machines. Psychological Review, 2021. Jacob 
Browning and Yann Lecun, AI and the limits of language. Noēma, 2022. 
13 David J. Chalmers, Does Thought Require Sensory Grounding: From Pure Thinkers to Large Language Models”.  
Proceedings and Addresses of the American Philosophical Association, 2023.   
14 Roma Patel and Ellie Pavlick, Mapping language models to grounded conceptual spaces. International 
Conference on Learning Representations, 2022. Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, 
Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? A case 
study in color. Proceedings of the 25th Conference on Computational Natural Language Learning, 2021.

A more straightforward reply is to observe that multimodal extended language models have 
elements of both sensory and bodily grounding. Vision-language models are trained on both text 
and on images of the environment.   Language-action models are trained to control bodies 
interacting with the environment. Vision-language-action models combine the two. Some 
systems control physical robots using camera images of the physical environment, while others 
control virtual robots in a virtual world. 
Virtual worlds are a lot more tractable than the physical world and there’s coming to be a lot of 
work in embodied AI that uses virtual embodiment. Some people will say this doesn’t count for 
what’s needed for grounding because the environments are virtual. I don’t agree. In my book on 
the philosophy of virtual reality, Reality+, I’ve argued that virtual reality is just as legitimate and 
real as physical reality for all kinds of purposes.15 Likewise, I think that virtual bodies can help 
support cognition just as physical bodies do. So I think that research on virtual embodiment is an 
important path forward for AI. 
This constitutes a fourth challenge on the path to conscious AI: build rich perception-language-
action models in virtual worlds. 
X = World Models and Self Models 
The computational linguists Emily Bender and Angelina McMillan-Major and the computer 
scientists Timnit Gebru and Margaret Mitchell have argued that large language models are 
“stochastic parrots”.16 The idea is roughly that like many talking parrots, LLMs are merely 
imitating language without understanding it. In a similar vein, others have suggested that LLMs 
are just doing statistical text processing. One underlying idea here is that language models are 
just modeling text and not modeling the world. They don’t have genuine understanding and 
meaning of the kind you get from a genuine world-model.  Many theories of consciousness 
(especially so-called representational theories) hold that world models are required for 
consciousness.17 
 
15 David J. Chalmers, Reality+: Virtual Worlds and the Problems of Philosophy. W. W. Norton, 2022. 
 
16 Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, Schmargaret Schmitchell. On the dangers of 
stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, 
Accountability, and Transparency. pp. 610-623, 2021. 
17 See William Lycan, Representational theories of consciousness, Stanford Encyclopedia of Philosophy.  
Self-models: Higher-order theories of consciousness (see Rocco Gennaro, ed, Higher-Order Theories of 
Consciousness: An Anthology, John Benjamins, 2004) hold that self-models (representations of one’s own mental 
states) are required for consciousness. In addition, illusionist theories of consciousness (see Keith Frankish, ed. 
Illusionism as a Theory of Consciousness, Imprint Academic, 2016, and Michael Graziano, Rethinking 
Consciousness, W.W. Norton, 2019) hold that self-models are required for the illusion of consciousness.

There’s a lot to say about this, but just briefly: I think it’s important to make a distinction 
between training methods and post-training processes (sometimes called inference). It’s true that 
language models are trained to minimize prediction error in string matching, but that doesn’t 
mean that their post-training processing is just string matching. To minimize prediction error in 
string matching, all kinds of other processes may be required, quite possibly including world-
models. 
An analogy: in evolution by natural selection, maximizing fitness during evolution can lead to 
wholly novel processes post-evolution. A critic might say, all these systems are doing is 
maximizing fitness. But it turns out that the best way for organisms to maximize fitness is to 
have these remarkable capacities—like seeing and flying and even having world-models. 
Likewise, it may well turn out that the best way for a system to minimize prediction error during 
training is for it to use novel processes, including world-models. 
 
It’s plausible that neural network systems such as transformers are capable at least in principle of 
having deep and robust world-models. And it’s plausible that in the long run, systems with these 
models will outperform systems without these models at prediction tasks. If so, one would expect 
that truly minimizing prediction error in these systems would require deep models of the world. 
For example, to optimize prediction in discourse about the New York City subway system, it will 
help a lot to have a robust model of the subway system. Generalizing, this suggests that good 
enough optimization of prediction error over a broad enough space of models ought to lead to 
robust world-models. 
If this is right, the underlying question is not so much whether it’s possible in principle for a 
language models to have world-models and self-models, but instead whether these models are 
already present in current language models. That’s an empirical question. I think the evidence is 
still developing here, but interpretability research gives at least some evidence of robust world 
models. For example, Kenneth Li and colleagues trained a language model on sequences of 
moves in the board game Othello, and gave evidence that it builds an internal model of the 64 
board squares and uses this model in determining the next move.18 There’s also much work on 
finding where and how facts are represented in language models.19  
There are certainly many limitations in current LLMs’ world-models. Standard models often 
seem fragile rather than robust, with language models often confabulating and contradicting 
themselves. Current LLMs seem to have especially limited self-models: that is, their models of 
their own processing and reasoning are poor. Self-models are crucial at least to self-
consciousness, and on some views (including so-called higher-order views of consciousness) 
they are crucial to consciousness itself.20 
 18 Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg, 
Emergent world representations: Exploring a sequence model trained on a synthetic task. arXiv.2210.13382, 2022.  19 Belinda Z Li, Maxwell Nye, and Jacob Andreas, Implicit representations of meaning in neural language models. 
arXiv:2106.00737, 2021. 20 Higher-order theories of consciousness (see Rocco Gennaro, ed, Higher-Order Theories of Consciousness: An 
Anthology, John Benjamins, 2004) hold that self-models (representations of one’s own mental states) are required 
for consciousness. In addition, illusionist theories of consciousness (see Keith Frankish, ed. Illusionism as a Theory