Dissociating language and thought in large language models — part 3/5

Cleaned-text code points [31099, 46720). Bibliography omitted here; source.md includes it.

Importantly, a human-like language model is not expected to rely solely on abstract rules. Humans use diverse cues in their language learning and processing that sometimes override or conflict with strict hierarchical syntactic processing [ 97 , 98 , e.g.,] . Humans also rely, to varying extents, on memorizing previously seen input, as opposed to purely applying abstract rules [ 89 , 21 ] . Thus, when evaluating formal competence in LLMs, it is essential to directly compare their performance with that of humans [ 99 ] . For instance, a re-examination [ 100 ] of an earlier study [ 101 ] showed that apparent syntactic agreement deficits in GPT-2 occurred on instances that were also challenging for humans. Overall, LLMs clearly learn some linguistic abstraction, even if the degree of that abstraction remains a matter of debate (as it does for humans).

#### LLMs learn constructions

Recent evidence suggests that LLMs learn syntactic constructions [ 102 , 103 , 104 ] . These constructions can be idiosyncratic, lexically sensitive, and relatively rare, such as “a beautiful five days in Austin” [ 105 ] . LLMs also show some amount of sensitivity to the Preposing in Prepositional Phrase construction (“Surprising though it may be…”), even when the gap crosses a finite clause boundary (“Surprising though I know it may be”) [ 106 ] . They achieve this sensitivity even though such examples crossing the finite clause boundary are vanishingly rare: only 58 examples out of 7 billion sentences in a corpus. The fact that models can learn that some vanishingly rare constructions are grammatical, whereas other equally rare constructions are not, suggests that LLMs meaningfully learn something about syntax.

Models are also sensitive to the form of the comparative correlative “the better the syntax, the better the semantics” [ 107 ] . However, this sensitivity does not mean that they are sensitive to the semantic implications of the construction. Indeed, it appears that inferences based on these sentences can be a challenge (e.g., knowing that if I say “the better the syntax, the better the semantics” and then tell you that the syntax is better, this means the semantics is better). This asymmetry nicely illustrates the formal/functional distinction: the model clearly knows how to use the construction and get the form right without necessarily being able to access the intended meaning. We discuss these issues in more detail in later sections.

### LLMs are predictive of activity in the human language network

As discussed above, language processing in humans relies on a dedicated brain network. This network exhibits all the hallmarks of formal linguistic competence: it is sensitive to abstract hierarchical rules in isolated phrases and sentences [ 31 , 108 , 109 , 110 ] , in naturalistic narratives [ 111 , 112 , 113 , 114 ] , and in syntactically well-formed but semantically empty ("jabberwocky") stimuli [ 31 , 49 , 109 ] . The language network is also sensitive to specific word co-occurrences [ 111 , e.g., as evidenced by sensitivity to n-gram surprisal; ] , indicating that it learns not only the rules, but also the patterns of language. The language network’s selectivity for linguistic vs. non-linguistic inputs, along with its sensitivity to linguistic rules and patterns, allows us to operationalize formal linguistic competence as a set of computations that in humans take place within the language network.

If LLMs and the human language network perform similar computations to achieve formal linguistic competence, we expect to observe similarities in their internal organization (see [ 55 ] for similar arguments in the domain of vision). And indeed, LLMs and the human language network exhibit non-trivial similarities.

First, the internal architecture of LLMs resembles that of the language network. Both operate at the level of abstract linguistic units (words/tokens) rather than modality-specific representations, such as pixels or acoustic waveforms, and combine these unit-level representations into composite representations of phrases and sentences. Neither system shows clear spatial segregation for syntactic and semantic processing (LLMs: [ 95 , 115 ] ; brain: [ 38 , 114 ] ), indicating that these processes are tightly functionally coupled in both.

Second, one can establish a direct mapping between internal LLM representations and neural activity patterns within the language network. This mapping can be successfully used to predict brain responses to novel sentences and words in previously unseen contexts [ 116 , 117 , 118 ] . This similarity between sentence activation patterns in LLMs and the brain is suggestive of similar representational mechanisms that support computations in these systems.

We do not claim that the correspondence between LLMs and the language network is one-to-one. For instance, LLMs learn patterns outside traditional human linguistic competency, such as predicting newline characters [ 119 ] . Nevertheless, the fact that internal representations learned by contemporary LLMs contain sufficient information to predict the language network’s responses to diverse linguistic strings indicates at least some correspondence between LLMs’ representations and those in the language network.

### Using LLMs as models of formal linguistic competence in humans

LLMs today generate highly coherent, grammatical texts that can be indistinguishable from human output. In doing so, they exhibit knowledge of hierarchical structure and linguistic abstractions, while resembling human brain responses during language processing. These models are not perfect learners of abstract linguistic rules, but neither are humans. We therefore conclude that LLMs possess substantial formal linguistic competence, at least in English.

LLMs have already overturned claims about the fundamental impossibility of acquiring certain linguistic knowledge—including hierarchical structure and abstract categories—from the statistics of linguistic input alone [ 120 ] . If language modeling continues to improve (including learning from more realistic kinds and amounts of data; Box 3.4 ), this would allow testing more general versions of this “poverty of the stimulus” argument [ 121 ] , including specific tests of what inductive biases might be necessary to successfully learn the rules and statistical regularities of human language. As such, LLMs have substantial value in the scientific study of language learning and processing.

### Excessive reliance on statistical regularities

### Unrealistic amounts of training data

### Insufficient tests on languages other than English

## Non-augmented LLMs fall short on functional linguistic competence

Real-life language use is impossible without non-linguistic cognitive skills. Understanding a sentence, reasoning about its implications, and deciding how to respond all rely on cognitive capacities that go beyond formal competence. In this section, we ask: how good are contemporary LLMs at functional linguistic competence?

We focus on four key capacities that are not language-specific but are nevertheless crucial for language use in real-life settings: i) formal reasoning—a host of abilities including logical and mathematical reasoning, computational thinking, and novel problem solving; ii) world knowledge—factual and commonsense knowledge about agents, objects, properties, actions, events, and ideas; iii) situation modeling—the dynamic tracking of objects, agents, and events as a narrative/conversation unfolds over time; and iv) social reasoning—understanding the social context of linguistic exchanges. An average conversation requires the use of all these capacities, yet none of them are specific to language use.

For each domain, we first describe its neural mechanisms in humans and then discuss how well contemporary LLMs have mastered the domain. We conclude that, unlike formal competence, functional competence of LLMs is uneven, often requiring specialized fine-tuning and/or lacking human-like robustness and generality. In Box 4 , we highlight the importance of properly evaluating LLMs; evaluation issues can occur in studies of either formal or functional competence, but we believe they have led to a particularly large amount of overclaiming about models’ functional competence.

### A. Fine-tuning on the task and the challenge of closed models

### B. Generalizable, robust performance

### Formal Reasoning

Language allows people to discuss highly abstract ideas, turn ideas into scientific and philosophical theories, construct logical syllogisms, and engage in formal debates. Unsurprisingly, language is often considered a cornerstone of complex reasoning [ 136 , 137 ] . However, neuroscience provides evidence that language and formal reasoning dissociate in cognitive systems, and so a model that has mastered formal linguistic competence will not necessarily exhibit logical reasoning abilities.

Humans. Despite their close interplay, language and reasoning rely on distinct cognitive and neural systems. Unlike language, formal reasoning engages brain regions known as the multiple demand network [ 138 ] , named so because these regions are engaged in many cognitively demanding tasks: logic [ 47 ] , mathematical reasoning [ 41 ] , physical reasoning [ 139 ] , and computer code comprehension [ 140 , 46 ] . Human patient studies have provided causal evidence for the role of the multiple demand network in logical reasoning by showing that the amount of damage to these regions correlates negatively with performance on standard tests of fluid intelligence [ 141 , 142 ] . Importantly, the multiple demand network supports reasoning even when the task is presented linguistically [ 41 , 140 , 47 ] — similar to how LLMs receive their prompts.

LLMs. Multiple studies have pointed out LLMs’ limitations on tasks requiring formal reasoning, such as math problems. GPT-3 performs well on two-digit addition and subtraction but not on more complex tasks, such as three-digit addition or two-digit multiplication [ 69 ] . GPT-4 similarly shows good performance on small-digit but not large-digit mathematical operations [ 143 ] . Reasoning tests that break common co-occurrence patterns in the input or require multi-step operations also lead to model failure [ 144 , 145 ] .

The most commonly cited reason for these failures is the failure of artificial neural nets to generalize to patterns outside their training distribution [ 145 , 146 ] . This generalization gap can be partially bridged by “chain of thought” approaches, whereby a model is prompted to generate intermediate computation steps before arriving at an answer [ 147 ] . However, even these approaches do not lead to foolproof results [ 143 ] . Thus, more and more researchers pair LLMs with external modules that can carry out structural logical and mathematical computations, such as the Mathematica plugin [ 148 ] or a probabilistic reasoning engine [ 149 ] . The shift toward augmenting LLMs with reasoning-specific modules is consistent with evidence from neuroscience: language and formal reasoning are distinct cognitive capacities that work best when they are supported by separate processing mechanisms.

### World models 1: factual and commonsense knowledge

A commonly debated capacity in LLMs is their ability to leverage internal world models [ 150 , 149 ] . We break down the notion of world models into two components: world knowledge (factual and commonsense, this section) and situation tracking (the ability to maintain and update information about objects, agents, etc.; next section).

Humans. Evidence from neuroscience shows a dissociation between linguistic and semantic (world) knowledge. Individuals with language deficits may struggle to produce grammatical utterances and retrieve contextually appropriate words, but their ability to reason about objects and events presented non-linguistically often remains intact [ 151 , 42 ] . On the other hand, individuals who suffer from semantic dementia (a neurodegenerative disorder) retain the ability to speak but struggle with tasks that rely on world knowledge (e.g., knowing that pumpkins are typically orange) even when the stimuli are presented non-verbally as pictures [ 152 ] . Thus, linguistic and semantic knowledge can be disentangled.

LLMs. LLMs have access to a wealth of knowledge about the world: word co-occurrence patterns in texts on the web contain both factual information (e.g., who was the first man on the moon) and commonsense information (e.g., the taste of lemon) [ 153 ] . If this information can be effectively extracted, LLMs would be able to serve as off-the-shelf knowledge bases [ 154 ] . However, world knowledge contained in LLM representations suffers from several major shortcomings.

First, LLMs routinely generate false statements, informally known as “hallucinations”. This observation is unsurprising: their training objective is to generate plausible sentence continuations, with no reference to the underlying factual correctness of the resulting claims. Some developers have fine-tuned LLMs to provide links to sources that back up their claims; however, those citations can also be inaccurate [ 155 ] .

Second, LLM outputs are often inconsistent: the same prompt phrased in different ways can elicit different responses [ 156 ] . They can also get “distracted” by intervening information, e.g., an irrelevant claim inserted between a premise and a conclusion [ 92 ] .

Third, commonsense knowledge is often underrepresented in language corpora: people are much more likely to communicate new or unusual information rather than commonly known facts [ 157 ] . As a result, LLMs can struggle on commonsense knowledge benchmarks [ 158 ] , especially once low-level statistical cues are controlled for [ 9 ] .

And fourth, explicitly stated factual knowledge is easy to access but hard to maintain, requiring constant updates; for instance, the answer to “Who is the current president of the US?” will change every 4 or 8 years. Whereas humans can update their knowledge representations via a single sentence, updating world knowledge in LLMs requires locating and editing this particular bit of knowledge in their internal parameters-a non-trivial task [ 159 ] , especially because these edits should affect some other bits of knowledge (e.g., that the previously current president is now the past president) but leave many other facts unaffected [ 160 ] .

A more human-like approach to world knowledge representation might require dissociating linguistic representation/processing and world knowledge storage/updates. Such approaches exist [ 161 , e.g.,] but have not yet reached dominance in the field, typically because of relatively low coverage of existing knowledge bases. Although we cannot rely on LLMs alone for accurate world knowledge claims, we might use them as a starting point for constructing detailed knowledge bases [ 162 ] and commonsense schemata [ 163 ] .

### World models 2: situation tracking

People can follow the plot of a story that spans multiple chapters or even multiple books. We can also remember many details weeks or months after a conversation. We accomplish these feats by leveraging language inputs to create a "situation model" — a mental model of entities, relations between them, and a sequence of states they had been in or events they had participated in [ 164 ] . Does the language network in humans construct a situation model based on its inputs? And how good are LLMs at building and updating situation models over time?