Dissociating language and thought in large language models — part 4/5

Cleaned-text code points [46720, 61867). Bibliography omitted here; source.md includes it.

Humans. The language network in humans does not appear to track structure above the clause level [ 165 , 166 ] . Instead, integration of meaning over longer periods of time likely takes place within the so-called default network [ 167 ] . Crucially, the default network tracks both linguistic and non-linguistic narratives [ 168 ] , indicating that situation modeling is not a language-specific skill.

LLMs. Situation modeling in LLMs faces two main challenges: (1) extracting information from many sentences in a row; (2) integrating incoming inputs to appropriately update information about entities and their states.

The first problem is currently being tackled by continuously increasing the models’ context window, i.e., the number of words they can process in one go. This approach will inevitably run into computational challenges: when summarizing a book, having a model that simultaneously attends to each word in that book is vastly inefficient (although see some attempts to overcome this issue, e.g., [ 169 ] ). A human-like solution to this problem might include hierarchical processing, e.g., generating a summary for each chapter and then for the whole book (for related approaches, see 170 , 171 ).

Even when LLMs operate over shorter spans of text that easily fit inside their context windows, the question is: can they update their internal representations to track changes in the world? Some evidence suggests that they can [ 172 ] , although LLMs make characteristically non-human-like mistakes when it comes to situation modeling: for instance, their outputs can refer to non-existent discourse entities (“Arthur doesn’t own a dog. The dog is brown.” [ 173 ] ). Thus, whether robust situation model building over shorter span of text is feasible using an LLM-only architecture remains a matter of debate.

### Social reasoning

“Water!”

Wittgenstein famously used single-word utterances like this to show that linguistic meaning radically depends on context. Although this word’s literal meaning is straightforward, the intended meanings are more varied. Is the word being gasped by a thirsty person in the desert? By a hiker warning his friend of a hidden stream? An impatient diner talking to a waiter? Work in cognitive science and linguistics has come to recognize that these context-dependent aspects of language are not just peripheral but a central part of human language production and understanding [ 28 , 12 ] .

The set of skills required to infer the intended meaning of an utterance beyond its literal content is known as pragmatics. Pragmatics likely engages a variety of neural mechanisms [ 174 , 175 , 176 ] , including both the language network and other brain regions. Thus, different types of pragmatic reasoning can be classified either as formal or as functional competence. Here, we focus on one core functional competence capacity required for pragmatics: social reasoning.

Humans. A wealth of neuroscientific evidence shows that the human brain has dedicated machinery for processing social information [ 44 , 177 ] . The most relevant to our current discussion is the theory of mind network [ 178 ] , a set of brain regions that are engaged when their owner is attempting to infer somebody’s mental state (with our without the use of language; [ 179 , 180 ] ). The specific contributions of the theory of mind network to language understanding can be divided into two broad categories. First, just like other functionally specialized brain modules, it is engaged when processing semantic content that is specifically related to its domain: narratives that require inferring the mental state of the characters engage the theory of mind network [ 180 ] , and texts that require inferring the characters’ intentions evoke greater activity than those that do not [ 181 , 182 ] . Second, the theory of mind network is engaged more strongly during nonliteral language comprehension, including phenomena like jokes, sarcasm, indirect speech, and conversational implicature [ 183 , 176 ] —in other words, in situations where understanding the meaning of an utterance requires inferring the intentions of the speaker. Thus, successful language understanding relies on our broader, non-language-specific social inference skills.

LLMs. Recent models, trained with RLHF, have shown strong performance in interpreting non-literal utterances, such as metaphors and polite deceits, suggesting they can reach human or near-human performance on at least some pragmatic tasks [ 184 ] . That said, LLMs exhibit uneven performance across pragmatic domains: their ability to interpret sarcasm or to complete jokes was limited even as their metaphor comprehension abilities soared [ 184 ] . Overall, at least some forms of pragmatic inference might be acquired via targeted fine-tuning. It remains an open question whether aspects of pragmatics that are easiest for LLMs are those that are supported by the language network in humans.

LLMs’ ability to solve theory of mind tasks has been subject to particular controversy. These tasks require both social knowledge and the ability to maintain a situation model. A typical example would feature character X moving an object from location A to location B while character Y is not around and so, does not see the move. The goal is to predict the true location of the object (location B) and the location where character Y believes the object is (location A). A bold claim that instruction-tuned LLMs have mastered theory-of-mind tasks [ 185 ] was quickly countered by a demonstration that including basic controls (such as character Y being told about the true object location) brought LLM performance to below-chance levels [ 186 ] . Several other studies have identified limitations in LLM performance on theory of mind tasks [ 187 , 188 , 189 , cf. 190 ] . One solution to overcome these limitations has been to augment an LLM with a symbolic tracker of entity states and character beliefs [ 191 ] , an approach that mirrors the separation between language and theory of mind processing in humans.

### Language input can bootstrap functional competence capabilities

Many non-linguistic cognitive capabilities can be substantially enhanced by language input. In humans, this relationship is particularly salient during development: babies learn new conceptual categories more easily when they are accompanied by linguistic labels [ 192 ] , and children with delayed language access have delayed social reasoning abilities [ 193 ] . Even in adulthood, knowledge of specific number words predicts the ability to conceptually represent exact numbers [ 194 ] . Coupled with the fact that language inputs contain vast quantities of information about the world, and that language is both a crucial data source and representational substrate for much of people’s world knowledge, this evidence suggests that, in principle, a model trained exclusively on language input could acquire much of functional linguistic competence.

Thus, we do not argue that functional linguistic competence is out of reach for language-based models; our main goals are (1) to highlight the conceptual distinction between formal and functional linguistic competence—which in the human brain draw on separate neural circuits, and (2) to demonstrate the gulf between LLMs’ formal and functional linguistic abilities. These facts lead to a speculation that, like the human brain, models that can master language use would also require or benefit from separate mechanisms for formal and functional competence. We discuss this idea next.

## Toward models that use language like humans

In this paper, we have advanced the thesis that formal and functional linguistic competence are distinct capabilities, with formal competence relying on distinct language machinery and function competence requiring the integration of diverse brain networks. We have shown that formal competence emerges in contemporary LLMs as a result of the word-in-context prediction objective; however, this objective alone appears insufficient for equipping LLMs with functional linguistic competence skills. Based on the neuroscience evidence, we suggest that models that succeed at real-life language use will need to be modular , mimicking the division of labor between formal and functional competence in the human brain.

We see at least two ways to separate LLM circuits responsible for formal and functional competence: explicitly building modularity into the architecture of the system (we call this Architectural Modularity ) or naturally inducing modularity through the training process, both through the training data and the objective function (we call this Emergent Modularity ).

Architectural Modularity has a long history; it involves stitching together separate components, perhaps with quite specialized architectures [ 195 , 196 ] . Modern-day examples include a transformer language model paired with a separate memory module [ 161 , 197 , e.g., ] or a model for visual question answering, which includes a language module, a vision module, and a reasoning module [ 198 , 199 ] . Such modular models achieve high task performance, are more efficient (i.e., can be trained on smaller datasets and have lower compute demands during inference), and show better generalizability (i.e., perform well on datasets with previously unseen properties). The modules of such models can be trained separately or together, similarly to how humans can flexibly combine different cognitive skills when learning to perform novel complex tasks.

Recently, the desire for this kind of modularity has expanded to include attempts to augment language models with the ability to call separate programs, as in including API calls [ 200 ] , mathematical calculators [ 201 ] , planners [ 202 ] , and other kinds of modules that do specific structured operations.

Another approach in this vein uses LLMs as modules to translate a natural language query into code, which can then be passed to a symbolic module, which then generates an answer. [ 149 ] outline a research program for this approach, showing that a version of GPT-3 fine-tuned to generate both natural language and code (Codex) can translate text input into meaningful structured probabilistic programs; inference in these programs can be used to reason over relational domains (like kinship systems), grounded domains (like visual scenes), and situations that require planning and understanding the plans of others. Their approach demonstrates a promising avenue for integrating what LLMs succeed at (namely, formal linguistic competence) with other cognitive modules that benefit from symbolic structure and abstraction.

The Emergent Modularity approach involves training models end-to-end (similarly to contemporary LLMs) while creating the conditions that facilitate the emergence of specialized model sub-components over the course of training. Modular structure has been shown to spontaneously emerge in some end-to-end neural network systems in domains other than language [ 203 , 204 , e.g., ] , which suggests that emergent modularity may constitute an optimal solution to many complex tasks. One strategy for this approach to be successful is for the model architecture to incentivize the development of individual, specialized modules within the model. Transformers, the most popular architecture today, satisfy this condition to some extent by allowing different attention heads to attend to different input features [ 205 , 206 , 207 , e.g.] ; certain approaches promote modularization even more explicitly, e.g., by endowing transformers with a mixture-of-experts architecture that incentivizes separate “experts” to carry out different computations [ 208 , 209 , 210 ] .

A modular model architecture is much better aligned with the brain’s functional architecture for language, which includes separate components for formal and functional competence. Is it possible to build formally and functionally competent systems that do not mimic the modular structure of the human brain? In theory, yes: systems with different underlying architectures (e.g., modular vs. non-modular) can, in principle, exhibit similar behaviors. However, explicitly disentangling formal and functional competence skills at the architectural level is perhaps the most fail-safe path toward ensuring that an AI model uses language in a humanlike way.

Box 5. The need for better benchmarks To assess progress on the road toward building models that use language in human-like ways, it is important to develop benchmarks that evaluate both formal and functional linguistic competence. This distinction can reduce the confusion that arises when discussing these models by combating the “good at language -> good at thought” and the “bad at thought -> bad at language” fallacies. Several existing benchmarks already evaluate formal linguistic competence in LLMs [ 79 , 76 ] and can be complemented by additional tests of core linguistic features: hierarchy and abstraction. Benchmarks for evaluating different domains of functional linguistic competence, like commonsense world knowledge (e.g., WinoGrande [ 211 ] ), can often be “hacked” by LLMs by leveraging flawed heuristics [ 212 ] . This issue is likely exacerbated in large-scale heterogeneous datasets like BIG-bench [ 5 ] . Moreover, functional competence benchmarks often rely on a certain, often underspecified level of required formal linguistic competence skills and/or mix together different functional competence abilities. Designing benchmarks that carefully disentangle different components of language knowledge and use would therefore constitute an important step toward a more informative assessment of LLMs.

## Concluding remarks

Over the last few years, the discourse around language models has consisted of a curious mix of overclaiming and underclaiming [ 66 ] . While some claim models are on the verge of intelligence, others have pointed out the many failures of LLMs on a broad range of tasks, from number multiplication to generating factually true statements. Here, we have put these seemingly inconsistent reactions in dialog with prior and ongoing work in computational linguistics, cognitive science, and neuroscience. In particular, we argue that LLMs are remarkably successful on tasks that require a particular type of structural and statistical linguistic competence—formal linguistic competence. Although their performance is not yet fully human-like, these models achieve an impressive degree of success in representing and using hierarchical relationships among words and building representations that are sufficiently abstract to generalize to new words and constructions. As such, these LLMs are underused in linguistics as candidate models of human language processing.

We also review some of the LLMs’ failures on tasks that target real-life language use, such as reasoning, while highlighting that the capabilities these tasks require are fundamentally distinct from formal language competence and rely on machinery in the human brain distinct from language processing machinery.