Dissociating language and thought in large language models — part 2/5

Cleaned-text code points [15659, 31099). Bibliography omitted here; source.md includes it.

In a 2019 interview, Chomsky remarked [ 50 ] : “We have to ask here a certain question: is [deep learning] engineering or is it science? […] On engineering grounds, it’s kind of worth having, like a bulldozer. Does it tell you anything about human language? Zero.” The view that deep learning models are not of scientific interest remains common in linguistics, and, despite many arguments for integrating such models into research on human language processing and acquisition [ 51 , 52 , 53 ] and a chorus of arguments that they should be taken seriously as linguistic and cognitive models [ 54 , 55 , 56 ] , their integration into language research still encounters resistance.

In this section, we evaluate the performance of LLMs qua language models by asking whether these models have made progress towards achieving formal linguistic competence–the kind of competence supported by the language-selective network in the human brain. We argue that LLMs have turned out to be surprisingly successful at mastering formal competence—qualitatively different in their formal linguistic capacities from models from before roughly 2018 in a way that was predicted by few practitioners in the field, and which was unexpected given longstanding claims that grammatically competent systems would require strong language-specific priors. With surprise comes information: models’ successes are informative for linguistic theorizing.

### Statistical language models: some fundamentals

LLMs arose from several earlier approaches in computational linguistics, including statistical language modeling, word embeddings, and connectionism (an earlier term for the approach that morphed into today’s deep learning). Similar to earlier statistical language models, LLMs are usually first trained on a word prediction task (the same task used for training n-gram models going back to Shannon’s work in the mid-20th century; see [ 57 ] for historical overview). Similar to approaches in distributional semantics and word embeddings (for overviews, see [ 58 , 59 ] ), LLMs represent linguistic information as vectors in a high-dimensional space. Similar to earlier connectionist approaches [ 60 , 61 ] , LLMs are neural networks — a class of machine learning systems that was originally inspired by the human brain and learns its parameters from the input data. All of these approaches stand in contrast to models that use explicit, structured hierarchical representations of syntactic rules (see [ 62 ] for a discussion of these two divergent paradigms).

N-grams and word embedding models achieved some success in various domains in natural language processing (e.g., spelling correction, spam classification, sentiment analysis). However, they never approached human-level performance on general language tasks like text generation, leading to claims that purely statistical approaches would never be able to capture the richness of natural language, particularly in complex syntactic, morphological, and semantic domains [ 63 , e.g.,] . For instance, it has been claimed that statistical approaches, which use linear strings of words as input, are in principle unable to learn rare and complex syntactic features that require representing phrases and sentences hierarchically [ 64 ] . This pessimism is now challenged by LLMs.

LLMs are typically first trained on a training set constructed from a massive amount of text from the web. During pretraining , LLMs have a simple objective: predict a held-out token (the basic unit in LLMs—often but not always corresponding to words or morphemes [ 65 ] ) based on a fixed number of previous tokens. The predicted token is then compared with the ground truth (which token actually occurred in that sentence), and the error signal is propagated back through the model to update its many parameters. The token prediction objective is often used as a pretraining step, and the model is then fine-tuned for a more specific task.

Although it is tempting [ 66 ] to move the goalposts and focus on what these models are still unable to do, we argue that the remarkable advances in LLMs’ ability to capture various linguistic phenomena should not be overlooked. Significant formal linguistic abilities arise in models on the scale of GPT-2 or BERT and seem to plateau at a high level in contemporary LLMs (Box 3.1 ).

Box 1. The path toward formal linguistic competence When did LLMs achieve formal linguistic competence? Table I shows text generations from an n-gram model, an RNN (a recurrent neural network, which were state-of-the-art language models in the mid-2010s, before transformers), GPT-2, GPT-3, and GPT-4. The n-gram model and RNN clearly fail to show formal linguistic competence: the sentences are not grammatical or sensible. In contrast, GPT-2 (a model produced just 8 years after the RNN in Row 2) generates grammatical output: the verbs agree with their subjects, parts of speech are used appropriately, relatively complex syntactic constructions like the relative clause “that has nothing to do with who we are” is appropriately tied to a noun phrase. However, compared to the GPT-3 and GPT-4 output, it makes less sense: what does “everything about being with people” mean? Finally, the outputs of GPT-3 and GPT-4 are both grammatical and semantically coherent, such that they can plausibly be taken for answers generated by a human. Table I . Examples of texts generated by increasingly more advanced language models in response to a prompt “The meaning of life is…” (following [ 67 ] and using their example verbatim in Row 2). Starting with GPT-2, outputs are grammatical and largely use the rules and patterns of English (formal linguistic competence). Model Text 3-gram model trained on Moby Dick The meaning of life is gulped and gone, bleeds with keenest anguish at the same muteness of humanity over all creations. Just so with Ahab, ’says I;’ look here you sabbee me, has every reason to be served. RNN from [ 67 ] The meaning of life is the tradition of the ancient human reproduction: it is less favorable to the good boy for when to remove her bigger. In the show’s agreement unanimously resurfaced. GPT-2 [ 68 ] The meaning of life is something simple that has nothing to do with who we are. Everything about being with people is tied to those characteristics, and people are just machines. GPT-3 davinci-002 [ 69 ] The meaning of life is a mystery to us all, and a question that will never be answered. Life is what we make of it, and it’s up to each of us to find our own meaning in life. GPT-4 (via ChatGPT interface) The meaning of life is subjective and varies greatly depending on personal beliefs, values, and experiences. Some people find meaning in relationships, personal achievements, or spiritual beliefs, while others may see it as a journey of self-discovery, learning, or contributing to the greater good.

Box 2. What about semantics? Does semantics fall under formal or functional linguistic competence? The answer depends on what kind of semantics we are talking about. One meaning of semantics, often associated with compositional and lexical semantics, concerns the way that meaning is derived from words and their combinations. We consider this aspect part of formal linguistic competence. Indeed, the language network in the brain responds both to lexical semantics, i.e., retrieving the meaning of individual words, and to compositional semantics, i.e., constructing the meaning of multi-word utterances [ 38 ] . Insofar as LLMs are highly sensitive to lexical and combinatorial semantics, they parallel the language network. The second meaning of semantics is something closer to“general conceptual knowledge” (used in contexts such as “non-verbal semantics”, e.g., extracting the meaning of a picture). This definition is closely related to the notion of world knowledge, which we discuss in our section on world knowledge. Given the fact that conceptual knowledge and reasoning does not have to operate over linguistic inputs but is nonetheless essential for fluent language use, we classify it under functional linguistic competence.

### Large language models learn core aspects of human language processing

For LLMs to be useful as models of language processing in humans, we must be convinced that the models encode the abstract phonological, morphological, syntactic, and semantic rules that characterize human language (see Box 3.1 for a distinction between “linguistic” and conceptual semantics). Although interesting differences exist between linguistic processing in LLMs and humans [ 70 , 71 ] , there are also important similarities. Here, we review evidence that LLMs succeed as models of formal linguistic competence. We focus primarily on syntax, showing evidence of mastery on grammatical benchmarks as well as evidence for emergent syntactic structure in LLMs. But similar successes, both in performance and emergent structure, have been shown in other linguistic domains (e.g., emergent phonological structure [ 72 ] , productive generation of morphologically complex neologisms [ 73 ] , rich lexical semantic information [ 74 ] , etc.).

#### LLMs perform well on benchmarks of diverse linguistic phenomena

By being trained for word prediction, transformer models learn a lot about the structure of language, including linguistic features that, even recently, were thought to be beyond the scope of statistical models. These models have succeeded not just on tests of general language understanding developed by the NLP community (e.g., GLUE tasks [ 75 ] ), but, critically for our purposes, on tests of linguistic competence in English and other languages with massive corpora available (see Box 3 for discussion of lower-resourced languages).

The benchmark BLiMP [ 76 ] , for instance, contains minimal pairs of grammatical vs. ungrammatical sentences across a diverse range of complex linguistic phenomena like filler-gap dependencies (“Bert knew what many writers find” vs. “*Bert knew that many writers find”) and negative polarity items (“The truck has clearly tipped over” vs. “*The truck has ever tipped over”). Strikingly, a model [ 77 ] submitted to the BabyLM challenge [ 78 ] achieved 86% on BLiMP (cf. human baseline of 89%) despite being trained on an amount of data comparable to what a human child might be exposed to (see Box 3). Models achieve similarly impressive results on other linguistic benchmarks like SyntaxGym [ 79 ] , and there are now dozens of investigations of specific complex linguistic phenomena (some of which we discuss below).

#### LLMs learn hierarchical structure

In human languages, words are combined to make compositional meanings. In a multi-word sentence, the individual words’ meanings do not simply get added linearly one by one. Instead, they can be combined hierarchically into tree-like structures.

The hierarchical structure in language manifests in many ways. One prominent example is non-local feature agreement. In English and many other languages, verbs agree with their subjects. For instance, a plural subject uses the verb “are”, whereas a singular subject uses “is”. A bigram model, which simply stores frequencies of two-word strings, could learn that “The keys are on the table” is more probable than “The keys is on the table” by knowing that “keys are” is more common than “keys is”. But such a model would not be able to learn that the subject and verb agree even if arbitrarily far apart: for instance, “The keys to the old, wooden kitchen cabinet are on the table” has six intervening words between the subject and verb, and yet “are” still agrees with “keys” and not with “cabinet”. However, a model that learns the underlying hierarchical structure of English should be able to keep track of this long-distance subject-verb dependency [ 80 ] .

Today’s LLMs perform long-distance number agreement well above chance, preferring the grammatical over a non-grammatical sentence continuation even in the presence of intervening distractor words [ 81 , 82 ] , although some earlier models can be distracted by frequency effects (such as differences in the frequency between the singular and plural forms [ 83 ] ). In a similar vein, LLMs can handle other constructions that require complex hierarchy, like filler-gap dependencies [ 84 ] . Finally, studies that examine the internal geometry of the models’ sentence representations [ 85 ] , studies that causally intervene on models’ internal representations [ 86 ] , and studies that turn on and off specific model “neurons” [ 87 , 88 ] have provided mechanistic insights into how an LLM might represent hierarchical structure and establish non-local structural dependencies.

#### LLMs learn linguistic abstractions

Following [ 89 ] , we define an abstraction as a generalized linguistic representation—–such as a part-of-speech category (e.g., noun or verb) or grammatical role (e.g., subject or object)–—that goes beyond simple storage of input and allows for generalization. The very notion of subject-verb agreement, outlined in the previous section, relies on the abstract categories of subject and verb. As described in [ 81 ] , in a sentence like “dogs in the neighborhood often… (bark/barks)”, a model might learn a shallow version of the agreement rule, namely, that the collocation of “dogs” and “bark” in the same sentence is more common than “dogs” and barks”. However, a model that has an abstract representation of categories like grammatical subject, grammatical number, and verb should be able to handle long-distance number agreement even for novel combinations of words.

One way to test a model’s knowledge of abstract rules is by using semantically nonsensical sentences, like “The colorless green ideas I ate with the chair… (sleep/sleeps)”. Models have been shown to perform the agreement task well in several languages, even on these semantically anomalous sentences [ 81 ] .

An even more stringent test for linguistic abstraction asks whether LLMs can apply morphosyntactic rules to novel words. A study of BERT’s abstraction capabilities [ 90 ] showed that BERT has some ability to generalize grammatical categories. They give the model novel words, used in phrases, as input (e.g., “the blick” where blick is likely a noun and “they dax” where dax is likely a verb) and test whether, based on the input, the model can generalize the part-of-speech category (e.g., assign a higher score to “I went to a blick” than to “I went to a dax”). They conclude that BERT succeeds partially at this task: it does learn to generalize, but only after repeated examples [but see 91 , 92 , for ways in which the word itself affects compositional ability] . Models also seem to (often) be able to use novel words appropriately [ 69 , 73 ] .

A large body of work has tested linguistic abstraction in LLMs using a method called probing [ 93 , 94 ] . In this literature, a classifier is often trained to take as input internal model representations and then predict as output an abstract category, such as part-of-speech or dependency role. The logic of the probe is to test whether these abstract categories can be successfully recovered from the internal model states. Using this approach, it has been claimed that LLMs “rediscover the classical NLP pipeline” [ 95 ] , learning at various layers features like part-of-speech categories, parses, named entities, and semantic roles (although see [ 96 ] ).