Dissociating language and thought in large language models — part 1/5

Cleaned-text code points [0, 15659). Bibliography omitted here; source.md includes it.

# Dissociating language and thought in large language models

###### Abstract

Large Language Models (LLMs) have come closest among all models to date to mastering human language, yet opinions about their linguistic and cognitive capabilities remain split. Here, we evaluate LLMs using a distinction between formal linguistic competence—knowledge of linguistic rules and patterns—and functional linguistic competence—understanding and using language in the world. We ground this distinction in human neuroscience, which has shown that formal and functional competence rely on different neural mechanisms. Although LLMs are surprisingly good at formal competence, their performance on functional competence tasks remains spotty and often requires specialized fine-tuning and/or coupling with external modules. We posit that models that use language in humanlike ways would need to master both of these competence types, which, in turn, could require the emergence of mechanisms specialized for formal linguistic competence, distinct from functional competence.

* The two lead authors contributed equally to this work.

A Preprint August 24, 2026

Keywords : Large Language Models, language and thought, cognitive neuroscience, linguistic competence, computational modeling

## The language-thought conflation

When we hear a sentence, we typically assume that it was produced by a rational, thinking agent (another person). The sentences that people generate in day-to-day conversations are based on their world knowledge (“Not all birds can fly.”), their reasoning abilities (“You’re 15, you can’t go to a bar.”), and their goals (“Would you give me a ride, please?”). Thus, we often use other people’s statements as a window into their minds.

In 1950, Alan Turing leveraged this tight relationship between language and thought to propose his famous test [ 1 ] . The Turing test uses language as an interface to cognition, allowing a human participant to probe the knowledge and reasoning capacities of two conversation partners to determine which of them is a human and which is a machine. Although the utility of the Turing test has since been questioned, it has undoubtedly shaped the way society today thinks of machine intelligence [ 2 ] .

The popularity of the Turing test, combined with the language-thought coupling in everyday life, has led to several common fallacies related to the language-thought relationship. One fallacy is that an entity (be it a human or a machine) that is good at language must also be good at thinking. If an entity generates long coherent stretches of text, it must possess rich knowledge and reasoning capacities. Let’s call this the “good at language -> good at thought” fallacy. This fallacy has come to the forefront due to the recent rise of Large Language Models (LLMs; see Glossary), including OpenAI’s GPT models, Anthropic’s Claude, and more open alternatives [ 3 ] like Meta’s LLaMa models and EleutherAI’s GPT-J. LLMs today can produce text that is difficult to distinguish from human output, outperform humans at some text comprehension tasks [ 4 , 5 ] , and show superhuman performance on next-word prediction [ 6 ] . As a result, claims have emerged—both in the popular press and in the academic literature—that LLMs are not only a major advance in language processing, but are also showing “sparks of artificial general intelligence” [ 7 ] . However, when evaluating LLMs’ capabilities, it is important to distinguish between their ability to think and their linguistic ability. The “good at language -> good at thought” fallacy makes it easy to confuse the two, leading people to mistakenly attribute intelligence and intentionality to even the most basic dialog systems (e.g., the chatbot Eliza from the 1960s [ 8 ] ).

The contrapositive of this fallacy is that a model that is bad at thinking must also be a bad model of language. Let’s call this the “bad at thought -> bad at language” fallacy. LLMs are commonly criticized for their lack of consistent, generalizable world knowledge [ 9 ] , lack of commonsense reasoning abilities [ 10 ] , and failure to understand what an utterance is really about [ 11 ] . Based on this evidence, some critics suggest that the models’ failure to produce linguistic output that fully captures the richness and sophistication of human thought means that they are not good models of human language .

Both the “good at language -> good at thought” and the “bad at thought -> bad at language” fallacies stem from the conflation of language and thought. This conflation is unsurprising: it is still novel, and thus uncanny, to encounter an entity that generates fluent sentences despite lacking a human identity. Thus, our heuristics for understanding what a language model is doing—heuristics that emerged from our language experience with other humans—are broken.

To mitigate the language-thought conflation fallacies, we propose to systematically distinguish between two kinds of linguistic competence: formal linguistic competence -the knowledge of rules and statistical regularities of language—and functional linguistic competence —the ability to use language in real-world situations. Our motivation for the formal/functional distinction comes from the human brain, where these skills are robustly dissociable. Both formal and functional linguistic competence are essential components of human language use: an effective communicator needs to both generate grammatical, meaningful utterances and strategically use those utterances to achieve diverse, context-dependent goals [ 12 , 13 ] .

Armed with this distinction, we evaluate the capabilities of contemporary LLMs and argue that LLMs exhibit a gap between formal and functional competence skills: for modern LLMs, formal competence in English is near human-level, whereas their functional competence remains patchy, with results depending on specific functional competence domains and on tasks within those domains. Moreover, whereas formal linguistic competence in LLMs improves drastically with the amount of training data, functional linguistic competence improvements with scale are less consistent, such that LLM developers have now shifted away from simple scaling-up of the language prediction task to more specialized methods targeting behaviors of interest (e.g., Reinforcement Learning from Human Feedback: RLHF [ 14 ] ) or coupling an LLM with external specialized modules (leading to so-called “augmented language models”; [ 15 ] ).

Therefore, we posit that the next-word prediction objective allows a model to master formal but not necessarily functional linguistic competence. What is required for mastery of functional competence is harder to pin down, in part because so much of human cognition (commonsense reasoning, scientific knowledge, everyday knowledge) can be conveyed through language and thus learned from language —even if these capacities are not themselves inherently linguistic. As a result, language models acquire a variety of non-linguistic capacities. But the ultimate ceiling for functional competence depends on important open questions about the information contained in the linguistic signal and the mechanisms used to leverage that information, as we discuss.

In the rest of the paper, we develop a framework for evaluating the competence of modern language models from a cognitive science perspective. In the first section, we elaborate on the constructs of formal and functional linguistic competence and motivate this distinction based on the evidence from human neuroscience. In the second section, we discuss the successes of LLMs in achieving formal linguistic competence, showing that models trained on word-in-context prediction capture numerous complex linguistic phenomena. In the third section, we consider several domains required for functional linguistic competence—formal reasoning, world knowledge, situation modeling, and social cognition—on which today’s LLMs often fail, or at least perform worse than humans. In the fourth section, we discuss the implications of our framework for building and evaluating future models of language and thought before summarizing our key conclusions in the final section.

## Formal vs. functional linguistic competence

### What does linguistic competence entail?

#### Formal linguistic competence

We define formal linguistic competence as a set of capacities required to produce and comprehend a given language. Broadly, being formally competent means getting the form of language right: knowing which strings could be valid words of a language (e.g., bnick cannot be a word in English but blick can [ 16 ] ), how to productively combine morphemes to form novel words (e.g., Barack Obama-less-ness but not Barack Obama-ness-less [ 17 ] ), learning enough about word meanings to know which words can go in which slots in a sentence [ 18 ] , and knowing how to combine words into valid sentences.

Because of its centrality in the history of linguistics, it is the last of these (forming words into sentences) that we focus on in our discussion of formal competence. Most users of Standard Written English say, “The dogs in my bedroom are asleep” rather than “The dogs in my bedroom is asleep”, because the verb “to be” must match the number of the noun that is the subject of the sentence (“the dogs”), even though that verb is closer to an intervening, singular noun (“bedroom”). Linguistic competence also requires exquisite sensitivity to the regularitiesof idiosyncratic linguistic constructions. For instance, although English speakers know not to use the indefinite article “a” with plural nouns—making a phrase like “a days” ill-formed—they also know that it is allowed in a special construction where an adjective and a numeral intervene: “a beautiful five days in New York” [ 19 , 20 ] .

Human language users likely learn rules, plus thousands of idiosyncratic constructions [ 21 ] , through some combination of sophisticated statistical learning [ 22 , 23 , 24 ] and innate conceptual and/or linguistic machinery [ 25 , 26 , 27 ] . The result is the human ability to understand and produce grammatical and coherent linguistic utterances.

#### Functional linguistic competence

In addition to being competent in the rules and statistical regularities of language, a competent language user uses language to accomplish goals in the world [ 28 , 12 , 29 ] : to talk about things that can be seen or felt or heard, to reason about diverse topics; to make requests; to cajole, prevaricate, and flatter. People use language in tandem with other perceptual and cognitive systems, such as our senses and our memory, and deploy words as part of a broader communication framework supported by our sophisticated social skills. A formal language system in isolation is useless unless it can interface with the rest of perception, cognition, and action.

The capacities required to use language to do things in the world are distinct from formal competence and depend crucially on non-linguistic cognition (Figure 1 ). Thus, we define functional linguistic competence as non-language-specific cognitive functions that are required when using language in tandem with non-language-specific capacities in real-world circumstances.

### Motivation for the distinction between formal vs. functional linguistic competence

Our motivation for the distinction between formal and functional linguistic competence comes from what we know about the architecture of the human mind. In humans, language is robustly dissociated from the rest of high-level cognition, as well as from perception and action. Below we briefly summarize a body of evidence from cognitive science and neuroscience that supports this dissociation.

#### The language network supports language processing in the human brain

Human language processing draws on a set of interconnected brain areas in the frontal and temporal lobes (typically in the left hemisphere). This language network supports both comprehension (spoken, written, and signed) [ 30 , 31 , 32 , 33 ] and production [ 34 , 35 ] ; is sensitive to linguistic regularities at multiple levels: from phonological/sub-lexical [ 36 ] to phrase/sentence level [ 37 , 38 ] ; and supports linguistic operations related both to the processing of word meanings and to combinatorial semantic and syntactic processing [ 38 , 35 ] . Damage to the language network leads to linguistic deficits [ 39 , 40 ] . This tight link between the language network and language function indicates that these brain regions are responsible for language processing in humans.

#### The language network does not support non-linguistic cognition

The language network is remarkably selective for language. Evidence of a strong dissociation between language processing and non-linguistic abilities comes from two main sources: a) functional brain imaging studies of neurotypical adults, and b) behavioral investigations of individuals with aphasia—a language impairment typically caused by a stroke or degeneration.

Brain imaging techniques like functional MRI (fMRI) are used to observe real-time activity in the language network in healthy individuals. Given its high spatial resolution, fMRI is well-suited to study whether any two cognitive abilities draw on the same brain structures. For example, to ask whether language and mathematical reasoning recruit the same brain areas, we can have participants perform a language task and a math task while in an MRI scanner and then test whether brain regions that are active during language processing are also active when participants solve a math problem. This approach reveals that the language network is extremely selective for language processing: it responds reliably when people listen to, read, or generate sentences, but not when they perform arithmetic tasks, engage in logical reasoning, understand computer programs, listen to music, categorize objects or events, reason about people’s mental states, or process non-verbal communicative information like facial expressions or gestures [ 41 , 42 , 43 , 44 , 37 , 45 , 46 , 47 , 48 ] .

Studies of individuals with aphasia provide a unique opportunity for testing which cognitive capacities rely on linguistic representations. Of particular interest are cases of ‘global aphasia’, which affects both production and comprehension. Individuals with global aphasia exhibit severe linguistic deficits that spare nothing but a small set of words. If some aspects of non-linguistic cognition draw on the same resources as language, then individuals with severe linguistic deficits should invariably exhibit impaired performance on the relevant non-linguistic tasks. However, despite the nearly complete loss of linguistic abilities, individuals with severe aphasia can have intact non-linguistic cognitive abilities: they can play chess, solve arithmetic problems, leverage their world knowledge to perform diverse tasks, reason about cause and effect, and navigate complex social situations [ 49 ] .

In summary, evidence from brain imaging studies and from individuals with aphasia is remarkably consistent: the mechanisms that process language in the human brain do not support non-linguistic cognitive tasks. This sharp dissociation suggests that, in examining language models’ functionality, we should separate their linguistic abilities from their abstract knowledge and reasoning abilities, which can be probed—–and perhaps even learned–—through a linguistic interface, but which require more than formal linguistic competence.

## LLMs have largely mastered formal linguistic competence in English