Dissociating language and thought in large language models — part 5/5

Cleaned-text code points [61867, 72564). Bibliography omitted here; source.md includes it.

The failures of LLMs on non-linguistic tasks do not undermine their utility as models of language processing. After all, the brain areas that support language processing in humans also cannot do math, solve logical problems, or even track the meaning of a story across sentences or paragraphs. If we take the human mind and brain—a good example of generalized intelligence—as a guide, we might expect that future advances in developing intelligent systems will require combining language models with models that represent abstract knowledge and support complex reasoning, rather than expecting a single model (trained with a single word prediction objective) to do it all. Finally, to detect and monitor such advances, we need benchmarks that cleanly separate formal and functional linguistic competence (Box 5 ). Formally and rigorously evaluating functional competence in LLMs will be informative for both science and engineering (see Outstanding Questions).

To those who have argued that most interesting aspects of human language cannot be learned from data alone, we say that LLMs compellingly demonstrate the possibility of learning complex syntactic features from linguistic input (even if, as of now, much more input is required than a typical child gets exposed to). To those who criticize LLMs for their inability to do complex arithmetic or to reason about the world, we say, give language models a break: given a strict separation of language and non-linguistic capabilities in the human mind, we should evaluate these capabilities separately, recognizing successes in formal linguistic competence even when non-linguistic capabilities lag behind. Finally, to those who are looking to improve the state of machine learning systems, we suggest that, instead of, or in addition to, continuously scaling up the models [ 213 ] , more promising solutions will come in the form of modular architectures—built-in or emergent—that, like the human brain, integrate language processing with additional systems that carry out perception, reasoning, and action.

## Glossary

• Abstraction is, for our purposes, a linguistic representation that allows for generalization. Part-of-speech is one such example: words like “dog” and “cat” belong to the abstract category of “nouns”.

• Architectural Modularity involves explicitly building distinct modules into a computational model, with each module responsible for achieving different goals.

• Fine-tuning is a process by which, after a model is pretrained, it receives additional training on new data, often for a specific purpose.

• Formal linguistic competence is the ability to get the form of language right. It includes knowledge of word formation (e.g, phonology and morphology), knowledge of word meaning, and knowledge of rules and statistical patterns for how words combine to create sentences. Note that our use of the term ‘competence’ differs from the classic competence/performance distinction in linguistics, given that in both models and humans separating competence and performance is often difficult.

• Emergent Modularity refers to the natural induction of modularity through the model training process, without explicitly building it into the architecture.

• Functional linguistic competence is the ability to use language to accomplish things in the world. It relies on a host of non-language-specific cognitive domains like formal reasoning, world knowledge, situation tracking, and social cognition.

• Hierarchical structure is a crucial property of language that allows it to be more than just a linear sequence of words. Rather, how words go together in a sentence is better captured by a tree-like structure, where some words and phrases are nested inside larger phrases.

• The language network is the interconnected set of brain regions that respond selectively to language but not to non-linguistic inputs and tasks.

• Large Language Models (LLMs) are models based on deep neural architectures (often but not always transformers) and trained on massive amounts of text using a word-in-context prediction task (sometimes with additional training objectives incorporated during or after the main training process). The term “large” refers to the number of parameters in these models, which ranges from millions to billions, as well as the size of the training data.

• Pretraining is the process by which a model is first trained on a general task (for LLMs, typically a text prediction task) before being trained or used for a more specialized purpose.

• Reinforcement Learning from Human Feedback (RLHF) is a process by which reinforcement learning techniques are used to impart human preferences (e.g., as to which of two model outputs is preferred) to a model. It seems to lead to significant improvements on functional tasks.

• Theory of mind is a cognitive skill that enables thinking and reasoning about the minds of others (i.e., what others know, believe, want, etc.).

• Tokens are the basic units in language models. In earlier language models, they were often words or morphemes. In today’s LLMs, they are often inferred from large amounts of text using an algorithm like Byte Pair Encoding. They can resemble words and morphemes, but sometimes also for subword or linguistically unnatural units.

## Highlights

• Formal linguistic competence (getting the form of language right) and functional linguistic competence (using language to accomplish goals in the world) are distinct cognitive skills.

• The human brain contains a network of areas that selectively support language processing (formal linguistic competence), but not other domains like logical or social reasoning (functional linguistic competence).

• In the late 2010’s, Large Language Models trained on word prediction tasks began achieving unprecedented success in formal linguistic competence, showing impressive performance on linguistic tasks that likely require hierarchy and abstraction.

• Consistent performance on tasks requiring functional linguistic competence is harder to achieve for Large Language Models and often involves augmentations beyond next word prediction.

• Evidence from cognitive science and neuroscience can illuminate the capabilities and limitations of Large Language Models and pave the way toward better, human-like models of both language and thought.

## Outstanding Questions

• How much functional competence can be acquired from the linguistic signal? Humans use language as a substrate for knowledge, and so LLMs acquire non-linguistic information from the linguistic signal. How much information in this signal can be used to bootstrap functional competence? Are there aspects of functional competence that cannot be learned from language at all?

• How can we train competent language models on smaller amounts of data? LLMs have achieved remarkable linguistic competence but they are trained on data very unlike what human children encounter. Although LLMs receive vastly more words (several orders of magnitude), they lack the richly structured and interactive input thought to be essential to child language acquisition. Would benefits emerge from training models in more interactive and human-like ways?

• Will the formal competence successes of LLMs transfer to other world languages? Most LLM evaluations have taken place in English and a handful of other world languages. Building models for lower-resourced languages and evaluating them on both formal and functional dimensions is an important ongoing project.

• How long will the LLM growth continue ? 10 years ago, most researchers in the field would not have predicted that LLMs would be as advanced as they are today. Will current AI approaches lead to further revolutionary achievements in language and thought or would the mastery of functional linguistic competence require radically new approaches?

• Which is more promising: architectural modularity or emergent modularity? If we want to build human-like modular systems, will it require explicitly building in functionally distinct components or can they be induced through end-to-end fine-tuning, e.g. with Reinforcement Learning from Human Feedback (RLHF)?

• How modular are today’s LLMs? We argue that an LLM that achieves both formal and functional competence may need to rely on separate mechanisms for different competence types. Mechanistic interpretability studies can shed light on the extent to which different cognitive tasks might segregate even within today’s LLMs.

• Should LLMs be described as individual language users or as distributions over potential user outputs? There are different ways to think about LLMs in the context of language use: e.g., as individual language users (“agent-based view”) or as tools augmenting human activities, like a calculator (“tool-based view”) [ 214 , 215 , 216 ] . Which of these views will be the most fruitful way to think about LLMs as they gain additional competencies and become more widely used?

• How much grounding do LLMs need to continue improving? How limited are text-only approaches, compared to approaches across different modalities [ 217 , 218 ] ?

• How much can LLMs ultimately tell us about human language and cognition? What are the cases where LLMs fall short as models of human language use? Are these discrepancies solvable or will they require language researchers to develop different paradigms?

## Acknowledgements

For helpful conversations, we thank Jacob Andreas, Alex Warstadt, Dan Roberts, Kanishka Misra, students in the 2023 UT Austin Linguistics 393 seminar, the attendees of the Harvard LangCog journal club, the attendees of the UT Austin Department of Linguistics SynSem seminar, Gary Lupyan, John Krakauer, members of the Intel Deep Learning group, Yejin Choi and her group members, Allyson Ettinger, Nathan Schneider and his group members, the UT NLL Group, attendees of the KUIS AI Talk Series at Koç University in Istanbul, Tom McCoy, attendees of the NYU Philosophy of Deep Learning conference and his group members, Sydney Levine, organizers and attendees of the ILFC seminar, and others who have engaged with our ideas. We also thank Aalok Sathe for help with document formatting and references.

## Funding Sources

KM acknowledges funding from NSF Grant 2104995. AI was supported by funds from the Quest Initiative for Intelligence. EF was supported by NIH awards R01-DC016607, R01-DC016950, and U01-NS121471 and by research funds from the Brain and Cognitive Sciences Department, McGovern Institute for Brain Research, and the Simons Foundation through the Simons Center for the Social Brain.

## Conflicts of Interest

The authors declare no Conflicts of Interest.