Emergent Analogical Reasoning in Large Language Models — part 2/5

Cleaned-text code points [15552, 30873). Bibliography omitted here; source.md includes it.

Figure 6 shows the results of this evaluation. GPT-3 showed stronger overall performance than human participants (Figure 6(a) ; logistic regression, main effect of GPT-3 vs. human participants: OR = 1.76 =1.76 , p = 6.3 × 10 − 5 p=6.3\times 10^{-5} , CI = [ 1.34 , 2.31 ] =[1.34,2.31] ), an effect that was driven primarily by stronger performance on zero-generalization problems (main effect of GPT-3 vs. human participants for zero-generalization problems: OR = 1.76 =1.76 , p = 0.0007 p=0.0007 , CI = [ 1.27 , 2.46 ] =[1.27,2.46] ). Performance was strongly affected by the number of generalizations in both GPT-3 and human participants (main effect of number of generalizations, GPT-3: OR = 0.51 =0.51 , p = 2 × 10 − 16 p=2\times 10^{-16} , CI = [ 0.45 , 0.57 ] =[0.45,0.57] ; human participants: OR = 0.66 =0.66 , p = 5.9 × 10 − 16 p=5.9\times 10^{-16} , CI = [ 0.6 , 0.73 ] =[0.6,0.73] ). GPT-3 and human participants also showed similar error patterns across transformation types (Figure 6(b) ) and generalization types (Figure 6(c) ), as quantified by a correlation analysis for accuracy across different problem subtypes ( r ⁡ ( 39 ) = 0.7 r(39)=0.7 , p = 3.6 × 10 − 7 p=3.6\times 10^{-7} ).

We also investigated a novel variant on letter string problems involving generalization from letters to real-world concepts (Figure 5(c) ). GPT-3 showed strong performance on these problems, though with some discrepancies for different transformation types (Figure 6(d) ). These results suggest that GPT-3 has developed an abstract notion of successorship that can be flexibly generalized between different domains (e.g., alphabetic successorship vs. temperature successorship).

One important caveat is that GPT-3’s performance on this task was somewhat sensitive to the way in which problems were formatted. For instance, performance suffered when no prompt was provided (Supplementary Figure 5(a) ), or when problems were presented in the form of a complete sentence (Supplementary Figure 5(b) ). However, even in these cases, GPT-3’s zero-shot performance was both within the range of human participants (within one standard deviation), and closely matched the pattern of human performance across problem types (correlation analysis, no prompt: r ⁡ ( 39 ) = 0.6 r(39)=0.6 , p = 5.3 × 10 − 5 p=5.3\times 10^{-5} , sentence format: r ⁡ ( 39 ) = 0.76 r(39)=0.76 , p = 4.2 × 10 − 6 p=4.2\times 10^{-6} ).

a

b

c

d

### 2.3 Four-term verbal analogies

Though matrix reasoning and letter string analogies involve a high degree of relational complexity, one limitation is that they consist of highly constrained, synthetic relations, such as alphabetic or numerical successorship. GPT-3’s ability to solve problems involving more real-world concepts (e.g., ‘a b c → \rightarrow a b d, cold cool warm → \rightarrow ?’) suggests that its analogical capabilities may not be limited to such artificial settings. To further evaluate GPT-3’s capacity to reason about real-world relational concepts, we tested it on four-term verbal analogy problems involving a broader range of semantic relations.

We evaluated GPT-3 on four separate datasets [ 19 , 20 , 17 , 21 ] . To the best of our knowledge, these constitute an exhaustive set of four-term verbal analogy problems for which human behavioral data is available [ 41 ] . Each dataset contains a series of four-term analogy problems in the form ‘A:B::C:?’, together with a set of answer choices (i.e., potential choices of D). For each problem, GPT-3 was evaluated by presenting the problem together with each potential answer choice, and selecting the option for which GPT-3 assigned a higher log probability. The problem and GPT-3’s choice were then appended to the context window for the next problem, thereby simulating any contextual effects that might arise when solving multiple problems in a row, as human participants typically do.

Figure 7 shows the results for all datasets. GPT-3 performed as well or better than human participants (minimum education level of high-school graduation, located in the United States and recruited using Amazon Mechanical Turk) on the UCLA Verbal Analogy Test (VAT) [ 17 ] , involving categorical, functional, antonym, and synonym relations (Figure 7(a) ), and on a dataset from Sternberg and Nigro [ 19 ] involving these same four relation types and linear order relations (Figure 7(b) ). On a dataset of SAT analogy problems from Turney et al. [ 20 ] , GPT-3 surpassed the estimated average level of performance for high school students taking the SAT (Figure 7(c) ). GPT-3 also showed performance in the same range as human participants (though numerically weaker) on a problem set from Jones et al. [ 21 ] involving categorical, compositional, and causal relations (Figure 7(d) ).

In addition to displaying generally strong performance on these problem sets, GPT-3 also displayed sensitivity to semantic content similar to that observed in human participants. In the dataset from Jones et al. [ 21 ] (Figure 7(d) ), participants performed worse on problems in which the analogs were semantically distant (i.e., the A and B terms had low semantic similarity to C and D), an effect that was also displayed by GPT-3 (logistic regression, effect of semantic distance for GPT-3: OR = 3.24 =3.24 , p = 0.0165 p=0.0165 , CI = [ 1.24 , 8.5 ] =[1.24,8.5] ). These results align with a more general phenomenon in which human reasoning is facilitated by semantically meaningful or coherent content [ 42 , 24 ] .

### 2.4 Story analogies

Human reasoners are able not only to form analogies between individual concepts, but can also identify correspondences between complex real-world events, involving many entities and relations. When making such comparisons, human reasoning is especially sensitive to higher-order relations – relations between relations – notably causal relations between events. Such higher-order relations play a central role in some cognitive theories of analogy [ 43 ] , and it is thus important to establish whether GPT-3 displays a similar sensitivity to them.

To address this question, we tested GPT-3 on a set of story analogies from Gentner et al. [ 18 ] . In each set, a source story is compared to two potential target stories, each of which is matched with the source story in terms of first-order relations, but only one of which shares the same causal relations as the source (see Methods Section 4.6.1 for examples). Gentner et al. found that human participants rated the target stories as more similar when they shared the same causal relations as the source story. These problems are further defined by two different comparison conditions. In the near analogy condition (referred to as ‘literal similarity’ vs. ‘mere appearance’ by Gentner et al.), the target stories also share the same basic entities as the source story, making for a less abstract, within-domain comparison. In the far analogy condition (referred to as ‘true analogy’ vs. ‘false analogy’ by Gentner et al.), the target stories involve different entities from the source story, but share first-order relations, resulting in a more challenging, cross-domain comparison.

To facilitate a direct comparison with GPT-3, we performed a new behavioral study with these materials. For each source story, participants indicated which of two target stories was more analogous. Both GPT-3 and human participants (N=54, UCLA undergraduates) showed a sensitivity to higher-order relations (Figure 8 ), most often selecting the target story that shared causal relations with the source (combined near and far analogy; GPT-3, binomial test: p = 0.0005 p=0.0005 ; human participants, one-sample t-test: t ⁡ ( 53 ) = 21.3 t(53)=21.3 , p = 1.1 × 10 − 27 p=1.1\times 10^{-27} ; null hypotheses for both tests is chance-level performance of 0.5). This effect was significant for both GPT-3 and human participants in the near analogy condition (GPT-3, binomial test: p = 0.0039 p=0.0039 ; human participants, one-sample t-test: t ⁡ ( 53 ) = 21.5 t(53)=21.5 , p = 8.5 × 10 − 28 p=8.5\times 10^{-28} ), but only human participants showed a significant effect in the far analogy condition (GPT-3, binomial test: p = 0.065 p=0.065 ; human participants, one-sample t-test: t ⁡ ( 53 ) = 16.7 t(53)=16.7 , p = 9.3 × 10 − 23 p=9.3\times 10^{-23} ).

Unlike the other task domains considered in the present work, this was a case in which college students clearly outperformed GPT-3 (logistic regression, main effect of GPT-3 vs. human participants: OR = 0.37 =0.37 , p = 0.0003 p=0.0003 , CI = [ 0.21 , 0.63 ] =[0.21,0.63] ). Indeed, a significant number of participants (15/54) selected the analogous story on every trial. However, in an initial investigation of GPT-4 [ 44 ] , we found that it displays stronger performance on this task, more robustly picking the analogous story even in the far analogy condition, and displaying nearly perfect performance in the near analogy condition (Supplementary Figure 6 , Section S4.3 ). It therefore seems likely that further scaling of large language models will enhance their sensitivity to causal relations.

### 2.5 Analogical problem-solving

In everyday thinking and reasoning, analogical comparisons are often made for the purpose of achieving some goal, or solving a novel problem. Thus far, our tests of GPT-3 have assessed its capacity for identifying analogies in text-based inputs with varying formats, but can GPT-3 also use these analogies to derive solutions to novel problems, as human reasoners do?

As a preliminary investigation of this issue, we performed a qualitative evaluation using a paradigm developed by Gick and Holyoak [ 22 ] . In that paradigm, participants are presented with a target problem in the form of a story. In the original study, Duncker’s radiation problem was used [ 45 ] . In that problem, a doctor wants to use radiation to destroy a malignant tumor, but destroying the tumor with a single high-intensity ray will also damage the surrounding healthy tissue. The solution – to use several low-intensity rays that converge at the site of the tumor – is rarely identified spontaneously, but participants are more likely to discover this solution when they are first presented with an analogous source story. In the original study, the source story involved a general who wants to capture a fortress ruled by an evil dictator, but cannot do so by sending his entire army along a single road, which would trigger landmines. The general instead breaks his army up into small groups that approach the fortress from multiple directions, thus avoiding triggering the mines.

We first presented GPT-3 with the target problem in isolation. GPT-3 proposed a solution that involved injecting a radiation source directly into the tumor, rather than identifying the intended solution based on the convergence of multiple low-intensity radiation sources (Supplementary Section S5.1 ). However, when first presented with the general story, followed by the target problem, GPT-3 correctly identified the convergence solution (Supplementary Section S5.2 ). GPT-3 was further able to correctly explain the analogy, and to identify the specific correspondences between the source story and target problem when prompted (e.g., general ↔ \leftrightarrow doctor, dictator ↔ \leftrightarrow tumor, army ↔ \leftrightarrow rays). We also found similar results when using distinct source analogs taken from another study [ 46 ] (Supplementary Section S5.3 ).

In a more challenging version of this paradigm, participants were first presented with both the general story, and two other non-analogous stories intended to serve as distractors. In this context, human participants were much less likely to identify the convergence solution. However, when given a prompt to explicitly consider the previously presented stories when trying to solve the radiation problem, participants were often able to correctly identify the analogous general story, and use this analogy to devise the convergence solution. Remarkably, we found that GPT-3 displayed these same effects. When presented with these same distracting, non-analogous stories, GPT-3 no longer identified the convergence solution, instead proposing the same solution that it proposed in response to the radiation problem alone (Supplementary Section S5.4 ). But when prompted to consider the previous stories, GPT-3 both correctly identified the general story as most relevant, and proposed the convergence solution (Supplementary Section S5.5 ).

We also evaluated GPT-3 using materials from a developmental study that employed a similar paradigm [ 23 ] . In that study, children were tasked with transferring gumballs from one bowl to another bowl that was out of reach, and provided with a number of materials for doing so (e.g., a posterboard, an aluminum walking cane, a cardboard tube), permitting multiple possible solutions. The key result was that when children were first presented with an analogous source story (about a magical genie trying to transfer jewels between two bottles), they were more likely to identify a solution to the target problem that was analogous to the events described in the source story.

When presented with this target problem, GPT-3 mostly proposed elaborate, but mechanically nonsensical solutions, with many extraneous steps, and no clear mechanism by which the gumballs would be transferred between the two bowls (Supplementary Sections S5.6 - S5.8 ). However, when asked to explicitly identify an analogy between the source story and target problem, GPT-3 was able to identify all of the major correspondences, even though it could not use this analogy to discover an appropriate solution. This finding suggests that GPT-3’s difficulty with this problem likely stems from its lack of physical reasoning skills, rather than being due to a difficulty with analogical mapping per se. It is also worth noting that in the original study, this task was presented to children with real physical objects, which likely aided the physical reasoning process relative to the purely text-based input provided to GPT-3. Overall, these results provide some evidence that GPT-3 is capable of using analogies for the purposes of problem-solving, but its ability to do so is constrained by the content about which it can reason, with particular difficulty in the domain of physical reasoning.

## 3 Discussion

We have presented an extensive evaluation of analogical reasoning in a state-of-the-art large language model. We found that GPT-3 appears to display an emergent ability to reason by analogy, matching or surpassing human performance across a wide range of text-based problem types. These included a novel problem set (Digit Matrices) modeled closely on Raven’s Progressive Matrices, where GPT-3 both outperformed human participants, and captured a number of specific signatures of human behavior across problem types. Because we developed the Digit Matrix task specifically for this evaluation, we can be sure GPT-3 had never been exposed to problems of this type, and therefore was performing zero-shot reasoning. GPT-3 also displayed an ability to solve analogies based on more meaningful relations, including four-term verbal analogies and analogies between stories describing complex real-world events.