Emergent Analogical Reasoning in Large Language Models — part 5/5

Cleaned-text code points [62152, 74013). Bibliography omitted here; source.md includes it.

### 4.5 Four-term verbal analogies

We evaluated GPT-3 on four separate four-term analogy datasets [ 19 , 20 , 17 , 21 ] . The UCLA-VAT dataset contains 80 problems, with four relation types: categorical (B/D is a member of the category A/C), functional (A/C is the function of B/D), antonym, and synonym. There are 20 problems for each relation type. Each problem contains two answer choices for the final term (D and D’). We evaluated GPT-3 by presenting the problem along with each possible answer choice (A:B::C:D or A:B::C:D’), using the standard colon notation, and selected the answer choice for which GPT-3 assigned a higher log probability to the final term. The problem and GPT-3’s selected answer were then recursively appended to the prompt for the next problem. The problems were presented in a shuffled order. We compared against human behavioral data from [ 17 ] (N=57, UCLA undergraduates). Example problems from each of the four relation categories are shown below:

Categorical vegetable : cabbage :: insect : ? 1. beetle 2. frog

Function drive : car :: burn : ? 1. wood 2. fire

Antonym love : hate :: rich : ? 1. poor 2. wealthy

Synonym rob : steal :: cry : ? 1. weep 2. laugh

The dataset of Sternberg and Nigro [ 19 ] contains 200 problems, including 40 problems for each of five relation types: categorical, functional, antonym, synonym, and linear order. We evaluated GPT-3 in the same way that we did for UCLA-VAT, and compared against human behavioral data from [ 19 ] (N=20, Yale undergraduates). An example problem illustrating the linear order relation type is shown below (the categorical, functional, antonym, and synonym problems were similar to those from the UCLA VAT):

Linear order month : year :: inch : ? 1. foot 2. length

The dataset of SAT problems from Turney et al. [ 20 ] contains 374 problems, covering a range of different relation types. Each problem contains five answer choices for both C and D terms (including the correct answer). We evaluated GPT-3 by presenting each of the five possible analogies for each problem, and selecting the choice for which the C and D terms were assigned the highest log probability. The problem, and GPT-3’s choice, were then appended to the prompt for the next problem. We compared against an estimate of the average performance level for high school students taking the SAT (see [ 40 ] ).

The dataset of Jones et al. [ 21 ] contains 120 problems, including 40 problems for each of three relation types: categorical, causal, and compositional. Half of these problems are categorized as semantically near (A and B are similar to C and D), and half are categorized as semantically far (A and B are dissimilar to C and D). Each problem contains two answer choices. We evaluated GPT-3 in the same way that we did for UCLA-VAT, and compared against human behavioral data from [ 21 ] (N=241, Wayne State University undergraduates). Example problems for each of the three relation categories are shown below:

Categorical diesel : fuel :: bed : ? 1. furniture 2. pillow

Causal motion : sickness :: drought : ? 1. famine 2. rain

Compositional steel : scissors :: apple : ? 1. cider 2. tree

### 4.6 Story analogies

#### 4.6.1 Materials

All story analogy materials were taken from a problem set created by Gentner et al. [ 18 ] (from their Experiment 2), and included in a verbal analogy inventory [ 41 ] . These materials involve 18 source stories. Each source story is accompanied by four potential target stories, forming four conditions: correct and incorrect near analogies (respectively termed ‘literal similarity’ and ‘mere appearance’ by Gentner et al.), both involving similar entities and first-order relations as the source, while differing from each other in higher-order causal relations; and correct and incorrect far analogies (respectively termed ‘true analogy’ and ‘false analogy’ by Gentner et al.), both involving similar first-order relations as the source but distinct entities, while differing from each other in causal relations. An example source story, along with target stories from each condition, is presented below:

Source story: Karla, an old hawk, lived at the top of a tall oak tree. One afternoon, she saw a hunter on the ground with a bow and some crude arrows that had no feathers. The hunter took aim and shot at the hawk but missed. Karla knew the hunter wanted her feathers so she glided down to the hunter and offered to give him a few. The hunter was so grateful that he pledged never to shoot at a hawk again. He went off and shot deer instead. Near analogy – correct target story: Once there was an eagle named Zerdia who nested on a rocky cliff. One day she saw a sportsman coming with a crossbow and some bolts that had no feathers. The sportsman attacked but the bolts missed. Zerdia realized that the sportsman wanted her tailfeathers so she flew down and donated a few of her tailfeathers to the sportsman. The sportsman was pleased. He promised never to attack eagles again. Near analogy – incorrect target story: Once there was an eagle named Zerdia who donated a few of her tailfeathers to a sportsman so he would promise never to attack eagles. One day Zerdia was nesting high on a rocky cliff when she saw the sportsman coming with a crossbow. Zerdia flew down to meet the man, but he attacked and felled her with a single bolt. As she fluttered to the ground Zerdia realized that the bolt had her own tailfeathers on it. Far analogy – correct target story: Once there was a small country called Zerdia that learned to make the world’s smartest computer. One day Zerdia was attacked by its warlike neighbor, Gagrach. But the missiles were badly aimed and the attack failed. The Zerdian government realized that Gagrach wanted Zerdian computers so it offered to sell some of its computers to the country. The government of Gagrach was very pleased. It promised never to attack Zerdia again. Far analogy – incorrect target story: Once there was a small country called Zerdia that learned to make the world’s smartest computer. Zerdia sold one of itssupercomputers to its neighbor, Gagrach, so Gagrach would promise never to attack Zerdia. But one day Zerdia was overwhelmed by a surprise attack from Gagrach. As it capitulated the crippled government of Zerdia realized that the attacker’s missiles had been guided by Zerdian supercomputers.

#### 4.6.2 Human behavioral experiment

Human behavioral data was collected in an online experiment. The experiment was approved by the UCLA Institutional Review Board (IRB protocol #22-000841, approved May 17, 2022), and all participants provided informed consent. All participants were UCLA undergraduates. Fifty-four participants (47 female, 18-44 years old, average age = 20.7 years old) completed the experiment. No statistical methods were used to pre-determine sample sizes. Participants received course credit for their participation.

After receiving instructions, participants were presented with 18 trials, each involving a different source story. On each trial, participants were presented with a source story (referred to as ‘Story 1’), followed by two target stories (referred to as ‘Story A’ and ‘Story B’), and asked ‘Which of Story A and Story B is a better analogy to Story 1?’. Participants could select either story A or story B, or could indicate that they were both equally analogous. Accuracy was computed as the proportion of trials for which participants selected the correct target story.

On half of the trials, the target stories were from the near analogy conditions. On the other half of the trials, the target stories were from the far analogy conditions. The order of the two target stories was randomly shuffled on all trials.

#### 4.6.3 Evaluating GPT-3

GPT-3 was evaluated by entering stories directly into the OpenAI playground . For each source story, GPT-3 was evaluated on both the near analogy comparison, and the far analogy comparison, and was also evaluated on both possible orderings for each pair of target stories, resulting in 18 × 2 × 2 = 72 18\times 2\times 2=72 total comparisons. For each comparison, the stories were presented in the following format: Consider the following story: Story 1: ≪ \ll source story text ≫ \gg Now consider two more stories: Story A: ≪ \ll target story A text ≫ \gg Story B: ≪ \ll target story B text ≫ \gg Which of Story A and Story B is a better analogy to Story 1?

Is the best answer Story A, Story B, or both are equally analogous? where ≪ \ll source story text ≫ \gg , ≪ \ll target story A text ≫ \gg , and ≪ \ll target story B text ≫ \gg were replaced by the text for the corresponding stories. In addition to answering the forced-choice question, GPT-3 sometimes spontaneously produced explanations, but only the forced-choice response was used in our analysis. GPT-3’s context window was cleared after obtaining the results of each comparison.

#### 4.6.4 Evaluating GPT-4

GPT-4 was evaluated by entering stories directly into the ChatGPT web interface . GPT-4 was evaluated on the same 72 problems, using the same format as was used for GPT-3. GPT-4’s context window was cleared after obtaining the results of each comparison.

#### 4.6.5 Statistical analyses

The task performed by both GPT-3 and human participants involved a three-choice discrimination (Story A is more analogous, Story B is more analogous, both are equally analogous). Statistical analyses were carried out to determine whether this discrimination was made at a level greater than expected from chance alone. To be conservative, we assumed a chance performance level of 50% accuracy. For GPT-3, a binomial test was performed (using data at the individual trial level). For human participants, a one-sample t-test was performed (using data averaged at the individual subject level). These analyses were carried out separately for the near analogy and far analogy conditions.

To compare GPT-3 with human performance, a logistic regression analysis was carried out at the individual trial level. The dependent variable was a binary variable coding for whether a particular response was correct or incorrect. A single binary predictor coded for GPT-3 vs. human responses.

### 4.7 Analogical problem-solving

Problems were entered directly into the OpenAI playground . Materials were taken from [ 22 ] and [ 23 ] . All prompts and responses are shown in Supplementary Section S5 . Each subsection shows the results for a single continuous session, with GPT-3’s responses presented in bold text. Responses were not truncated or curated in any way.

## Data Availability

Data for all human behavioral experiments, along with the Digit Matrices, letter string analogy, and UCLA VAT problem sets, can be downloaded from:

https://github.com/taylorwwebb/emergent_analogies_LLM

The four-term verbal analogy problem sets from Sternberg and Nigro [ 19 ] and Jones et al. [ 21 ] , and the story analogy materials from Gentner et al. [ 18 ] can be downloaded from:

http://cvl.psych.ucla.edu/resources/AnalogyInventory.zip

Information about the problem set of SAT four-term verbal analogies from Turney et al. [ 20 ] can be found at:

https://aclweb.org/aclwiki/SAT_Analogy_Questions_(State_of_the_art)

## Code Availability

Code for all simulations can be downloaded from:

https://github.com/taylorwwebb/emergent_analogies_LLM

## Acknowledgements

We would like to thank Bryor Snefjella and Peter Turney for helpful feedback and discussions. Preparation of this paper was supported by NSF grant IIS-1956441 and AFOSR MURI grant FA9550-22-1-0380 to H.L.

## Author Contributions Statement

T.W., K.J.H., and H.L. conceived project and planned experiments. T.W. implemented experiments and analyzed results. T.W., K.J.H., and H.L. drafted manuscript.

## Competing Interests Statement

The authors declare no competing interests.