Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies — part 1/3

Cleaned-text code points [0, 15668). Bibliography omitted here; source.md includes it.

# Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

###### Abstract

A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly . In this work, we introduce StrategyQA , a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy . A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well ( 87 87 %) on this task, while our best baseline reaches an accuracy of ∼ 66 % \sim 66\% .

## 1 Introduction

Developing models that successfully reason over multiple parts of their input has attracted substantial attention recently, leading to the creation of many multi-step reasoning Question Answering (QA) benchmarks Welbl et al. (2018) ; Talmor and Berant (2018) ; Khashabi et al. (2018) ; Yang et al. (2018) ; Dua et al. (2019) ; Suhr et al. (2019) .

Commonly, the language of questions in such benchmarks explicitly describes the process for deriving the answer. For instance (Figure 1 , Q2), the question “Was Aristotle alive when the laptop was invented?” explicitly specifies the required reasoning steps. However, in real-life questions, reasoning is often implicit . For example, the question “Did Aristotle use a laptop?” (Q1) can be answered using the same steps, but the model must infer the strategy for answering the question – temporal comparison, in this case.

Answering implicit questions poses several challenges compared to answering their explicit counterparts. First, retrieving the context is difficult as there is little overlap between the question and its context (Figure 1 , Q1 and ‘E’). Moreover, questions tend to be short, lowering the possibility of the model exploiting shortcuts in the language of the question. In this work, we introduce StrategyQA , a boolean QA benchmark focusing on implicit multi-hop reasoning for strategy questions , where a strategy is the ability to infer from a question its atomic sub-questions. In contrast to previous benchmarks Khot et al. (2020a) ; Yang et al. (2018) , questions in StrategyQA are not limited to predefined decomposition patterns and cover a wide range of strategies that humans apply when answering questions.

Eliciting strategy questions using crowdsourcing is non-trivial. First, authoring such questions requires creativity . Past work often collected multi-hop questions by showing workers an entire context, which led to limited creativity and high lexical overlap between questions and contexts and consequently to reasoning shortcuts Khot et al. (2020a) ; Yang et al. (2018) . An alternative approach, applied in Natural Questions Kwiatkowski et al. (2019) and MS-MARCO Nguyen et al. (2016) , overcomes this by collecting real user questions. However, can we elicit creative questions independently of the context and without access to users?

Second, an important property in StrategyQA is that questions entail diverse strategies. While the example in Figure 1 necessitates temporal reasoning, there are many possible strategies for answering questions (Table 1 ). We want a benchmark that exposes a broad range of strategies. But crowdsourcing workers often use repetitive patterns, which may limit question diversity.

To overcome these difficulties, we use the following techniques in our pipeline for eliciting strategy questions: (a) we prime crowd workers with random Wikipedia terms that serve as a minimal context to inspire their imagination and increase their creativity; (b) we use a large set of annotators to increase question diversity, limiting the number of questions a single annotator can write; and (c) we continuously train adversarial models during data collection, slowly increasing the difficulty in question writing and preventing recurring patterns Bartolo et al. (2020) .

Beyond the questions, as part of StrategyQA , we annotated: (a) question decompositions : a sequence of steps sufficient for answering the question (‘D’ in Figure 1 ), and (b) evidence paragraphs: Wikipedia paragraphs that contain the answer to each decomposition step (‘E’ in Figure 1 ). StrategyQA is the first QA dataset to provide decompositions and evidence annotations for each individual step of the reasoning process.

Our analysis shows that StrategyQA necessitates reasoning on a wide variety of knowledge domains (physics, geography, etc.) and logical operations (e.g. number comparison). Moreover, experiments show that StrategyQA poses a combined challenge of retrieval and QA, and while humans perform well on these questions, even strong systems struggle to answer them.

In summary, the contributions of this work are: 1. Defining strategy questions – a class of question requiring implicit multi-step reasoning.

2. StrategyQA , the first benchmark for implicit multi-step QA, that covers a diverse set of reasoning skills. StrategyQA consists of 2,780 questions, annotated with their decomposition and per-step evidence.

3. A novel annotation pipeline designed to elicit quality strategy questions, with minimal context for priming workers.

The dataset and codebase are publicly available at https://allenai.org/data/strategyqa .

## 2 Strategy Questions

### 2.1 Desiderata

We define strategy questions by characterizing their desired properties. Some properties, such as whether the question is answerable, also depend on the context used for answering the question. In this work, we assume this context is a corpus of documents, specifically, Wikipedia, which we assume provides correct content.

#### Multi-step

Strategy questions are multi-step questions, that is, they comprise a sequence of single-step questions . A single-step question is either (a) a question that can be answered from a short text fragment in the corpus (e.g. steps 1 and 2 in Figure 1 ), or (b) a logical operation over answers from previous steps (e.g. step 3 in Figure 1 ). A strategy question should have at least two steps for deriving the answer. Example multi- and single- step questions are provided in Table 2 . We define the reasoning process structure in § 2.2 .

#### Feasible

Questions should be answerable from paragraphs in the corpus. Specifically, for each reasoning step in the sequence, there should be sufficient evidence from the corpus to answer the question. For example, the answer to the question “Would a monocle be appropriate for a cyclop?” can be derived from paragraphs stating that cyclops have one eye and that a monocle is used by one eye at the time. This information is found in our corpus, Wikipedia, and thus the question is feasible. In contrast, the question “Does Justin Beiber own a Zune?” is not feasible, because answering it requires going through Beiber’s belongings, and this information is unlikely to be found in Wikipedia.

#### Implicit

A key property distinguishing strategy questions from prior multi-hop questions is their implicit nature. In explicit questions, each step in the reasoning process can be inferred from the language of the question directly. For example, in Figure 1 , the first two questions are explicitly stated, one in the main clause and one in the adverbial clause. Conversely, reasoning steps in strategy questions require going beyond the language of the question. Due to language variability, a precise definition of implicit questions based on lexical overlap is elusive, but a good rule-of-thumb is the following: if the question decomposition can be written with a vocabulary limited to words from the questions, their inflections, and function words, then it is an explicit question. If new content words must be introduced to describe the reasoning process, the question is implicit. Examples for implicit and explicit questions are in Table 2 .

#### Definite

A type of questions we wish to avoid are non-definitive questions, such as “Are hamburgers considered a sandwich?” and “Does chocolate taste better than vanilla?” for which there is no clear answer. We would like to collect questions where the answer is definitive or, at least, very likely, based on the corpus. E.g., consider the question “Does wood conduct electricity?” . Although it is possible that a damp wood will conduct electricity, the answer is generally no .

To summarize, strategy questions are multi-step questions with implicit reasoning (a strategy) and a definitive answer that can be reached given a corpus. We limit ourselves to Boolean yes/no questions, which limits the output space, but lets us focus on the complexity of the questions, which is the key contribution. Example strategy questions are in Table 1 , and examples that demonstrate the mentioned properties are in Table 2 . Next (§ 2.2 ), we describe additional structures annotated during data collection.

### 2.2 Decomposing Strategy Questions

Strategy questions involve complex reasoning that leads to a yes/no answer. To guide and evaluate the QA process, we annotate every example with a description of the expected reasoning process.

Prior work used rationales or supporting facts , i.e., text snippets extracted from the context DeYoung et al. (2020) ; Yang et al. (2018) ; Kwiatkowski et al. (2019) ; Khot et al. (2020a) as evidence for an answer. However, reasoning can rely on elements that are not explicitly expressed in the context. Moreover, answering a question based on relevant context does not imply that the model performs reasoning properly Jiang and Bansal (2019) .

Inspired by recent work Wolfson et al. (2020) , we associate every question-answer pair with a strategy question decomposition . A decomposition of a question q q is a sequence of n n steps ⟨ s ( 1 ) , s ( 2 ) , … , s ( n ) ⟩ \langle s^{(1)},s^{(2)},...,s^{(n)}\rangle required for computing the answer to q q . Each step s ( i ) s^{(i)} corresponds to a single-step question and may include special references , which are placeholders referring to the result of a previous step s ( j ) s^{(j)} . The last decomposition step (i.e. s ( n ) s^{(n)} ) returns the final answer to the question. Table 3 shows decomposition examples.

Wolfson et al. (2020) targeted explicit multi-step questions (first row in Table 3 ), where the decomposition is restricted to a small vocabulary derived almost entirely from the original question. Conversely, decomposing strategy questions requires using implicit knowledge, and thus decompositions can include any token that is needed for describing the implicit reasoning (rows 2-4 in Table 3 ). This makes the decomposition task significantly harder for strategy questions.

In this work, we distinguish between two types of required actions for executing a step. Retrieval : a step that requires retrieval from the corpus, and operation , a logical function over answers to previous steps. In the second row of Table 3 , the first two steps are retrieval steps, and the last step is an operation. A decomposition step can require both retrieval and an operation (see last row in Table 3 ).

To verify that steps are valid single-step questions that can be answered using the corpus (Wikipedia), we collect supporting evidence for each retrieval step and annotate operation steps. A supporting evidence is one or more paragraphs that provide an answer to the retrieval step.

In summary, each example in our dataset contains a) a strategy question, b) the strategy question decomposition, and c) supporting evidence per decomposition step. Collecting strategy questions and their annotations is the main challenge of this work, and we turn to this next.

## 3 Data Collection Pipeline

Our goal is to establish a procedure for collecting strategy questions and their annotations at scale. To this end, we build a multi-step crowdsourcing 1 1 1 We use Amazon Mechanical Turk as our framework. pipeline designed for encouraging worker creativity, while preventing biases in the data.

We break the data collection into three tasks: question writing (§ 3.1 ), question decomposition (§ 3.2 ), and evidence matching (§ 3.3 ). In addition, we implement mechanisms for quality assurance (§ 3.4 ). An overview of the data collection pipeline is in Figure 2 .

### 3.1 Creative Question Writing (CQW)

Generating natural language annotations through crowdsourcing (e.g., question generation) is known to suffer from several shortcomings. First, when annotators generate many instances, they use recurring patterns that lead to biases in the data. Gururangan et al. (2018) ; Geva et al. (2019) . Second, when language is generated conditioned on a long context, such as a paragraph, annotators use similar language Kwiatkowski et al. (2019) , leading to high lexical overlap and hence, inadvertently, to an easier problem. Moreover, a unique property of our setup is that we wish to cover a broad and diverse set of strategies. Thus, we must discourage repeated use of the same strategy.

We tackle these challenges on multiple fronts. First, rather than using a long paragraph as context, we prime workers to write questions given single terms from Wikipedia, reducing the overlap with the context to a minimum. Second, to encourage diversity, we control the population of annotators, making sure a large number of annotators contribute to the dataset. Third, we use model-in-the-loop adversarial annotations Dua et al. (2019) ; Khot et al. (2020a) ; Bartolo et al. (2020) to filter our questions, and only accept questions that fool our models. While some model-in-the-loop approaches use fixed pre-trained models to eliminate “easy” questions, we continuously update the models during data collection to combat the use of repeated patterns or strategies.

We now provide a description of the task, and elaborate on these methods (Figure 2 , upper row).

#### Task description

Given a term (e.g., silk ), a description of the term, and an expected answer (yes or no), the task is to write a strategy question about the term with the expected answer, and the facts required to answer the question.

#### Priming with Wikipedia terms

Writing strategy questions from scratch is difficult. To inspire worker creativity, we ask to write questions about terms they are familiar with or can easily understand. The terms are titles of ‘‘popular’’ 2 2 2 We filter pages based on the number of contributors and the number of backward links from other pages. Wikipedia pages. We provide workers only with a short description of the given term. Then, workers use their background knowledge and web search skills to form a strategy question.

#### Controlling the answer distribution

We ask workers to write questions where the answer is set to be ‘yes’ or ‘no’. To balance the answer distribution, the expected answer is dynamically sampled inversely proportional to the ratio of ‘yes’ and ‘no’ questions collected until that point.

#### Model-in-the-loop filtering