Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies — part 2/3

Cleaned-text code points [15668, 31216). Bibliography omitted here; source.md includes it.

To ensure questions are challenging and reduce recurring language and reasoning patterns, questions are only accepted when verified by two sets of online solvers . We deploy a set of 5 pre-trained models (termed Ptd ) that check if the question is too easy. If at least 4 out of 5 answer the question correctly, it is rejected. Second, we use a set of 3 models (called Fntd ) that are continuously fine-tuned on our collected data and are meant to detect biases in the current question set. A question is rejected if all 3 solvers answer it correctly. The solvers are RoBERTa Liu et al. (2019) models fine-tuned on different auxiliary datasets; details in § 5.1 .

#### Auxiliary sub-task

We ask workers to provide the facts required to answer the question they have written, for several reasons: 1) it helps workers frame the question writing task and describe the reasoning process they have in mind, 2) it helps reviewing their work, and 3) it provides useful information for the decomposition step (§ 3.2 ).

### 3.2 Strategy Question Decomposition (SQD)

Once a question and the corresponding facts are written, we generate the strategy question decomposition (Figure 2 , middle row). We annotate decompositions before matching evidence in order to avoid biases stemming from seeing the context.

The decomposition strategy for a question is not always obvious, which can lead to undesirable explicit decompositions. For example, a possible explicit decomposition for Q1 (Figure 1 ) might be (1) What items did Aristotle use? (2) Is laptop in #1? ; but the first step is not feasible. To guide the decomposition, we provide workers with the facts written in the CQW task to show the strategy of the question author. Evidently, there can be many valid strategies and the same strategy can be phrased in multiple ways – the facts only serve as a soft guidance.

#### Task description

Given a strategy question, a yes/no answer, and a set of facts, the task is to write the steps needed to answer the question.

#### Auxiliary sub-task

We observe that in some cases, annotators write explicit decompositions, which often lead to infeasible steps that cannot be answered from the corpus. To help workers avoid explicit decompositions, we ask them to specify, for each decomposition step, a Wikipedia page they expect to find the answer in. This encourages workers to write decomposition steps for which it is possible to find answers in Wikipedia, and leads to feasible strategy decompositions, with only a small overhead (the workers are not required to read the proposed Wikipedia page).

### 3.3 Evidence Matching (EVM)

We now have a question and its decomposition. To ground them in context, we add a third task of evidence matching (Figure 2 , bottom row).

#### Task description

Given a question and its decomposition (a list of single-step questions), the task is to find evidence paragraphs on Wikipedia for each retrieval step. Operation steps that do not require retrieval (§ 2.2 ) are marked as operation .

#### Controlling the matched context

Workers search for evidence on Wikipedia. We index Wikipedia 3 3 3 We use the Wikipedia Cirrus dump from 11/05/2020. and provide a search interface where workers can drag-and-drop paragraphs from the results shown on the search interface. This guarantees that annotators choose paragraphs we included in our index, at a pre-determined paragraph-level granularity.

### 3.4 Data Verification Mechanisms

#### Task qualifications

For each task, we hold qualifications that test understanding of the task, and manually review several examples. Workers who follow the requirements are granted access to our tasks. Our qualifications are open to workers from English speaking countries who have high reputation scores. Additionally, the authors regularly review annotations to give feedback and prevent noisy annotations.

#### Real-time automatic checks

For CQW, we use heuristics to check question validity, e.g., whether it ends with a question mark, and that it doesn’t use language that characterizes explicit multi-hop questions (for instance, having multiple verbs). For SQD, we check that the decomposition structure forms a directed acyclic graph, i.e. (i) each decomposition step is referenced by (at least) one of the following steps, such that all steps are reachable from the last step; and (ii) steps don’t form a cycle. In the EVM task, a warning message is shown when the worker marks an intermediate step as an operation (an unlikely scenario).

#### Inter-task feedback

At each step of the pipeline, we collect feedback about previous steps. To verify results from the CQW task, we ask workers to indicate whether the given answer is incorrect (in the SQD, EVM tasks), or if the question is not definitive (in the SQD task) (§ 2.1 ). Similarly, to identify non-feasible questions or decompositions, we ask workers to indicate if there is no evidence for a decomposition step (in the EVM task).

#### Evidence verification task

After the EVM step, each example comprises a question, its answer, decomposition and supporting evidence. To verify that a question can be answered by executing the decomposition steps against the matched evidence paragraphs, we construct an additional evidence verification task (EVV). In this task, workers are given a question, its decomposition and matched paragraphs, and are asked to answer the question in each decomposition step purely based on the provided paragraphs. Running EVV on a subset of examples during data collection, helps identify issues in the pipeline and in worker performance.

## 4 The StrategyQA Dataset

We run our pipeline on 1,799 Wikipedia terms, allowing a maximum of 5 questions per term. We update our online fine-tuned solvers ( Fntd ) every 1K questions. Every question is decomposed once, and evidence is matched for each decomposition by 3 different workers. The cost of annotating a full example is $4.

To encourage diversity in strategies used in the questions, we recruited new workers throughout data collection. Moreover, periodic updates of the online solvers prevent workers from exploiting shortcuts, since the solvers adapt to the training distribution. Overall, there were 29 question writers, 19 decomposers, and 54 evidence matchers participating in the data collection.

We collected 2,835 questions, out of which 55 were marked as having an incorrect answer during SQD (§ 3.2 ). This results in a collection of 2,780 verified strategy questions, for which we create an annotator-based data split Geva et al. (2019) . We now describe the dataset statistics (§ 4.1 ), analyze the quality of the examples, (§ 4.2 ) and explore the reasoning skills in StrategyQA (§ 4.3 ).

### 4.1 Dataset Statistics

We observe (Table 4 ) that the answer distribution is roughly balanced (yes/no). Moreover, questions are short ( < 10 <10 words), and the most common trigram occurs in roughly 1 % 1\% of the examples. This indicates that the language of the questions is both simple and diverse. For comparison, the average question length in the multi-hop datasets HotpotQA Yang et al. (2018) and ComplexWebQuestions Talmor and Berant (2018) is 13.7 13.7 words and 15.8 15.8 words, respectively. Likewise, the top trigram in these datasets occurs in 9.2% and 4.8% of their examples, respectively.

More than half of the generated questions are filtered by our solvers, pointing to the difficulty of generating good strategy questions. We release all 3,305 filtered questions as well.

To characterize the reasoning complexity required to answer questions in StrategyQA , we examine the decomposition length and the number of evidence paragraphs. Figure 3 and Table 4 (bottom) show the distributions of these properties are centered around 3-step decompositions and 2 evidence paragraphs, but a considerable portion of the dataset requires more steps and paragraphs.

### 4.2 Data Quality

#### Do questions in StrategyQA require multi-step implicit reasoning?

To assess the quality of questions, we sampled 100 random examples from the training set, and had two experts (authors) independently annotate whether the questions satisfy the desired properties of strategy questions (§ 2.1 ). We find that most of the examples (81%) are valid multi-step implicit questions, 82% of questions are implicit, and 95.5% are multi-step (Table 5 ).

#### Do questions in StrategyQA have a definitive answer?

We let experts review the answers to 100 random questions, allowing access to the Web. We then ask them to state for every question whether they agree or disagree with the provided answer. We find that the experts agree with the answer in 94% of the cases, and disagree only in 2%. For the remaining 4%, either the question was ambiguous, or the annotators could not find a definite answer on the Web. Overall, this suggests that questions in StrategyQA have clear answers.

#### What is the quality of the decompositions?

We randomly sampled 100 decompositions and asked experts to judge their quality. Experts judged if the decomposition is explicit or utilizes a strategy. We find that 83% of the decompositions validly use a strategy to break down the question. The remaining 17% decompositions are explicit, however, in 14% of the cases the original question is already explicit. Second, experts checked if the phrasing of the decomposition is “natural”, i.e., it reflects the decomposition of a person that does not already know the answer. We find that 89% of the decompositions express a “natural” reasoning process, while 11% may depend on the answer. Last, we asked experts to indicate any potential logical flaws in the decomposition, but no such cases occurred in the sample.

#### Would different annotators use the same decomposition strategy?

We sample 50 examples, and let two different workers decompose the questions. Comparing the decomposition pairs, we find that a) for all pairs, the last step returns the same answer, b) in 44 out of 50 pairs, the decomposition pairs follow the same reasoning path , and c) in the other 6 pairs, the decompositions either follow a different reasoning process (5 pairs) or one of the decompositions is explicit (1 pair). This shows that different workers usually use the same strategy when decomposing questions.

#### Is the evidence for strategy questions in Wikipedia?

Another important property is whether questions in StrategyQA can be answered based on context from our corpus, Wikipedia, given that questions are written independently of the context. To measure evidence coverage, in the EVM task (§ 3.3 ), we provide workers with a checkbox for every decomposition step, indicating whether only partial or no evidence could be found for that step. Recall that three different workers match evidence for each decomposition step. We find that 88.3% of the questions are fully covered: evidence was matched for each step by some worker. Moreover, in 86.9% of the questions, at least one worker found evidence for all steps. Last, in only 0.5% of the examples, all three annotators could not match evidence for any of the steps. This suggests that overall, Wikipedia is a good corpus for questions in StrategyQA , that were written independently of the context.

#### Do matched paragraphs provide evidence?

We assess the quality of matched paragraphs by analyzing both example-level and step-level annotations. First, we sample 217 decomposition steps with their corresponding paragraphs matched by one of the three workers. We let 3 different crowdworkers decide whether the paragraphs provide evidence for the answer to that step. We find that in 93% of the cases, the majority vote is that the evidence is valid. 4 4 4 With moderate annotator agreement of κ = 0.42 \kappa=0.42 .

Next, we analyze annotations of the verification task (§ 3.4 ), where workers are asked to answer all decomposition steps based only on the matched paragraphs. We find that the workers could answer sub-questions and derive the correct answer in 82 out of 100 annotations. Moreover, in 6 questions indeed there was an error in evidence matching, but another worker that annotated the example was able to compensate for the error, leading to 88% of the questions where evidence matching succeeds. In the last 12 cases indeed evidence is missing, and is possibly absent from Wikipedia.

Lastly, we let experts review the paragraphs matched by one of the three workers to all the decomposition steps of a question, for 100 random questions. We find that for 79 of the questions the matched paragraphs provide sufficient evidence for answering the question. For 12 of the 21 questions without sufficient evidence, the experts indicated they would expect to find evidence in Wikipedia, and the worker probably could not find it. For the remaining 9 questions, they estimated that evidence is probably absent from Wikipedia.

In conclusion, 93% of the paragraphs matched at the step-level were found to be valid. Moreover, when considering single-worker annotations, ∼ \sim 80% of the questions are matched with paragraphs that provide sufficient evidence for all retrieval steps. This number increases to 88% when aggregating the annotations of three workers.

#### Do different annotators match the same evidence paragraphs?

To compare the evidence paragraphs matched by different workers, we check whether for a given decomposition step, the same paragraph IDs are retrieved by different annotators. Given two non-empty sets of paragraph IDs 𝒫 1 , 𝒫 2 \mathcal{P}_{1},\mathcal{P}_{2} , annotated by two workers, we compute the Jaccard coefficient J ⁡ ( 𝒫 1 , 𝒫 2 ) = | 𝒫 1 ∩ 𝒫 2 | | 𝒫 1 ∪ 𝒫 2 | J(\mathcal{P}_{1},\mathcal{P}_{2})=\frac{|\mathcal{P}_{1}\cap\mathcal{P}_{2}|}{|\mathcal{P}_{1}\cup\mathcal{P}_{2}|} . In addition, we take the sets of corresponding Wikipedia page IDs 𝒯 1 , 𝒯 2 \mathcal{T}_{1},\mathcal{T}_{2} for the matched paragraphs, and compute J ⁡ ( 𝒯 1 , 𝒯 2 ) J(\mathcal{T}_{1},\mathcal{T}_{2}) . Note that a score of 1 is given to two identical sets, while a score of 0 corresponds to sets that are disjoint. The average similarity score is 0.43 for paragraphs and 0.69 for pages. This suggests that evidence for a decomposition step can be found in more than one paragraph in the same page, or in different pages.

### 4.3 Data Diversity

We aim to generate creative and diverse questions. We now analyze diversity in terms of the required reasoning skills and question topic.

#### Reasoning skills

To explore the required reasoning skills in StrategyQA , we sampled 100 examples and let two experts (authors) discuss and annotate each example with a) the type of strategy for decomposing the question, and b) the required reasoning and knowledge skills per decomposition step. We then aggregate similar labels (e.g. botanical → \rightarrow biological) and compute the proportion of examples each strategy/reasoning skill is required for (an example can have multiple strategy labels).

Table 6 demonstrates the top strategies, showing that StrategyQA contains a broad set of strategies. Moreover, diversity is apparent (Figure 4 ) in terms of both domain-related reasoning (e.g. biological and technological) and logical functions (e.g. set inclusion and “is member of”). While the reasoning skills sampled from questions in StrategyQA do not necessarily reflect their prevalence in a “natural” distribution, we argue that promoting research on methods for inferring strategies is an important research direction.

#### Question topics