Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies — part 3/3

Cleaned-text code points [31216, 44820). Bibliography omitted here; source.md includes it.

As questions in StrategyQA were triggered by Wikipedia terms, we use the ‘‘instance of’’ Wikipedia property to characterize the topics of questions. 5 5 5 It is usually a 1-to-1 mapping from a term to a Wikipedia category. In cases of 1-to-many, we take the first category. Figure 5 shows the distribution of topic categories in StrategyQA . The distribution shows StrategyQA is very diverse, with the top two categories (“human” and “taxon”, i.e. a group of organisms) covering only a quarter of the data, and a total of 609 topic categories.

We further compare the diversity of StrategyQA to HotpotQA , a multi-hop QA dataset over Wikipedia paragraphs. To this end, we sample 739 pairs of evidence paragraphs associated with a single question in both datasets, and map the pair of paragraphs to a pair of Wikipedia categories using the “instance of” property. We find that there are 571 unique category pairs in StrategyQA , but only 356 unique category pairs in HotpotQA . Moreover, the top two category pairs in both of the datasets (“human-human”, “taxon-taxon”) constitute 8% and 27% of the cases in StrategyQA and HotpotQA , respectively. This demonstrates the creativity and breadth of category combinations in StrategyQA .

### 4.4 Human Performance

To see how well humans answer strategy questions, we sample a subset of 100 questions from StrategyQA and have experts (authors) answer questions, given access to Wikipedia articles and an option to reveal the decomposition for every question. In addition, we ask them to provide a short explanation for the answer, the number of searches they conducted to derive the answer, and to indicate whether they have used the decomposition. We expect humans to excel at coming up with strategies for answering questions. Yet, humans are not necessarily an upper bound because finding the relevant paragraphs is difficult and could potentially be performed better by machines.

Table 7 summarizes the results. Overall, humans infer the required strategy and answer the questions with high accuracy. Moreover, the low number of searches shows that humans leverage background knowledge, as they can answer some of the intermediate steps without search. An error analysis shows that the main reason for failure (10%) is difficulty to find evidence, and the rest of the cases (3%) are due to ambiguity in the question that could lead to the opposite answer.

## 5 Experimental Evaluation

In this section, we conduct experiments to answer the following questions: a) How well do pre-trained language models (LMs) answer strategy questions? b) Is retrieval of relevant context helpful? and c) Are decompositions useful for answering questions that require implicit knowledge?

### 5.1 Baseline Models

Answering strategy questions requires external knowledge that cannot be obtained by training on StrategyQA alone. Therefore, our models and online solvers (§ 3.1 ) are based on pre-trained LMs, fine-tuned on auxiliary datasets that require reasoning. Specifically, in all models we fine-tune RoBERTa Liu et al. (2019) on a subset of: • BoolQ Clark et al. (2019) : A dataset for boolean question answering.

• MNLI Williams et al. (2018) : A large natural language inference (NLI) dataset. The task is to predict if a textual premise entails, contradicts or is neutral with respect to the hypothesis.

• Twenty Questions ( 20Q ): A collection of 50K short commonsense boolean questions. 6 6 6 https://github.com/allenai/twentyquestions

• DROP Dua et al. (2019) : A large dataset for numerical reasoning over paragraphs.

Models are trained in two configurations: • No context : The model is fed with the question only, and outputs a binary prediction using the special CLS token.

• With context : We use BM25 Robertson et al. (1995) to retrieve context from our corpus, while removing stop words from all queries. We examine two retrieval methods: a) question-based retrieval: by using the question as a query and taking the top k = 10 k=10 results, and b) decomposition-based retrieval: by initiating a separate query for each (gold or predicted) decomposition step and concatenating the top k = 10 k=10 results of all steps (sorted by retrieval score). In both cases, the model is fed with the question concatenated to the retrieved context, truncated to 512 tokens (the maximum input length of RoBERTa ), and outputs a binary prediction.

#### Predicting decompositions

We train a seq-to-seq model, termed BART decomp {}_{\text{decomp}} , that given a question, generates its decomposition token-by-token. Specifically, we fine-tune BART Lewis et al. (2020) on StrategyQA decompositions.

#### Baseline models

As our base model, we train a model as follows: We take a RoBERTa Liu et al. (2019) model and fine-tune it on DROP , 20Q and BoolQ (in this order). The model is trained on DROP with multiple output heads, as in Segal et al. (2020) , which are then replaced with a single Boolean output. 7 7 7 For brevity, exact details on model training and hyper-parameters will be released as part of our codebase. We call this model RoBERTa* .

We use RoBERTa* and RoBERTa to train the following models on StrategyQA : without context ( RoBERTa* ∅ ), with question-based retrieval ( RoBERTa* IR-Q {}_{\text{IR-Q}} , RoBERTa IR-Q {}_{\text{IR-Q}} ), and with predicted decomposition-based retrieval ( RoBERTa* IR-D {}_{\text{IR-D}} ).

We also present four oracle models: • RoBERTa* ORA-P {}_{\text{ORA-P}} : uses the gold paragraphs (no retrieval).

• RoBERTa* IR-ORA-D {}_{\text{IR-ORA-D}} : performs retrieval with the gold decomposition.

• RoBERTa* ORA-P-D last-step {}^{\text{last-step}}_{\text{ORA-P-D}} : exploits both the gold decomposition and the gold paragraphs. We fine-tune RoBERTa on BoolQ and SQuAD Rajpurkar et al. (2016) to obtain a model that can answer single-step questions. We then run this model on StrategyQA to obtain answers for all decomposition sub-questions, and replace all placeholder references with the predicted answers. Last, we fine-tune RoBERTa* to answer the last decomposition step of StrategyQA , for which we have supervision.

• RoBERTa* ORA-P-D last-step-raw {}^{\text{last-step-raw}}_{\text{ORA-P-D}} : RoBERTa* that is fine-tuned to predict the answer from the gold paragraphs and the last step of the gold decomposition, without replacing placeholder references.

#### Online solvers

For the solvers integrated in the data collection process (§ 3.1 ), we use three no-context models and two question-based retrieval models. The solvers are listed in Table 8 .

### 5.2 Results

#### Strategy QA performance

Table 9 summarizes the results of all models (§ 5.1 ). RoBERTa* IR-Q {}_{\text{IR-Q}} substantially outperforms RoBERTa IR-Q {}_{\text{IR-Q}} , indicating that fine-tuning on related auxiliary datasets before StrategyQA is crucial. Hence, we focus on RoBERTa* for all other results and analysis.

Strategy questions pose a combined challenge of retrieving the relevant context, and deriving the answer based on that context. Training without context shows a large accuracy gain of 53.9 → 63.6 53.9\rightarrow 63.6 over the majority baseline. This is far from human performance, but shows that some questions can be answered by a large LM fine-tuned on related datasets without retrieval. On the other end, training with gold paragraphs raises performance to 70.7 70.7 . This shows that high-quality retrieval lets the model effectively reason over the given paragraphs. Last, using both gold decompositions and retrieval further increases performance to 72.0 72.0 , showing the utility of decompositions.

Focusing on retrieval-based methods, we observe that question-based retrieval reaches an accuracy of 63.6 63.6 and retrieval with gold decompositions results in an accuracy of 62.0 62.0 . This shows that the quality of retrieval even with gold decompositions is not high enough to improve the 63.6 63.6 accuracy obtained by RoBERTA* ∅ , a model that uses no context. Retrieval with predicted decompositions results in an even lower accuracy of 61.7 61.7 . We also analyze predicted decompositions below.

#### Retrieval evaluation

A question decomposition describes the reasoning steps for answering the question. Therefore, using the decomposition for retrieval may help obtain the relevant context and improve performance. To test this, we directly compare performance of question- and decomposition-based retrieval with respect to the annotated gold paragraphs. We compute Recall@10, i.e., the fraction of the gold paragraphs retrieved in the top-10 results of each method. Since there are 3 annotations per question, we compute Recall@10 for each annotation and take the maximum as the final score. For a fair comparison, in decomposition-based retrieval, we use the top-10 results across all steps.

Results (Table 9 ) show that retrieval performance is low, partially explaining why retrieval models do not improve performance compared to RoBERTa* ∅ , and demonstrating the retrieval challenge in our setup. Gold decomposition-based retrieval substantially outperforms question-based retrieval, showing that using the decomposition for retrieval is a promising direction for answering multi-step questions. Still, predicted decomposition-based retrieval does not improve retrieval compared to question-based retrieval, showing better decomposition models are needed.

To understand the low retrieval scores, we analyzed the query results of 50 random decomposition steps. Most failure cases are due to the shallow pattern matching done by BM25, e.g., failure to match synonyms. This shows that indeed there is little word overlap between decomposition steps and the evidence, as intended by our pipeline design. In other examples, either a key question entity was missing because it was represented by a reference token, or the decomposition step had complex language, leading to failed retrieval. This analysis suggests that advances in neural retrieval might be beneficial for StrategyQA .

#### Human retrieval performance

To quantify human performance in finding gold paragraphs, we ask experts to find evidence paragraphs for 100 random questions. For half of the questions we also provide decomposition. We observe average Recall@10 of 0.586 0.586 and 0.513 0.513 with and without the decomposition, respectively. This shows that humans significantly outperform our IR baselines. However, humans are still far from covering the gold paragraphs, since there are multiple valid evidence paragraphs (§ 4.2 ), and retrieval can be difficult even for humans. Lastly, using decompositions improves human retrieval, showing decompositions indeed are useful for finding evidence.

#### Predicted decompositions

Analysis shows that BART DECOMP {}_{\text{DECOMP}} ’s decompositions are grammatical and well-structured. Interestingly, the model generates strategies, but often applies them to questions incorrectly. E.g., the question “Can a lifeboat rescue people in the Hooke Sea?” is decomposed to “1) What is the maximum depth of the Hooke Sea? 2) How deep can a lifeboat dive? 3) Is #2 greater than or equal to #1?” . While the decomposition is well-structured, it uses a wrong strategy (lifeboats do not dive).

## 6 Related Work

Prior work has typically let annotators write questions based on an entire context Khot et al. (2020a) ; Yang et al. (2018) ; Dua et al. (2019) ; Mihaylov et al. (2018) ; Khashabi et al. (2018) . In this work, we prime annotators with minimal information (few tokens) and let them use their imagination and own wording to create questions. A related priming method was recently proposed by Clark et al. (2020) , who used the first 100 characters of a Wikipedia page.

Among multi-hop reasoning datasets, our dataset stands out in that it requires implicit decompositions. Two recent datasets Khot et al. (2020a) ; Mihaylov et al. (2018) have considered questions requiring implicit facts. However, they are limited to specific domain strategies, while in our work we seek diversity in this aspect.

Most multi-hop reasoning datasets do not fully annotate question decomposition Yang et al. (2018) ; Khot et al. (2020a) ; Mihaylov et al. (2018) . This issue has prompted recent work to create question decompositions for existing datasets Wolfson et al. (2020) , and to train models that generate question decompositions Perez et al. (2020) ; Khot et al. (2020b) ; Min et al. (2019) . In this work, we annotate question decompositions as part of the data collection.

## 7 Conclusion

We present StrategyQA , the first dataset of implicit multi-step questions requiring a wide-range of reasoning skills. To build StrategyQA , we introduced a novel annotation pipeline for eliciting creative questions that use simple language, but cover a challenging range of diverse strategies. Questions in StrategyQA are annotated with decomposition into reasoning steps and evidence paragraphs, to guide the ongoing research towards addressing implicit multi-hop reasoning.

## Acknowledgement

We thank Tomer Wolfson for helpful feedback and the REVIZ team at Allen Institute for AI, particularly Michal Guerquin and Sam Skjonsberg. This research was supported in part by the Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800). Dan Roth is partly supported by ONR contract N00014-19-1-2620 and DARPA contract FA8750-19-2-1004, under the Kairos program. This work was completed in partial fulfillment for the Ph.D degree of Mor Geva.