Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies

arxiv:2101.02235

Original source · Paper JSON

Retrieval, embeddings, benchmarks · 2021-01-06

Cites (0) · Cited by (7)

Complete cleaned Markdown · Source Markdown including bibliography

Read in parts

1. Part 1
2. Part 2
3. Part 3

Contents

• Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
• Abstract
• 1 Introduction
• 2 Strategy Questions
• 2.1 Desiderata
• Multi-step
• Feasible
• Implicit
• Definite
• 2.2 Decomposing Strategy Questions
• 3 Data Collection Pipeline
• 3.1 Creative Question Writing (CQW)
• Task description
• Priming with Wikipedia terms
• Controlling the answer distribution
• Model-in-the-loop filtering
• Auxiliary sub-task
• 3.2 Strategy Question Decomposition (SQD)
• Task description
• Auxiliary sub-task
• 3.3 Evidence Matching (EVM)
• Task description
• Controlling the matched context
• 3.4 Data Verification Mechanisms
• Task qualifications
• Real-time automatic checks
• Inter-task feedback
• Evidence verification task
• 4 The StrategyQA Dataset
• 4.1 Dataset Statistics
• 4.2 Data Quality
• Do questions in StrategyQA require multi-step implicit reasoning?
• Do questions in StrategyQA have a definitive answer?
• What is the quality of the decompositions?
• Would different annotators use the same decomposition strategy?
• Is the evidence for strategy questions in Wikipedia?
• Do matched paragraphs provide evidence?
• Do different annotators match the same evidence paragraphs?
• 4.3 Data Diversity
• Reasoning skills
• Question topics
• 4.4 Human Performance
• 5 Experimental Evaluation
• 5.1 Baseline Models
• Predicting decompositions
• Baseline models
• Online solvers
• 5.2 Results
• Strategy QA performance
• Retrieval evaluation
• Human retrieval performance
• Predicted decompositions
• 6 Related Work
• 7 Conclusion
• Acknowledgement