Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 1/8
Cleaned-text code points [0, 15131). Bibliography omitted here; source.md includes it.
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 5085–5109 December 7-11, 2022 ©2022 Association for Computational Linguistics SUPER -NATURAL INSTRUCTIONS : Generalization via Declarative Instructions on 1600+ NLP Tasks ♢Yizhong Wang2 ♢Swaroop Mishra3 ♣Pegah Alipoormolabashi4 ♣Yeganeh Kordi5 Amirreza Mirzaei4 Anjana Arunkumar3 Arjun Ashok6 Arut Selvan Dhanasekaran3 Atharva Naik7 David Stap8 Eshaan Pathak9 Giannis Karamanolakis10 Haizhi Gary Lai11 Ishan Purohit12 Ishani Mondal13 Jacob Anderson3 Kirby Kuznia3 Krima Doshi3 Maitreya Patel3 Kuntal Kumar Pal3 Mehrad Moradshahi14 Mihir Parmar3 Mirali Purohit15 Neeraj Varshney3 Phani Rohitha Kaza3 Pulkit Verma3 Ravsehaj Singh Puri3 Rushang Karia3 Shailaja Keyur Sampat3 Savan Doshi3 Siddhartha Mishra16 Sujan Reddy17 Sumanta Patro18 Tanay Dixit19 Xudong Shen20 Chitta Baral3 Yejin Choi1,2 Noah A. Smith1,2 Hannaneh Hajishirzi1,2 Daniel Khashabi21 1Allen Institute for AI 2Univ. of Washington 3Arizona State Univ. 4Sharif Univ. of Tech. 5Tehran Polytechnic 6PSG College of Tech. 7IIT Kharagpur 8Univ. of Amsterdam 9UC Berkeley 10Columbia Univ. 11Factored AI 12Govt. Polytechnic Rajkot 13Microsoft Research 14Stanford Univ. 15Zycus Infotech 16Univ. of Massachusetts Amherst 17National Inst. of Tech. Karnataka 18TCS Research 19IIT Madras 20National Univ. of Singapore 21Johns Hopkins Univ. Abstract How well can NLP models generalize to a va- riety of unseen tasks when provided with task instructions? To address this question, we first introduce SUPER -NATURAL INSTRUCTIONS ,1 a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collec- tion covers 76 distinct task types, including but not limited to classification, extraction, in- filling, sequence tagging, text rewriting, and text composition. This large and diverse collec- tion of tasks enables rigorous benchmarking of cross-task generalization under instructions— training models to follow instructions on a sub- set of tasks and evaluating them on the remain- ing unseen ones. Furthermore, we build Tk-INSTRUCT , a trans- former model trained to follow a variety of in-context instructions (plain language task def- initions or k-shot examples). Our experiments show that Tk-I NSTRUCT outperforms existing instruction-following models such as Instruct- GPT by over 9% on our benchmark despite being an order of magnitude smaller. We fur- ther analyze generalization as a function of var- ious scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.2 1 Introduction The NLP community has witnessed great progress in building models for generalization to unseen tasks via in-context instructions (Mishra et al., 1SUPER -NATURAL INSTRUCTIONS represents a super- sized expansion of NATURAL INSTRUCTIONS (Mishra et al., 2022b) which had 61 tasks. 2The dataset, models, and a leaderboard can be found at https://instructions.apps.allenai.org. ♢ Co-first authors ♣ Co-second authors •Input:“Context:…‘That's fantastic, I'm glad we came to something we both agree with.’ Utterance: ‘Me too. I hope you have a wonderful camping trip.’” •Output: “Yes” •Explanation: “The participant engages in small talk when wishing their opponent to have a wonderful trip.” •Input: “Context: …‘Sounds good, I need food the most, what is your most needed item?!’ Utterance:‘My item is food too’.” •Output: “Yes” •Explanation: “The utterance onlytakesthe negotiation forward and there is no side talk. Hence, the correct answer is ‘No’.” Definition “...Givenanutteranceandrecentdialoguecontextcontainingpast3 utterances(whereveravailable),output‘Yes’iftheutterance containsthesmall-talkstrategy,otherwiseoutput‘No’.Small-talkis acooperativenegotiationstrategy.Itisusedfordiscussingtopics apartfromthenegotiation,tobuildarapportwiththeopponent.” TaskInstruction •Input: “Context: …‘I am excited to spend time with everyone from camp!’Utterance:‘That’s awesome!I really love being out here with my son.Do you think you could spare some food?’” •ExpectedOutput:“Yes” PositiveExamples NegativeExamples EvaluationInstances Tk-Instruct Figure 1: An example task from SUP-NATINST adopted from Chawla et al. (2021). A successful model is ex- pected to use the provided instructions (including task definition and demonstration examples) to output re- sponses to a pool of evaluation instances. 2022b; Sanh et al., 2022; Wei et al., 2022) using large pretrained language models (Raffel et al., 2020; Brown et al., 2020). As remarkable as mod- els like InstructGPT (Ouyang et al., 2022) are, the contribution of various design choices to their suc- cess is opaque. In particular, the role of super- vised data has remained understudied due to lim- ited data released by the corporate entities behind major models. In addition, it is nearly impossible for the research community to extend and re-train these gigantic models. Addressing these two chal- 5085 Resource → SUP-NATINST (this work) NATINST (Mishra et al., 2022b) CROSS FIT (Ye et al., 2021) PROMPT SOURCE (Bach et al., 2022) FLAN (Wei et al., 2022) INSTRUCT GPT (Ouyang et al., 2022) Has task instructions? ✓ ✓ ✗ ✓ ✓ ✓ Has negative examples? ✓ ✓ ✗ ✗ ✗ ✗ Has non-English tasks? ✓ ✗ ✗ ✗ ✓ ✓ Is public? ✓ ✓ ✓ ✓ ✓ ✗ Number of tasks 1616 61 269 176 62 – Number of instructions 1616 61 – 2052 620 14378 Number of annotated tasks types 76 6 13 13 ∗ 12 10 Avg. task definition length (words) 56.6 134.4 – 24.8 8.2 – Table 1: A comparison of SUP-NATINST to a few notable datasets in the field. We obtain the number of tasks, instructions, and task types of other datasets from their original paper. “–” indicates the fields are not applicable or unknown. Standards for categorizing task types vary across different datasets (see Fig. 2). * PROMPT SOURCE does not provide task type annotation for all their tasks, for which we report only the 13 task types annotated for training T0 (Sanh et al., 2022) instead. Translation Question Answering Program Execution Question Generation Sentiment Analysis Text Categorization Text Matching Toxic Language Detection Misc. Cause Effect Classification Information Extraction Textual Entailment Commonsense Classification Named Entity Recognition Fill in The Blank Text Completion Sentence Composition Title Generation Wrong Candidate Generation Question Understanding Language Identification Sentence Perturbation Answerability Classification Coreference Resolution Summarization Text Quality Evaluation Paraphrasing Text to Code Dialogue Generation Question Rewriting Pos Tagging Word Semantics Story Composition Linguistic Probing Speaker Identification Data to Text Word Analogy Gender Classification Dialogue Act Recognition Stereotype Detection Negotiation Strategy Detection Coherence Classification Ethics Classification Explanation Keyword Tagging Answer Verification Mathematics Word Relation Classification Sentence Ordering Intent Identification Code to Text Text Simplification Dialogue State Tracking Grammar Error Detection Section Classification Fact Verification Stance Detection Overlap Extraction Grammar Error Correction Question Decomposition Number Conversion Irony Detection Speaker Relation Classification Style Transfer Spelling Error Detection Spam Classification Sentence Compression Punctuation Error Detection Poem Generation Paper Review Entity Generation Entity Relation Classification Discourse Connective Identification Discourse Relation Classification Preposition Prediction Sentence Expansion (a) SUP-NATINST (this work) Answer Generation Question Generation Classification Minimal Text Modification Incorrect Answer Generation Verification (b) NATINST QA Multiple Choice QA Extractive Bias and Fairness QA Closed Book Sentiment Summarization NLI Paraphrase Topic Classification Coreference Story Completion Structure to Text Word Sense Disambiguation (c) PROMPT SOURCE (T0 subset) Summarization Translation Natural language inference Misc. Commonsense Sentiment Paraphrase Struct to Text Closed-book QA Coreference Reading comprehension w/ commonsense (d) FLAN Generation Open QA Brainstorming Chat Rewrite Summarization Classification Other Closed QA Extract (e) INSTRUCT GPT Figure 2: Compared to other datasets, SUP-NATINST covers a more diverse range of task types. InstructGPT reports a very coarse categorization of their task types. Bubble size represents the number of tasks of each type in log scale. lenges necessitates the availability of large-scale public benchmarks of a broad range of NLP tasks and their instructions to facilitate developing and evaluating models that can generalize to unseen tasks. In this paper, we construct a meta-dataset (i.e., dataset of datasets; Triantafillou et al., 2019) that consists of a wide variety of NLP tasks with their instructions, and train a model that can perform a new task given the instruction, outperforming InstructGPT (which uses 16× more parameters). Our dataset, SUPER -NATURAL INSTRUCTIONS (SUP-NATINST for short), is a large benchmark of 1,616 NLP tasks and their natural language instruc- tions. It brings in a diverse variety of tasks—76 broad task types spanning 55 different languages. Each task is paired up with an instruction that con- sists of the task definition for mapping an input text to a task output and several examples for demon- strating the desired or undesired output (see Fig.1 as an example task). These tasks and their instruc- tions are contributed by 88 NLP practitioners, in response to our public call. These contributions are consolidated after several rounds of peer-review and crowdsourced feedback to ensure quality. Hav- ing this diverse and large-scale data enables us to carefully split the tasks into training and test sets and systematically study how state-of-the-art methods perform on them. Table 1 and Figure 2 highlight properties of SUP-NATINST compared to relevant benchmarks, emphasizing the diversity of tasks and instruction types in our benchmark. Our model, Tk-INSTRUCT , is a generative model for transforming task inputs given declar- ative in-context instructions (task definition or k- shot examples). It is built by multi-task training of the T5 model (Raffel et al., 2020) over all the task instructions in our training set, and is eval- 5086 uated on unseen tasks in the test set. Interest- ingly, an 11B-parameter Tk-INSTRUCT can out- perform the 175B-parameter InstructGPT model by 9.9 ROUGE-L points when evaluated on 119 unseen English tasks, and the multilingual variant mTk-INSTRUCT outperforms InstructGPT by 13.3 points on 35 non-English tasks (§6.1). According to human evaluation, Tk-INSTRUCT generates re- sponses at least as well as the ground truth for 77% of the testing instances (§6.2), confirming its strong generalization to unseen tasks. The compelling empirical performance of Tk- INSTRUCT confirms the importance of super-sized meta datasets such as our SUP-NATINST to facil- itate research towards generalizable NLP models. We conduct extensive analysis to understand the important factors for this generalization (§7). Our analysis shows that scaling up the diversity of train- ing tasks and the model size are both important for strong generalization to unseen tasks. Finally, we estimate performance upper bounds, suggesting further room for improvement. 2 Related Work Language instructions are a versatile way of defin- ing goals, which is why they have been studied in the context of a variety of applications, such as instructions in grounded environments (Shrid- har et al., 2020; Stepputtis et al., 2020; Min et al., 2022b; Weir et al., 2022) and database com- mands (Kim et al., 2020). Here, we focus on appli- cations of instructions for general NLP tasks. Recent literature has been motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few ex- amples (Ye and Ren, 2021; Bragg et al., 2021) or language definitions (Efrat and Levy, 2020; Weller et al., 2020; Zhong et al., 2021; Mishra et al., 2022b,a; Parmar et al., 2022). Our work is re- lated to the existing benchmarks in this line of work, as delineated in Table 1 along various dimen- sions. Our benchmark extends NATINST (Mishra et al., 2022b) with 26× more tasks and greater va- riety of task types (Fig. 2). While CROSS FIT (Ye et al., 2021) focuses on benchmarking with a few in-context examples, our benchmark also offers task instructions. Concurrent to our work, PROMPT SOURCE (Bach et al., 2022) is another benchmark of tasks and their language instructions (prompts). An im- portant distinction between this benchmark and ours is the phrasing of the task definitions: while PROMPT SOURCE task definitions are relatively concise, our task definitions are collected with the intention of providing a complete definition of each task and therefore are longer (24 tokens vs. 56 to- kens on average; Table 1). More recently, BIG- BENCH (Srivastava et al., 2022) introduces a col- lection of 204 tasks and also provides short task descriptions and input prefixes that can be used for prompting LMs. With little overlap to our collec- tion of tasks, they focus more on finding challeng- ing tasks that can be used to test different behaviors of current LMs. Nevertheless, we believe that all these efforts in collecting different tasks as well as the task instructions are complementary, and the community will benefit from considering different benchmarks. Finally, the well-adopted InstructGPT model (Ouyang et al., 2022) is partially enabled by a large dataset of prompts that are collected via various synthetic data augmentation which, unfor- tunately, is not publicly available. Beyond cross-task generalization, our bench- mark can also be used to study multi-task learn- ing more broadly, which is a longstanding goal for AI (Caruana, 1997). Traditionally, this literature fo- cuses on setups that involve evaluation on tasks that are observed during training (Collobert and Weston, 2008; Hashimoto et al., 2017). More recent studies show promise that large-scale multi-task learning can enable strong generalization to similar tasks via unified encoding (Khashabi et al., 2020; Xie et al., 2022) or better finetuning results on downstream tasks (McCann et al., 2018; Aribandi et al., 2022). Our proposed benchmark provides diverse tasks for studying multi-tasking at a massive scale. 3 S UPER -NATURAL INSTRUCTIONS SUPER -NATURAL INSTRUCTIONS is a meta- dataset (Triantafillou et al., 2019) consisting of a variety of NLP tasks (see Fig. 2a) and instructions that describe them in plain language. Instruction schema. All task instructions follow the same uniform schema (see Fig. 1) which is composed of the following parts: • D EFINITION defines a given task in natural lan- guage. This is a complete definition of how an input text (e.g., a sentence or a document) is ex- pected to be mapped to an output text. • P OSITIVE EXAMPLES are samples of inputs and their correct outputs, along with a short explana- tion for each. 5087