Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 6/8
Cleaned-text code points [67377, 77691). Bibliography omitted here; source.md includes it.
Task Category Metric List of Tasks Textual Entailment Exact Match task937_defeasible_nli_atomic_textual_entailment task202_multinli_textual_entailment task936_defeasible_nli_atomic_textual_entailment task641_e_snli_textual_entailment task1344_rte_textual_entailment task1615_sick_textual_entailment task1385_anli_textual_entailment task935_defeasible_nli_atomic_textual_entailment task199_multinli_textual_entailment task1388_cb_textual_entailment task1554_scitail_textual_entailment task640_e_snli_textual_entailment task534_farstail_textual_entailment task201_multinli_textual_entailment task1386_anli_textual_entailment task463_pasinlu_textual_entailment task1387_anli_textual_entailment task738_perspectrum_textual_entailment task1529_scitailv1.1_textual_entailment task190_snli_textual_entailment task200_multinli_textual_entailment task1612_sick_textual_entailment task970_sherliic_textual_entailment task890_gwsd_textual_entailment task464_pasinlu_textual_entailment task1516_imppres_textual_entailment task642_e_snli_textual_entailment Cause Effect Classification Exact Match task1178_xcopa_cause_effect_classification task391_cod3s_cause_effect_classification task939_indicnlp_cause_effect_classification task392_cod3s_cause_effect_classification task938_indicnlp_cause_effect_classification task1168_xcopa_cause_effect_classification task828_copa_cause_effect_classification task1628_copa_hr_cause_effect_classification task943_indicnlp_cause_effect_classification task1182_xcopa_cause_effect_classification task1171_xcopa_cause_effect_classification task968_xcopa_cause_effect_classification task942_indicnlp_cause_effect_classification task1181_xcopa_cause_effect_classification task1172_xcopa_cause_effect_classification task1393_copa_cause_effect_classification task1174_xcopa_cause_effect_classification task1627_copa_hr_cause_effect_classification task1177_xcopa_cause_effect_classification task1184_xcopa_cause_effect_classification task1185_xcopa_cause_effect_classification task1176_xcopa_cause_effect_classification task614_glucose_cause_effect_classification task1629_copa_hr_cause_effect_classification task1175_xcopa_cause_effect_classification task827_copa_cause_effect_classification task1173_xcopa_cause_effect_classification task1180_xcopa_cause_effect_classification task1170_xcopa_cause_effect_classification task1183_xcopa_cause_effect_classification task969_xcopa_cause_effect_classification task941_indicnlp_cause_effect_classification task1626_copa_hr_cause_effect_classification task940_indicnlp_cause_effect_classification task393_cod3s_cause_effect_classification task1169_xcopa_cause_effect_classification task1179_xcopa_cause_effect_classification Coreference Resolution Exact Match task1391_winogrande_coreference_resolution task1664_wino_bias_coreference_resolution task304_numeric_fused_head_coreference_resolution task892_gap_coreference_resolution task891_gap_coreference_resolution task330_gap_coreference_resolution task401_numeric_fused_head_coreference_resolution task033_winogrande_coreference_resolution task133_winowhy_coreference_resolution task329_gap_coreference_resolution task249_enhanced_wsc_coreference_resolution task648_winograd_wsc_coreference_resolution task1390_wsc_fiexed_coreference_resolution task893_gap_coreference_resolution Dialogue Act Recognition Exact Match task879_schema_guided_dstc8_dialogue_act_recognition task362_spolin_dialogue_act_recognition task1533_dailydialog_dialogue_act_recognition task1534_dailydialog_dialogue_act_recognition task880_schema_guided_dstc8_dialogue_act_recognition task1531_dailydialog_dialogue_act_recognition task1394_meta_woz_dialogue_act_recognition Answerability Classification Exact Match task020_mctaco_answerability_classification task050_multirc_answerability_classification task1439_doqa_answerability_classification task233_iirc_answerability_classification task226_stack_overflow_answerability_classification task396_persianqa_answerability_classification task1640_adverserial_qa_answerability_classification task232_iirc_answerability_classification task1442_doqa_answerability_classification task242_tweetqa_answerability_classification task1624_disfl_qa_answerability_classification task520_aquamuse_answerability_classification task290_tellmewhy_answerability_classification task349_squad2.0_answerability_classification Word Analogy Exact Match task1155_bard_word_analogy task1152_bard_word_analogy task1158_bard_word_analogy task1156_bard_word_analogy task1157_bard_word_analogy task1159_bard_word_analogy task1153_bard_word_analogy task1154_bard_word_analogy Overlap Extraction ROUGE-L task039_qasc_overlap_extraction task281_points_of_correspondence_overlap_extraction Keyword Tagging ROUGE-L task613_liar_keyword_tagging task645_wiki_auto_all_data_keyword_tagging task620_ohsumed_keyword_tagging task036_qasc_keyword_tagging task623_ohsumed_keyword_tagging Question Rewriting ROUGE-L task670_ambigqa_question_rewriting task121_zest_question_rewriting task1195_disfl_qa_question_rewriting task442_com_qa_question_rewriting task1345_qqp_question_rewriting task035_winogrande_question_rewriting task671_ambigqa_question_rewriting task1562_zest_question_rewriting task1622_disfl_qa_question_rewriting task034_winogrande_question_rewriting task402_grailqa_question_rewriting Title Generation ROUGE-L task1356_xlsum_title_generation task1540_peer_read_title_generation task1659_billsum_title_generation task569_recipe_nlg_title_generation task1342_amazon_us_reviews_title_generation task220_rocstories_title_generation task1561_clickbait_news_bg_title_generation task418_persent_title_generation task1358_xlsum_title_generation task769_qed_title_generation task219_rocstories_title_generation task602_wikitext_title_generation task1586_scifact_title_generation task743_eurlex_title_generation task500_scruples_title_generation task619_ohsumed_title_generation task510_reddit_tifu_dataset_title_generation task288_gigaword_title_generation task1161_coda_19_title_generation Data to Text ROUGE-L task957_e2e_data_to_text task1631_open_pi_data_to_text task1598_nyc_data_to_text task1728_web_nlg_data_to_text task102_commongen_data_to_text task677_ollie_data_to_text task1407_dart_data_to_text task1409_dart_data_to_text task760_msr_sqa_data_to_text Grammar Error Correction ROUGE-L task1557_jfleg_grammar_error_correction Table 5: 12 Evaluation categories (§5.1), their evaluation metrics (Exact Matching or ROUGE-L, §5.1), and all the tasks in each category. 5101 Task Type Textual Entailment Task ID task1344_rte_textual_entailment Definition In this task, you’re given two sentences. Indicate if the first sentence clearly entails the second sentence (i.e., one can conclude the 2nd sentence by reading the 1st one). Indicate your answer with “1” if the first sentence entails the second sentence, otherwise answer with “0”. Positive Ex- ample Input: Sentence 1: No Weapons of Mass Destruction Found in Iraq Yet. Sentence 2:Weapons of Mass Destruction Found in Iraq. Output: 0 Explanation: In our first statement we clearly say that Iraq does not have any weapon of mass destruction but the second sentence says that weapon of mass destruction is found in Iraq which is a contradiction. Hence output will be 0 for non entailment. Negative Ex- ample Input: Sentence 1: Valero Energy Corp., on Monday, said it found "extensive" additional damage at its 250,000- barrel-per-day Port Arthur refinery. Sentence 2: Valero Energy Corp. produces 250,000 barrels per day. Output: 0 Explanation: The first statement mentions that there was damage found in the 250,000 barrel-per-day Port Aurthur refinery. Which means that they produce 250,000 barrels a day. Hence the output should have been 1 for entailment. Instance Input: Sentence 1: Like the United States, U.N. officials are also dismayed that Aristide killed a conference called by Prime Minister Robert Malval in Port-au-Prince in hopes of bringing all the feuding parties together. Sentence 2: Aristide had Prime Minister Robert Malval murdered in Port-au-Prince. Valid Output: [“0”] Table 6: An example task in the Textual Entailment category of our dataset, adopted from RTE (Dagan et al., 2005; Bentivogli et al., 2008). Task Type Cause Effect Classification Task ID task828_copa_cause_effect_classification Definition In this task your given two statements. You must judge whether the second sentence is the cause or effect of the first one. Label the instances as “cause” or “effect” based on your judgment. The sentences are separated by a newline character. Positive Ex- ample Input: The women met for coffee. They wanted to catch up with each other. Output: cause Explanation: The women met for coffee because they wanted to catch up with each other. Negative Ex- ample Input: My body cast a shadow over the grass. The sun was rising. Output: effect Explanation: The rising of the sun isn’t an effect of casting a shadow over the grass. Instance Input: The woman tolerated her friend’s difficult behavior. The woman knew her friend was going through a hard time. Valid Output: [“cause”] Table 7: An example task in the Cause Effect Classification category of our dataset, adopted from COPA (Roemmele et al., 2011). Task Type Coreference Resolution Task ID task1391_winogrande_coreference_resolution Definition In this task, you are given a question containing a blank (_) and two options. You should pick the best option to answer the question. Please answer with “A” or “B”. Positive Ex- ample Input: Katrina gave Christine a stuffed animal for their birthday, but _ already had this one. (A) Katrina (B) Christine Output: B Explanation: Since the blank is someone who received the gift and already had a stuffed animal, the answer must be “Christine”. Negative Ex- ample Input: Kevin had to use less sunscreen when at the beach tanning than Justin because _ had less sensitive skin. (A) Kevin (B) Justin Output: (A) Explanation: Here, an additonal parentheses has been added to the answer. Note that, a valid answer must be “A” or “B”. Instance Input: Benjamin hated being in the sand and just watched Nelson make castle since _ hated to be messy. (A) Benjamin (B) Nelson Valid Output: [“A”] Table 8: An example task in the Cause Effect Classification category of our dataset, adopted from WinoGrande (Sak- aguchi et al., 2020). 5102