Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 8/8
Cleaned-text code points [91414, 104025). Bibliography omitted here; source.md includes it.
Task Type Title Generation Task ID task418_persent_title_generation Definition Given a document, generate a short title of the document. The title should convey the main idea/event/topic about which the document is being written. Note that URLs in the text have been replaced with [Link]. Positive Ex- ample Input: In a letter posted on the White House web site Tuesday John Brennan assistant to President Barack Obama for homeland security and counterterrorism said Schmidt will have regular access to the president and play a vital role in the country’s security. Schmidt’s selection comes more than 10 months after Obama declared cyber security a priority and ordered a broad administration review. A senior White House official said Obama was personally involved in the selection process and chose Schmidt because of his unique background and skills. Schmidt will have regular and direct access to the president for cyber security issues the official said. The official spoke on the condition of anonymity to discuss the selection process. At the same time cyber experts and potential job candidates have complained that the position lacks the budgetary and policy-making authority needed to be successful. Schmidt will report to the National Security Council and closely support the National Economic Council on cyber issues. Schmidt’s selection suggests that economic and business interests in the White House held more sway in the selection process. Schmidt president and CEO of the Information Security Forum a nonprofit international consortium that conducts research in information security has served as chief security officer for Microsoft and as cyber security chief for online auction giant eBay. He was reportedly preferred by Lawrence Summers director of the economic council. A good format for the titel can be the simple subject + object + verb. Output: White House picks new cyber coordinator Explanation: The title is relevant to the main topic of document, that is, the selection of Schmidt as the cybersecutiy chief. Negative Ex- ample Input: Lauren Cohan’s Walking Dead fate may be written in the Whiskey Cavalier. While the show hasn’t been picked up yet Cohan hasn’t yet signed on for season 9 of Walking Dead and rumors have circulated for months about her possible exit from the zombie show. Just two days ago show runner Scott Gimple told TV Line that he is incredibly hopeful about working out a new deal with Cohan who has played Maggie since the second season. “These things do happen in TV” he said. But we are talking. Output: Scott Gimple signed Lauren Cohan for season 9 Explanation: Though the topic is relevant and have correct names from the document, this is incorrect fact, as it is mentioned that Cohan hasn’t yet signed. Instance Input: Days after at least 58 people were killed in a Las Vegas mass shooting , Hillary Clinton called for better gun control . Clinton also had some words for President Trump , particularly of his handling of Hurricane Maria and the devastation in Puerto Rico . Clinton , on her book tour for “What Happened ,” called her memoir “a story of resilience .” Fallon also had female staff writers write thank you notes to Clinton . “Thank you , Miley , tonight ’s show writers and all of the women and young girls out there who are smart , strong and deserving of every opportunity ,” Clinton said . As for election night , Clinton said she was disappointed both that she lost and that President Trump won . Valid Output: [“Hillary Clinton calls for gun control after Las Vegas shooting”] Table 15: An example task in the Title Generation category of our dataset, adopted from PerSenT (Bastan et al., 2020). Task Type Data to Text Task ID task957_e2e_data_to_text Definition In this task, we ask you convert a data table of restaurant descriptions into fluent natural-sounding English sentences. The input is a string of key-value pairs; the output should be a natural and grammatical English sentence containing all the information from the input. Positive Ex- ample Input: name[Aromi], eatType[restaurant], food[English], area[city centre] Output: Aromi is an English restaurant in the city centre. Explanation: The output sentence faithfully converts the data in the input into a natural-sounding sentence. Negative Ex- ample Input: name[Blue Spice], eatType[coffee shop], priceRange[more than ˘00a330], customer rating[5 out of 5], area[riverside], familyFriendly[yes], near[Avalon] Output: Blue Spice is a Colombian coffee shop located by the riverside, near Avalon in Boston. Its prices are over ˘00a330. Its customer ratings are 5 out of 5. Explanation: While the output contains most of the information from the input, it hallucinates by adding ungrounded information such as “Colombian” and “Boston”. Instance Input: name[The Mill], eatType[restaurant], area[riverside], near[The Rice Boat] Valid Output: [“A restaurant called The Mill, can be found near the riverside next to The Rice Boat.”] Table 16: An example task in the Data to Text category of our dataset, adopted from E2E (Novikova et al., 2017). . 5106 Task Type Grammar Error Correction Task ID task1557_jfleg_grammar_error_correction Definition In this task, you will be shown an incorrect English sentence. You need to generate a corrected form of the input sentence. Positive Ex- ample Input: The car’s wheel are loose. Output: The car’s wheel is loose. Explanation: The instance of are is replaced by the word is. This makes the sentence grammatically correct. Negative Ex- ample Input: This way is the way to go. Output: This way may be the way to go. Explanation: The example does not correct the misuse of the word way. Instead, it should shorten the sentence to: this is the way to go. Instance Input: I think it ’s harder for successful preson to risk somethnig , thay coluld lost much more then others . Valid Output: [“I think it ’s harder for a successful person to risk something becuase they could lose much more than others .”] Table 17: An example task in the Grammar Error Correction category of our dataset, adopted from JFLEG (Napoles et al., 2017). . 5107 -60 -40 -20 0 20 40 60 80 100 task133_winowhy_coreference_resolution task190_snli_textual_entailment task760_msr_sqa_data_to_text task281_points_of_correspondence_overlap_extraction task201_multinli_textual_entailment task1385_anli_textual_entailment task738_perspectrum_textual_entailment task642_e_snli_textual_entailment task1386_anli_textual_entailment task1407_dart_data_to_text task670_ambigqa_question_rewriting task1387_anli_textual_entailment task671_ambigqa_question_rewriting task1388_cb_textual_entailment task1439_doqa_answerability_classification task199_multinli_textual_entailment task641_e_snli_textual_entailment task1562_zest_question_rewriting task121_zest_question_rewriting task1345_qqp_question_rewriting task1557_jfleg_grammar_error_correction task233_iirc_answerability_classification task1534_dailydialog_dialogue_act_recognition task232_iirc_answerability_classification task362_spolin_dialogue_act_recognition task393_cod3s_cause_effect_classification task1390_wsc_fiexed_coreference_resolution task290_tellmewhy_answerability_classification task1409_dart_data_to_text task614_glucose_cause_effect_classification task937_defeasible_nli_atomic_textual_entailment task304_numeric_fused_head_coreference_resolution task1195_disfl_qa_question_rewriting task1622_disfl_qa_question_rewriting task392_cod3s_cause_effect_classification task442_com_qa_question_rewriting task391_cod3s_cause_effect_classification task1342_amazon_us_reviews_title_generation task677_ollie_data_to_text task020_mctaco_answerability_classification task1533_dailydialog_dialogue_act_recognition task226_stack_overflow_answerability_classification task957_e2e_data_to_text task288_gigaword_title_generation task1728_web_nlg_data_to_text task1358_xlsum_title_generation task1529_scitailv1.1_textual_entailment task349_squad2.0_answerability_classification task1442_doqa_answerability_classification task640_e_snli_textual_entailment task648_winograd_wsc_coreference_resolution task035_winogrande_question_rewriting task102_commongen_data_to_text task1598_nyc_data_to_text task1516_imppres_textual_entailment task034_winogrande_question_rewriting task500_scruples_title_generation task935_defeasible_nli_atomic_textual_entailment task1659_billsum_title_generation task1391_winogrande_coreference_resolution task1531_dailydialog_dialogue_act_recognition task892_gap_coreference_resolution task602_wikitext_title_generation task743_eurlex_title_generation task1624_disfl_qa_answerability_classification task1154_bard_word_analogy task1615_sick_textual_entailment task219_rocstories_title_generation task1554_scitail_textual_entailment task1344_rte_textual_entailment task623_ohsumed_keyword_tagging task050_multirc_answerability_classification task890_gwsd_textual_entailment task828_copa_cause_effect_classification task1159_bard_word_analogy task1356_xlsum_title_generation task510_reddit_tifu_dataset_title_generation task1152_bard_word_analogy task1612_sick_textual_entailment task970_sherliic_textual_entailment task329_gap_coreference_resolution task936_defeasible_nli_atomic_textual_entailment task418_persent_title_generation task1161_coda_19_title_generation task1155_bard_word_analogy task613_liar_keyword_tagging task619_ohsumed_title_generation task1586_scifact_title_generation task1640_adverserial_qa_answerability_classification task401_numeric_fused_head_coreference_resolution task879_schema_guided_dstc8_dialogue_act_recognition task880_schema_guided_dstc8_dialogue_act_recognition task1393_copa_cause_effect_classification task1540_peer_read_title_generation task1631_open_pi_data_to_text task202_multinli_textual_entailment task1153_bard_word_analogy task569_recipe_nlg_title_generation task036_qasc_keyword_tagging task520_aquamuse_answerability_classification task893_gap_coreference_resolution task402_grailqa_question_rewriting task039_qasc_overlap_extraction task1664_wino_bias_coreference_resolution task827_copa_cause_effect_classification task1158_bard_word_analogy task620_ohsumed_keyword_tagging task242_tweetqa_answerability_classification task220_rocstories_title_generation task249_enhanced_wsc_coreference_resolution task1157_bard_word_analogy task1156_bard_word_analogy task200_multinli_textual_entailment task033_winogrande_coreference_resolution task891_gap_coreference_resolution task1394_meta_woz_dialogue_act_recognition task330_gap_coreference_resolution task645_wiki_auto_all_data_keyword_tagging task769_qed_title_generation PerformanceMetricDelta Tk-Instruct > Heuristic Baselines Tk-Instruct < Heuristic Baselines Figure 11: Tk-INSTRUCT ’s per-task performance improvement over the best of two heuristic baselines on the 119 evaluation tasks of the English track. 5108 -20 -10 0 10 20 30 40 50 60 task463_pasinlu_textual_entailment task1171_xcopa_cause_effect_classification task1172_xcopa_cause_effect_classification task1176_xcopa_cause_effect_classification task1168_xcopa_cause_effect_classification task1178_xcopa_cause_effect_classification task1180_xcopa_cause_effect_classification task1182_xcopa_cause_effect_classification task464_pasinlu_textual_entailment task1177_xcopa_cause_effect_classification task1169_xcopa_cause_effect_classification task1174_xcopa_cause_effect_classification task941_indicnlp_cause_effect_classification task968_xcopa_cause_effect_classification task1170_xcopa_cause_effect_classification task1184_xcopa_cause_effect_classification task939_indicnlp_cause_effect_classification task943_indicnlp_cause_effect_classification task396_persianqa_answerability_classification task1185_xcopa_cause_effect_classification task1627_copa_hr_cause_effect_classification task938_indicnlp_cause_effect_classification task940_indicnlp_cause_effect_classification task942_indicnlp_cause_effect_classification task969_xcopa_cause_effect_classification task1175_xcopa_cause_effect_classification task1179_xcopa_cause_effect_classification task1183_xcopa_cause_effect_classification task1629_copa_hr_cause_effect_classification task534_farstail_textual_entailment task1561_clickbait_news_bg_title_generation task1173_xcopa_cause_effect_classification task1181_xcopa_cause_effect_classification task1628_copa_hr_cause_effect_classification task1626_copa_hr_cause_effect_classification PerformanceMetricDelta mTk-Instruct > Heuristic Baselines mTk-Instruct < Heuristic Baselines Figure 12: Tk-INSTRUCT ’s per-task performance improvement over the best of two heuristic baselines on the 35 evaluation tasks of the cross-lingual track. 5109