Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 7/8

Cleaned-text code points [77691, 91414). Bibliography omitted here; source.md includes it.

Task Type Dialogue Act Recognition
Task ID task1394_meta_woz_dialogue_act_recognition
Definition In this task, you are given four sentences: a bot task sentence, a bot role sentence, a user task sentence
and a user role sentence. Your job is to classify given sentences into one of the 47 different domains. The
domains are: “UPDATE_CALENDAR”, “PRESENT_IDEAS”, “MOVIE_LISTINGS”, “AUTO_SORT”,
“GAME_RULES”, “CONTACT_MANAGER”, “BANK_BOT”, “MUSIC_SUGGESTER”, “CHECK_STATUS”,
“PET_ADVICE”, “HOW_TO_BASIC”, “NAME_SUGGESTER”, “QUOTE_OF_THE_DAY_BOT”, “GUI-
NESS_CHECK”, “INSURANCE”, “RESTAURANT_PICKER”, “MAKE_RESTAURANT_RESERV ATIONS”,
“WEDDING_PLANNER”, “SKI_BOT”, “HOME_BOT”, “PLAY_TIMES”, “BUS_SCHEDULE_BOT”,
“WHAT_IS_IT”, “PHONE_PLAN_BOT”, “DECIDER_BOT”, “PHONE_SETTINGS”, “TIME_ZONE”,
“LIBRARY_REQUEST”, “UPDATE_CONTACT”, “CATALOGUE_BOT”, “PROMPT_GENERATOR”,
“SCAM_LOOKUP”, “SPORTS_INFO”, “POLICY_BOT”, “CITY_INFO”, “APARTMENT_FINDER”,
“EVENT_RESERVE”, “SHOPPING”, “EDIT_PLAYLIST”, “LOOK_UP_INFO”, “ORDER_PIZZA”,
“WEATHER_CHECK”, “APPOINTMENT_REMINDER”, “GEOGRAPHY”, “STORE_DETAILS”, “AGREE-
MENT_BOT”, “ALARM_SET”.
Positive Ex-
ample
Input: Bot’s task: Inform the user that the topping they are asking for is unavailable. Bot’s role: You are a bot
designed to help customers order pizza. User’s task: Ask if a certain pizza topping is available. User’s role: You are
interacting with a pizza restaurant bot.
Output: ORDER_PIZZA
Explanation: According to the descriptions of the four sentences, we know that the type of task is ORDER_PIZZA.
Negative Ex-
ample
Input: Bot’s task: Help the user with their pizza order. Bot’s role: You are a bot designed to help customers order
pizza. User’s task: Ask the bot for three different pizzas. User’s role: You are interacting with a pizza restaurant bot.
Output: UPDATE_CALENDAR
Explanation: According to the descriptions of the tasks and roles, we know that the type of task is ORDER_PIZZA,
but the output is UPDATE_CALENDAR, so it is incorrect.
Instance Input: Bot’s task: Tell the user when the movie is playing on Friday night. Bot’s role: You are a a bot designed to
provide movie listings. User’s task: Ask the bot for the movie times for a movie on Friday night. User’s role: You
are interacting with a bot designed to provide movie listings.
Valid Output: [“MOVIE_LISTINGS”]
Table 9: An example task in the Dialogue Act Recognition category of our dataset, adopted from MetaLWOz (Sha-
lyminov et al., 2020).
Task Type Answerability Classification
Task ID task1640_adverserial_qa_answerability_classification
Definition Given a paragraph from a wikipedia article about some topic, and a question related to the topic, determine whether
the question is answerable from the paragraph. If the question is answerable, answer “True”, otherwise, answer
“False”.
Positive Ex-
ample
Input: Another approach to brain function is to examine the consequences of damage to specific brain areas.
Even though it is protected by the skull and meninges, surrounded by cerebrospinal fluid, and isolated from the
bloodstream by the blood 2013 brain barrier, the delicate nature of the brain makes it vulnerable to numerous
diseases and several types of damage. In humans, the effects of strokes and other types of brain damage have been a
key source of information about brain function. Because there is no ability to experimentally control the nature of
the damage, however, this information is often difficult to interpret. In animal studies, most commonly involving
rats, it is possible to use electrodes or locally injected chemicals to produce precise patterns of damage and then
examine the consequences for behavior. Question: What is surrounded by cerebrospinal fluid?
Output: True
Explanation: The paragraph comes from the wikipedia page on the brain. The answer to the question is the brain
which can be found in the paragraph.
Negative Ex-
ample
Input: NASCAR (headquartered in Daytona Beach) begins all three of its major auto racing series in Florida at
Daytona International Speedway in February, featuring the Daytona 500, and ends all three Series in November at
Homestead-Miami Speedway. Daytona also has the Coke Zero 400 NASCAR race weekend around Independence
Day in July. The 24 Hours of Daytona is one of the world’s most prestigious endurance auto races. The Grand Prix
of St. Petersburg and Grand Prix of Miami have held IndyCar races as well. Question: What is the starting time of
NASCAR’s big events?
Output: False
Explanation: This paragraph comes from the wikipedia article on Florida. The answer to the given question is
February which can be found in the paragraph, however the output is given as False.
Instance Input: Another approach to brain function is to examine the consequences of damage to specific brain areas.
Even though it is protected by the skull and meninges, surrounded by cerebrospinal fluid, and isolated from the
bloodstream by the blood 2013 brain barrier, the delicate nature of the brain makes it vulnerable to numerous
diseases and several types of damage. In humans, the effects of strokes and other types of brain damage have been a
key source of information about brain function. Because there is no ability to experimentally control the nature of
the damage, however, this information is often difficult to interpret. In animal studies, most commonly involving
rats, it is possible to use electrodes or locally injected chemicals to produce precise patterns of damage and then
examine the consequences for behavior. Question: What are the benefts of the blood brain barrier?
Valid Output: [“True”]
Table 10: An example task in the Answerability Classification category of our dataset, adopted from Adversari-
alQA (Bartolo et al., 2020).
5103

Task Type Word Analogy
Task ID task1156_bard_word_analogy
Definition Two analogies that relate actions to the tools used to perform the action is given in the form “A : B. C : ?”. “A : B”
relates action A to tool B. Your task is to replace the question mark (?) with the appropriate tool for the given action
C, following the “A : B” relation.
Positive Ex-
ample
Input: eat : fork. cook : ?
Output: pan
Explanation: The given analogy relates actions to the tools used to perform them. A fork can be used to eat. To
cook, a pan can be used.
Negative Ex-
ample
Input: dig : shovel. wash : ?
Output: sink
Explanation: The given analogy relates actions to the tools used to perform them. A knife can be used to cut. To
wash, a sink CANNOT be used.
Instance Input: cut : knife. wash : ?
Valid Output: [“soap”, “washcloth”, “detergent”, “rag”]
Table 11: An example task in the Word Analogy category of our dataset, adopted from BARD (Fulda et al., 2017).
Task Type Overlap Extraction
Task ID task281_points_of_correspondence_overlap_extraction
Definition You will be given three sentences. Read them, then identify a noun phrase (person, place, or thing) or event that is
shared between all three sentences. As the output, write the span of the text corresponding to that phrase in each
sentence. Keep the order of the sentences, that is, your answer should look like: 1: *a phrase from sentence 1* 2: *a
phrase from sentence 2* 3: *a phrase from sentence 3*
Positive Ex-
ample
Input: 1: Four employees of the store have been arrested , but its manager – herself a woman – was still at large
Saturday , said Goa police superintendent Kartik Kashyap . 2: If convicted , they could spend up to three years in
jail , Kashyap said . 3: The four store workers arrested could spend 3 years each in prison if convicted .
Output: 1: Four employees of the store 2: they 3: The four store workers
Explanation: All three mentioned parts refer to the same entity, the four employees. “of the store” in first sentence
must be included, since it is part of the same noun phrase describing the employees.
Negative Ex-
ample
Input: 1: But an Arizona official told CNN Bates never trained with the agency . 2: He didn’t come to Arizona , the
official from the Maricopa County Sheriff ’s Office said , and he certainly didn’t train with us . 3: Maricopa County
Sheriff ’s Office in Arizona says Robert Bates never trained with them .
Output: 1: the agency 3: Maricopa County Sheriff ’s Office in Arizona
Explanation: The two noun phrases given in this example are correct, but there’s no noun phrase from sentence 2.
You should include all three sentences in your response.
Instance Input: 1: The President is headed to Panama for a regional summit , and Julie Pace of The Associated Press reports
one of the big questions is whether he ’ll make history and have a face-to-face meeting with Cuban leader Raul
Castro . 2: And so what the White House is going to be weighing is whether this meeting would be a way to
generate more progress or whether it would be a premature reward for the Castros . 3: White House weighing
whether Obama should meet with Raul Castro .
Valid Output: [“1: Cuban leader Raul Castro 2: the Castros 3: Raul Castro”, “1: face-to-face meeting 2: this
meeting 3: meet”]
Table 12: An example task in the Overlap Extraction category of our dataset, adopted from PointsOfCorrespon-
dence (Lebanoff et al., 2020).
5104

Task Type Keyword Tagging
Task ID task620_ohsumed_keyword_tagging
Definition Given an abstract, generate a keyword (a noun phrase) that best describes the focus or contribution of the paper.
Such keywords can be directly from the given abstract or outside it.
Positive Ex-
ample
Input: Abstract: Our results suggest that ethylene oxide retention after sterilization is increased in cuprammonium
cellulose plate dialyzers containing potting compound. In contrast, cuprammonium cellulose plate dialyzers without
potting compound were characterized by a rapid disappearance of retained ethylene oxide after sterilization. Whether
these findings explain the low incidence of SARD with cuprammonium cellulose plate dialyzers that do not contain
potting material is a matter for continued study and experimentation.
Output: Sterilization
Explanation: This term is directly present in the abstract and it is one of the main topic in it. So can be chosen as
the medical subject heading.
Negative Ex-
ample
Input: Abstract: Our results suggest that ethylene oxide retention after sterilization is increased in cuprammonium
cellulose plate dialyzers containing potting compound. In contrast, cuprammonium cellulose plate dialyzers without
potting compound were characterized by a rapid disappearance of retained ethylene oxide after sterilization. Whether
these findings explain the low incidence of SARD with cuprammonium cellulose plate dialyzers that do not contain
potting material is a matter for continued study and experimentation.
Output: Plasma V olume
Explanation: This term is not directly present in the abstract and it is no way related to the abstract. So can not be
chosen as the medical subject heading. “Cellulose” can be become a mesh term
Instance Input: Abstract: There is controversy regarding the appropriate utilization of health care resources in the man-
agement of tricyclic antidepressant overdosage. Antidepressant overdose patients presenting to the emergency
department (ED) are routinely admitted to intensive care units, but only a small proportion develop cardiac arrhyth-
mias or other complications requiring such an environment. The authors reviewed the findings in 165 patients
presenting to an ED with antidepressant overdose. They found that major manifestations of toxicity on ED evaluation
(altered mental status, seizures, arrhythmias, an d conduction defects) were commonly associated with a complicated
hospital course. Patients with the isolated findings of sinus tachycardia or QTc prolongation had no complications.
No patient experienced a serious toxic event without major evidence of toxicity on ED evaluation and continued
evidence of toxicity during the hospital course. These data support the concept that proper ED evaluation can
identify a large body of patients with trivial ingestions who may not require hospital observation.
Valid Output: [“Antidepressive Agents, Tricyclic”, “Arrhythmia”, “California”, “Electrocardiography”, “Emergen-
cies”, “Emergency Service, Hospital”, “Female”, “Human”, “Length of Stay”, “Male”, “Prognosis”, “Retrospective
Studies”, “Tachycardia, Sinus”]
Table 13: An example task in the Keyword Tagging category of our dataset, adopted from OHSUMED (Hersh et al.,
1994).
Task Type Question Rewriting
Task ID task1622_disfl_qa_question_rewriting
Definition Convert a disfluent question to a proper question. A disfluent question is a question that has some interruptions in it
while framing. A proper question is the correct form of the question without any disfluency.
Positive Ex-
ample
Input: Why was uh where was the Rhine regulated with an upper canal?
Output: Where was the Rhine regulated with an upper canal?
Explanation: The above disfluent question is correctly converted to a proper question. The ’uh’ and the ’why’
before the correction should be removed to make it fluent.
Negative Ex-
ample
Input: When did Maududi exert the least impact no where did he exert?
Output: When did Maududi exert the least impact?
Explanation: The above disfluent question is not correctly converted to a proper question since the original meaning
in the disfluent question is where did Maudidi exert the least impact but the output asks when did Maudidi exert the
least impact.
Instance Input: What kind of, no hold up, what describes the proportionality of acceleration to force and mass?
Valid Output: [“What describes the proportionality of acceleration to force and mass?”]
Table 14: An example task in the Question Rewriting category of our dataset, adopted from Disfl-QA (Gupta et al.,
2021).
5105