Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 5/8

Cleaned-text code points [53884, 67377). Bibliography omitted here; source.md includes it.

Supplemental Material
A Crowdsourcing Human Feedback
We use Amazon Mechanical Turk (AMT) to crowd-
source feedback on the quality of the collected
instructions. We limit our crowdworkers to pre-
dominantly English-speaking countries (USA, UK,
Canada, and Australia), and to those who have fin-
ished over 1K HITs with an approval rating of over
99%.
Fig. 6 shows the crowdsourcing template used
for collecting crowdworker feedback on our in-
structions. We show the instructions (the task defi-
nition, along with positive and negative examples)
followed by forms for their feedback. We allow
the crowdworkers to give us a qualitative measure
of their perceived quality as well as text boxes for
more concrete items (such as typos or phrasings
that may benefit from more clear articulation). For
each task, we solicit the feedback of 3 crowdwork-
ers and then use this feedback to improve the task
definitions or the examples for each task.
B Crowdsourcing Human Judgements of
Generation Quality
We perform a crowdsourcing experiment on Ama-
zon Mechanical Turk (AMT) to assess the quality
of the generated responses of models. Specifically,
we ask crowdworkers to indicate if they prefer the
predicted answer by the model or the ground truth
outputs for each instances. The annotation inter-
face is shown in Fig. 7. It is essentially the same
template used for the quality assessment of the
dataset (§A), except that here the crowdworkers
are shown a pair of responses for each instances—
the reference text (from our benchmark) and the
one generated by the model—turning the task into
a comparative evaluation.
For each instance, we obtain annotations from
an annotator as to whether they prefer either re-
sponse over the other or they would rate them
equally (“tie”). The model receives a credit of 1.0
if the worker favors the model’s prediction at least
as well as the ground truth label (otherwise, the
model would receive a credit of 0.0). The overall
accuracy score for the model is computed by av-
eraging instance-level scores. To reduce the costs,
the human evaluation of our models is done on 60
randomly selected tasks (about half of our evalu-
ation tasks), and on 10 random instances of each
task.
Since it is non-trivial to find non-English speak-
ing crowdworkers (Pavlick et al., 2014), this eval-
uation was restricted to English language tasks.
Therefore, since our task is focused on English
tasks, we required workers to be based in a country
with a population predominantly of native English
speakers (USA, Canada, UK, and Australia) and
have completed at least 5000 HITs with≥99% as-
signment approval rate.
The resulting human-evaluation metric indicates
how often were model predictions equal or pre-
ferred to our ground truth labels. In this evaluation,
the theoretical upper bound is 100% where the
model is rated at least as well as the ground truth.
The results of human evaluation are shown in the
bottom row of Fig. 3.
C Instruction Schema
Our instruction schema is based on that of
NATINST (Mishra et al., 2022b), but we sim-
plify it to make data collection easier. Our
DEFINITION field serves as the union of Mishra
et al. (2022b)’s DEFINITION , THINGS TO AVOID ,
and EMPHASIS & C AUTION . Additionally, we
drop their TITLE and PROMPT as their content is
most often covered by DEFINITION .
D Model Implementation Details
T5 experiments. We use T5 for training our Tk-
INSTRUCT , estimating the performance of the su-
pervised approach and conducting analysis.
Our experiments that finetune the T5-11B model
are conducted based on the Google’s T5 library 9
and we use their T5.1.1.xxl checkpoint10 by default,
which is pre-trained only on C4. 11 These experi-
ments are run on Google V3-256 TPUs using a
batch size of 1,048,576 tokens (1,024 examples), a
constant learning rate of 1e-5 and a total of 1000
steps. Each training run takes 4 hours to complete.
Our analyses that use T5 models smaller than
11B parameters are conducted based on Hug-
gingface’s transformers library and model check-
points12 (Wolf et al., 2020) on GPU machines.
9https://github.com/google-research/
text-to-text-transfer-transformer
10https://console.cloud.google.com/storage/browser/
t5-data/pretrained_models/t5.1.1.xxl
11We also tried to finetune Tk-I NSTRUCT from the T5-LM
checkpoint but the final performance is worse. Therefore, we
decided to use the T5.1.1.xxl checkpoint.
12https://huggingface.co/models?sort=downloads&
search=google%2Ft5
5097

Figure 6: The crowdsourcing template we use to receive feedback on our collected tasks.
Figure 7: Crowdsourcing interface used for human assessment of our baselines (§6.2).
When fine-tuning models, we train them for two
epochs with a batch size of 16 and a constant learn-
ing rate of 1e-5. The maximum input length is
set to 1024, and the maximum output length is set
to 128. These experiments are conducted with 8
A100 GPUs with 48GB GPU memory per each. We
use DeepSpeed13 for model parallelization, with
bfloat16 precision enabled to save the GPU mem-
13https://github.com/microsoft/DeepSpeed
ory. Each training run takes 6 hours to complete.
GPT-3 and InstructGPT experiments. We use
the OpenAI API 14 for conducting the GPT-3 ex-
periments. We use their “davinci” engine for the
GPT-3 language model experiments and their “text-
davinci-001” engine for the InstructGPT experi-
ments. When making the requests, we set the tem-
perature as 0, top_p as 1 and the maximum gen-
14https://beta.openai.com/docs/introduction/overview
5098

eration length as 128. Due to the high cost, we
randomly sample 20 instances from each of our
119 test tasks to estimate the performance of GPT-
3 and InstructGPT. All API requests were made on
May 30, 2022.
Encoding instruction with input For every
problem setup, we map a given instruction It and
an input instance x into a textual format, obtaining
enc(It, x). Each instruction It consists of multiple
elements as described in our instruction schema
(§3). We map each element of the instruction to a
textual format and prepend it to the input instance.
Fig. 8 shows how we encode the full instruction.
We study different combinations of these instruc-
tion elements in §7.2. The encoded instance is
then fed to an encoder-decoder model to predict y:
M : enc(It, x)→ y.
Definition : {{definition}}
Positive Example 1−
input :{{p_ex1.input}}
output :{{p_ex1.output}}
explanation :{{p_ex1.exp}}
Positive Example 2−
···
Negative Example 1−
input :{{n_ex1.input}}
output :{{n_ex1.output}}
explanation :{{n_ex1.exp}}
Negative Example 2−
···
Now complete the following example−
input : {{x.input}}
output :
Figure 8: Encoding task instruction with input.
E Evaluation Metrics
We adopt ROUGE-L as our automatic evaluation
metric in this work. However, it remains a question
for how much ROUGE-L can reflect model’s per-
formance on different tasks. Although we cannot
test ROUGE-L’s correlation with each task-specific
metric of the tasks included in our data, we do in-
vestigate whether ROUGE-L can be used for clas-
sification tasks. Fig. 9 plots the ROUGE-L scores
and accuracy of several models on different types
of tasks. These task types are usually regarded as
classification tasks and have very short ground truth
output. We can see that for all these task types, the
trend of ROUGE-L correlates well with the trend of
accuracy. For some task types, we do see some gap
between these two metrics. The reason is because
there are some generation tasks categorized into
these types. These results indicate that ROUGE-
L is a good proxy for accuracy for classification
tasks.
F Distribution of Tasks
As is described in §3, SUP-NATINST provides the
annotation for categorizing tasks along three differ-
ent dimensions: task type, language, and domain.
Fig. 10 shows the distribution of tasks among these
three dimensions. This meta-information can be
used to study model’s generalization ability in dif-
ferent senses. Despite the diversity of the data, we
acknowledge the skew toward certain tasks and lan-
guages, which we leave to be addressed by future
work.
G Evaluation Tasks
Table 5 lists the 12 task categories used for our eval-
uation and all the tasks included in each category
(introduced in §5.1). To provide a better sense of
what those tasks look like, we also select one rep-
resentative task from each category and list them
in Tables 6–17. Due to the large number of tasks
in our dataset, we cannot list all 1,616 tasks in this
paper. We refer the reader to our dataset.
H Performance Improvement per
Evaluation Task
To provide more detailed analysis ofTk-INSTRUCT
on each individual task, Fig. 11 presents the
per-task improvement of our Tk-I NSTRUCT (3B)
model over the best of two heuristic baselines on
the English evaluation tasks, and Fig. 12 presents
the per-task improvement of the mTk-INSTRUCT
model on the cross-lingual evaluation tasks. For
most of the evaluation tasks, we see a notable ex-
tent of generalization by Tk-INSTRUCT .
5099

0
10
20
30
40
50
60
70
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
TextualEntialment
 RougeL
 Accuracy
0
10
20
30
40
50
60
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
WordAnalogy
 RougeL
 Accuracy
0
10
20
30
40
50
60
70
80
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
AnswerabilityClassification
 RougeL
 Accuracy
0
10
20
30
40
50
60
70
80
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
DialogueActRecognition
 RougeL
 Accuracy
0
10
20
30
40
50
60
70
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
CoreferenceResolution
 RougeL
 Accuracy
0
10
20
30
40
50
60
70
80
Copying
Instance Input
Copying Demo.
Output
T0 InstructGPT Tk-Instruct
(Ours)
CauseEffectClassification
 RougeL
 Accuracy
Figure 9: Rouge-L v.s. Accuracy for task types that are usually regarded as classification tasks. The trends of these
two metrics are highly correlated with a Pearson coefficient of 0.970.
(a) Task Types
111111111111
22222223333444555555666777788999101010111212121414141515151920212225252727343637404146
748290
200
394
# of Tasks
1
10
100
1000
Spelling Error Detection
Spam Classification
Sentence Compression
Punctuation Error Detection
Poem Generation
Paper Review
Entity Generation
Entity Relation Classification
Discourse Connective Identification
Discourse Relation Classification
Preposition Prediction
Sentence Expansion
Overlap Extraction
Grammar Error Correction
Question Decomposition
Number Conversion
Irony Detection
Speaker Relation Classification
Style Transfer
Grammar Error Detection
Section Classification
Fact Verification
Stance Detection
Code to Text
Text Simplification
Dialogue State Tracking
Keyword Tagging
Answer Verification
Mathematics
Word Relation Classification
Sentence Ordering
Intent Identification
Coherence Classification
Ethics Classification
Explanation
Gender Classification
Dialogue Act Recognition
Stereotype Detection
Negotiation Strategy Detection
Data to Text
Word Analogy
Story Composition
Linguistic Probing
Speaker Identification
Question Rewriting
Pos Tagging
Word Semantics
Dialogue Generation
Text Quality Evaluation
Paraphrasing
Text to Code
Answerability Classification
Coreference Resolution
Summarization
Question Understanding
Language Identification
Sentence Perturbation
Title Generation
Sentence Composition
Text Completion
Fill in The Blank
Commonsense Classification
Named Entity Recognition
Textual Entailment
Wrong Candidate Generation
Information Extraction
Cause Effect Classification
Misc.
Toxic Language Detection
Text Matching
Text Categorization
Sentiment Analysis
Question Generation
Program Execution
Question Answering
Translation(b) Languages
1111111
2222222233334444556667889991112121213141516161617182121242626344043
1169
# of Tasks
1
10
100
1000
Finnish
Igbo
Sinhala
Xhosa
Assamese
Zhuang
Romanian
Somali
Tagalog
Burmese
Malay
Czech
Kannada
Estonian
Haitian
Turkish
Swedish
Kurdish
Swahili
Lao
Central Khmer
Filipino
Yoruba
Russian
Croatian
Bulgarian
Vietnamese
Indonesian
Korean
Thai
Catalan
Dutch
Hebrew
Galician
Portuguese
Oriya
Panjabi
Urdu
Malayalam
Marathi
Telugu
Arabic
Polish
Tamil
Gujarati
Italian
German
Bengali
Chinese
French
Hindi
Persian
Japanese
Spanish
English(c) Domains
11111111111111111
2222222
3333334444456666771012121213141516171924303232353739394244515359656976788095101122137143207211278
# of Tasks
1
10
100
1000
Image Caption
Art
Astronomy
Business Ethics
Econometrics
Electrical Engineering
Human Sexuality
International Law
Formal Fallacy
Management
Marketing
Medical Genetics
Prehistory
Accounting
Public Relations
Security: National Security
US Foreign Policy
Pop Culture
Formal logic
SQL
Literature
Psychology
Moral Scenarios
Philosophy
Conference
Chemistry
Geography
Economics
Jurisprudence
Animals
Game
Debatepedia
Logic
Physics
Nutrition
Professions
Personal Narratives
Linguistics
English Exams
Global Facts
World Religions
Statistics
Sports
Medicine
Stereotypes
Food
Countries
Movies
Scientific Research Papers
Computer Science
Healthcare
Biology
Knowledge Base
Anthropology
Justice
Books
School Science Textbooks
Government and Politics
Code
Public Places
Fiction
Web
Story
Law
Natural Science
Narrative
Reviews
Social Media
Miscellaneous
History
Mathematics
TED Talks
Dialogue
Sociology
Captions
Commonsense
Wikipedia
News
Figure 10: Distribution of SUP-NATINST tasks in terms of their (a) task types (b) languages (c) domains. y-axes are
in log scale.
5100