Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 2/8

Cleaned-text code points [15131, 28925). Bibliography omitted here; source.md includes it.

• N EGATIVE EXAMPLES are samples of inputs
and their incorrect/invalid outputs, along with
a short explanation for each.
The above schema is based on that of Mishra et al.
(2022b), though it is simplified. See Appendix C
for the comparison.
Task instances. Given the instructions for each
task, a model is expected to solve instances of that
task. We use a unified format to organize the in-
stances of all our tasks. More precisely, each in-
stance consists of a textual input and a list of ac-
ceptable textual outputs. We limit the number of
instances in each task to 6.5K to avoid an imbal-
ance of instances between tasks.
Benchmark collection. The benchmark was
collected through a large community effort on
GitHub.3 Tasks were collected and contributed by
NLP practitioners who were either responding to
our public invitation4 or students who were encour-
aged to contribute as part of their class project. 5
Contributors were encouraged to be creative and
source the tasks from several resources: (a) exist-
ing public NLP datasets, (b) available intermediate
annotations in crowdsourcing experiments (e.g.,
paraphrasing questions or rating their quality dur-
ing crowdsourcing a QA dataset), or (c) synthetic
tasks that can be communicated to an average hu-
man in a few sentences (e.g., basic algebraic opera-
tions like number comparison, finding the longest
palindrome substring, etc.). When using existing
datasets or crowdsourcing annotations, contribu-
tors were encouraged to adopt the instructions used
to create this dataset whenever available. This was
done to ensure that the instructions were sufficient
to define the tasks to average human readers. Tasks
along with instructions and other meta information
were contributed as JSON files via GitHub pull re-
quests, which were reviewed by automated checks
and peers. We had 88 contributors from diverse
locations and backgrounds contribute to our reposi-
tory.
Quality control. Controlling the quality of this
community-contributed data was done in several
phases: (1) Upon creating a GitHub pull request
of the proposed task, it immediately went through
an automatic test. This process verified that the
introduced file contained the expected fields and
adhered to our desired properties (e.g., no duplicate
3https://github.com/allenai/natural-instructions
4https://blog.allenai.org/9d3f24d5a9db
5CSE 576 “Topics in NLP” course, Arizona State Univ.
instances, the output labels are not heavily imbal-
anced, etc.) and (2) The proposed task was then
peer-reviewed by 1–2 other expert contributors to
ensure the clarity and sufficiency of instruction con-
tent. The review process was done iteratively until
the reviewers were content with the quality of the
proposed instruction. Specifically, reviewers were
asked to verify that the instruction is clear and suf-
ficient for an average language speaker to solve the
underlying task (evaluation instances) while being
grammatical, fluent, and concise. On average, the
review of each GitHub pull request took about 4–
6 iterations over the span of multiple days before
being merged. (3) Lastly, the added tasks were pre-
sented to crowdworkers in order to collect feedback
on the quality of the provided instructions, such as
typos, clarity, or other issues (details in §A). Sub-
sequently, one of the authors used this feedback to
improve the task definitions of the instances. This
feedback was done only for English tasks, as find-
ing high-quality crowdworkers in other languages
is nontrivial (Pavlick et al., 2014).
Diversity of tasks. Collecting tasks for SUP-
NATINST was carefully supervised to cover a wide
variety of natural language understanding tasks, do-
mains, and languages. To better understand this di-
versity, we comprehensively categorize tasks along
three different dimensions:
• T ASK TYPE defines the nature of the mapping
from instance inputs to outputs (e.g., question
answering, classification, etc.).
• L ANGUAGE indicates the language(s) of the in-
stances.
• D OMAIN indicates the domain(s) to which
the text of the tasks belong to (e.g., politics,
medicine, dialogue, etc.).
These different measures of categorization can be
used to study different senses of generalization. In
our empirical studies (§5), we study generalization
along the axis of task types. We refer the reader
to Fig. 10 in the appendix for the distribution of
tasks among different task types, languages, and
domains.
Statistics. Table 2 shows various statistics for the
benchmark. In total, the dataset includes 1616 tasks
and 5M instances. On average, each instruction is
paired with 2.8 positive and 2.4 negative examples.
The average definition length is 56.6 in words.
5088

statistic
# of tasks 1616
# of task types 76
# of languages 55
# of domains 33
# of non-English tasks 576
avg. definition length (words per task) 56.6
avg. # of positive examples (per task) 2.8
avg. # of negative examples (per task) 2.4
avg. # of instances (per task) 3106.0
Table 2: Statistics of SUP-NATINST.
4 T k-INSTRUCT : Learning to Follow
Instructions at Scale
Defining Generalization to Unseen Tasks. Each
task t is defined via its natural language instruction
It, and each task has a set of input/output instances
(Xt, Yt). A model M is expected to produce the
output y, given the input x and the task instruction
It: M (It, x) = y, for (x, y)∈ (Xt, Yt). In partic-
ular, we would like to evaluate model M on tasks
that are not observed (i.e., their instances were not
used for training M). The only source of signal
for learning the task at inference time is in-context
instructions It that contain a definition and demon-
stration examples of the task.
Tk-INSTRUCT . We introduce Tk-I NSTRUCT , a
model that is meta-trained on SUP-NATINST for
solving tasks given their in-context instructions.
Previous work has shown the effectiveness of such
meta-training in improving model’s ability to do in-
context learning with either prompts (Zhong et al.,
2021; Sanh et al., 2022) or demonstration examples
(Min et al., 2022a). Because of the large variety
of tasks in SUP-NATINST, we are able to do this
multi-task meta-training at a larger scale than be-
fore. We conduct our experiments and analysis
based on the T5 model (Raffel et al., 2020). Since
each instruction It consists of multiple elements as
described in our instruction schema (§3), we map
these elements to textual format and append them
before the input instance. Fig. 8 in the appendix
shows how we encode the full instructions. We
study different combinations of these instruction
elements in §7.2. By default, we will use our most
effective instruction elements (i.e., task definition
and two positive examples) unless otherwise speci-
fied. In the same manner, we train the multilingual
variant mTk-INSTRUCT based on the mT5 model
(Xue et al., 2021).
5 Benchmarking Cross-Task
Generalization with SUP-NATINST
Here we provide our recommended recipe for
benchmarking generalization via SUP-NATINST.
5.1 Evaluation Setup
An Evaluation Split of Unseen Tasks. We split
the large collection of tasks in SUP-NATINST into
two subsets: one for evaluation and the other for su-
pervision. For evaluation tasks, we fix a manually-
selected collection of 12 categories that represent
154 tasks. The large variety of tasks in SUP-
NATINST enables us to choose a diverse set of tasks
for evaluation – such as those at word, sentence,
and document levels, covering both classification
and generation formats. Appendix G lists our eval-
uation tasks with examples for representative tasks.
For an efficient evaluation, we sample a maximum
of 100 instances for each task, which results in
15,310 testing instances in total. The remaining
tasks are used for training models.6
Divided Tracks for English and X-lignual Tasks.
SUP-NATINST consists of tasks across multiple
languages, which enables evaluating the model’s
generalization to unseen tasks not only in English
but also in other languages. Therefore, we divide
our evaluation tasks into two tracks: one for En-
glish cross-task generalization(119 tasks) and the
other for cross-lingual cross-task generalization
(35 tasks). To the best of our knowledge, this is the
first study in cross-lingual cross-task generaliza-
tion (i.e., generalization to unseen tasks in different
languages). Fig. 11 and Fig. 12 in the appendix
contain the evaluation tasks for each track.
Evaluation Metrics. Due to the diversity of our
tasks and the open-ended generation nature of our
formulation,7 we adopt ROUGE-L (Lin, 2004) for
reporting aggregated performance results. This is a
soft string overlap metric that can be applied to a
wide range of text generation tasks. We show that
the ranking from this metric correlates well with
accuracy for classification tasks in Appendix E. We
also conduct a human evaluation in §6.2.
6To avoid data leakage, we exclude tasks from the training
set if they are sourced from the same dataset as any test task.
This results in 757 training tasks for the English track and
1271 training tasks for the cross-lingual track.
7Unlike Sanh et al. (2022) and Wei et al. (2022), who
evaluate their models on classification tasks via option ranking
(i.e., scoring the correct answer(s) higher than other candidate
answers), we evaluate our models in anopen-ended generation
setting with no task-specific assumptions. We believe this is a
more realistic measure of generalization to unseen tasks.
5089

5.2 Baselines and Existing Models
Here we discuss a variety of baselines and com-
petitive models for our target application. See Ap-
pendix D for implementation details.
Heuristic baselines. We first evaluate the follow-
ing heuristics to evaluate the possible shortcuts in
the data. Copying Demo Output copies the output
of a random demonstration example. Since we bal-
ance the labels for our test tasks, the performance of
this baseline will roughly equal a random guess or
a majority baseline for classification tasks. Copy-
ing Instance Input copies the given instance input.
This strategy performs well on tasks where the
target output largely overlaps with the input (e.g.,
question rewriting, grammar error correction).
Off-the-shelf pretrained language models. We
evaluate existing LMs that are not fine-tuned with
instruction-specific data. Specifically, we evalu-
ate the 11B-parameter T5 (Raffel et al., 2020) as
a direct counterpart of Tk-INSTRUCT . Due to the
infilling pretraining objective of the original T5
model, it cannot continue text well. Therefore,
we evaluate its “LM-adapted” version, which is
further trained with a language modeling objec-
tive (Lester et al., 2021). Additionally, we evaluate
GPT-3 (Brown et al., 2020), a 175B-parameter au-
toregressive LM that has shown remarkable ability
in following demonstrations provided in its prompt.
Instruction-tuned models. In addition to our Tk-
INSTRUCT (§4), we evaluate existing models that
are fine-tuned to follow language instructions. In
particular, we evaluate InstructGPT (Ouyang et al.,
2022) which uses reinforcement learning to incor-
porate human preferences into a GPT-3 pretrained
model, and T0 (Sanh et al., 2022) which finetunes
T5 on a collection of task prompts in PROMPT -
SOURCE (Bach et al., 2022).
Upper bound estimates. We estimate an upper
bound on models’ generalization to unseen tasks by
fine-tuning an oracle model on the tasks’ labeled
instances. Since this model observes the hidden
instances of the evaluation tasks, it is, by definition,
an estimated upper bound to our generalization-
based models. Specifically, we fine-tune a T5-11B
model on the 119 English evaluation tasks, and
a mT5-13B model on the 35 non-English tasks,
with 1K random training instances per task, without
overlap with the evaluation instances.
Methods↓ / Evaluation→ En X-lingual
Heuristic
Baselines
Copying Instance Input 14.2 5.4
Copying Demo Output 28.5 50.3
Pretrained LMs T5-LM (11B) 30.2 –
GPT3 (175B) 45.0 51.3
Instruction-tuned
Models
T0 (11B) 32.3 –
InstructGPT (175B) 52.1 52.8
Tk-INSTRUCT (ours, 11B) 62.0 –
mTk-I NSTRUCT (ours, 13B) 57.1 66.1
Upper-bound (est.) Supervised Training 74.3 94.0
Table 3: The overall performance of different methods
on unseen tasks in the test set of SUP-NATINST (§6.1).
We report ROUGE-L here as our aggregated metric.
Models that leverage instructions show stronger gen-
eralization to unseen tasks. In particular, our model
that is fine-tuned on a diverse set of tasks outperforms
InstructGPT and T0 by a large margin.
15.2
40.0
64.6 64.1
76.9
14.2
32.3
52.1 54.3
62.0
0
10
20
30
40
50
60
70
80
Copying
Instance Input
T0 (11B) InstructGPT Tk-Instruct
(3B)
Tk-Instruct
(11B)
 Human ROUGE-L
Figure 3: Human evaluation vs. ROUGE-L for several
methods (§6.2). The trends of these two metrics are
highly correlated with a Pearson coefficient of 0.998.
6 Experimental Results
6.1 Overall Results
Table 3 summarizes our overall benchmarking re-
sults. We use the same input encoding that contains
the most effective instructional elements (task defi-
nition and two positive examples without the nega-
tive examples and explanations) for all the methods.
To better understand models’ generalization to dif-
ferent tasks, we also break down the performance
according to the task categories in Fig. 4. We refer
the reader to Appendix H for more detailed analysis
on each individual task.
Instruction-tuning enables stronger generaliza-
tion to unseen tasks. Generally instruction-tuned
models perform better compared to their untuned
LM counterparts (Tk-I NSTRUCT vs. T5-LM, In-
structGPT vs. GPT-3) and heuristic baselines. This
indicates models do learn to follow instructions by
finetuning on instruction data, and this can gen-
eralize to new instructions for unseen tasks. T0
is an exception, which is only slightly better than
5090