Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks — part 3/8

Cleaned-text code points [28925, 41818). Bibliography omitted here; source.md includes it.

0
25
50
75
100ROUGE-L
43
1
33
49
65
76
Textual Entailment
50
10
46
62
7579
Cause Effect Classification
25
8
27
50
66
86
Coreference Resolution
41
2
32
63
73
86
Dialogue Act Recognition
53
0
50 57
70
80
Answerability Classification
15 7 14
50 55
100
Word Analogy
0
25
50
75
100ROUGE-L
15
24 19
44
60
84
Overlap Extraction
11 16
45
57
6775
Keyword Tagging
13
63
34
65 7073
Question Rewriting
8 9
19
38 4046
Title Generation
20
38 36
47 5358
Data to Text
7
87
74
86 8888
Grammar Error Correction
Supervised Training Copying Demo. Output Copying Instance Input T0 (11B) InstructGPT (175B) Tk-Instruct (Ours, 11B)
Figure 4: Performance per evaluation task type.Tk-I NSTRUCT consistently performs better than other generalization-
based methods on all task types, while there is still a sizable gap compared to supervised training.
T5-LM. We suspect this is because the style of
prompting in T0’s training data is very different
from our style of instructions.
Our Tk-I NSTRUCT outperforms InstructGPT.
Our Tk-INSTRUCT and mTk-INSTRUCT models,
which are trained with a variety of tasks, gener-
alize best to unseen tasks for both English and
non-English tasks in all evaluation task categories.
InstructGPT also shows a great extent of general-
ization to our evaluation tasks. However, we want
to note it is not clear if InstructGPT’s training data
overlaps with our evaluation tasks since their data
is unavailable.
There is a sizable gap for improvement. De-
spite the impressive performance of current models,
there is a sizable gap between the generalization of
instruction-based models and the supervised train-
ing approach, leaving more room for improvement.
6.2 Human Evaluation
For language generation tasks, automatic metrics
are only an approximation of human judgments;
we conduct a human evaluation to confirm the find-
ings so far. Specifically, we ask crowdworkers to
indicate if they prefer the predicted answer by the
model or the ground truth outputs for each instance
with ties being allowed (see Appendix B for de-
tails). The resulting human evaluation metric indi-
cates how often model predictions were rated as at
least as good as our ground truth labels. The theo-
retical upper bound of this metric is 100% when the
model is rated at least as good as the ground truth
for all the instances. The results of human evalua-
tion (shown in Fig. 3) align quite well with our au-
tomatic metrics and confirm the human-perceived
quality of our models.
7 Further Analysis
We conduct further analysis to understand the im-
portant factors for models to generalize across tasks.
Due to the computational cost, this analysis is done
on the English track and using the T5-3B check-
point, except for the experiments on model sizes.
7.1 Scaling Trends of Generalization
We study Tk-INSTRUCT ’s generalization perfor-
mance with respect to three scaling factors: the
number of training tasks, the number of instances
per task, and the model sizes. Fig. 5 presents the
performance change by scaling each of them.
More observed tasks improve the generalization.
We fine-tuneTk-INSTRUCT with different numbers
of tasks that are randomly sampled from the whole
training set (Fig. 5a). The model generalization
performance grows log-linearly 8 as we increase
the set of tasks used for training. Previous work
(Mishra et al., 2022b; Sanh et al., 2022; Wei et al.,
2022) has made similar observations on a much
smaller scale, while we show that this trend holds
even with 757 diverse training tasks.
A large number of training instances do not
help generalization. We then vary the number
of instances per task that are used for finetuning
(Fig. 5b). While the conventional wisdom in super-
vised learning is that more training instances usu-
ally helps (Banko and Brill, 2001; Sun et al., 2017;
Hestness et al., 2017), in our setup, the model’s
performance saturates when only 64 instances per
task are used for training. A large number of train-
ing instances would instead lead to longer training
time and risk overfitting to the training tasks.
8A linear function of an exponential increase of parameters,
i.e., growth at a constant multiplicative rate.
5091

43.7
45.6
48.4
51.252.9
54.3
30
35
40
45
50
55
60
6 60 600
ROUGE-L
NumberofTrainingTasks
(a)
48.5
50.7
54.753.753.253.7
30
35
40
45
50
55
60
6 60 600
ROUGE-L
NumberofInstancesperTrainingTask (b)
40.1
42.1
48.0
54.3
62.0
30
35
40
45
50
55
60
65
40 400 4000
ROUGE-L
NumberofModelParameters(Millions) (c)
Figure 5: Scaling trends of models performance (§7.1) as a function of (a) the number of training tasks; (b) the
number of instances per training task; (c) model sizes. x-axes are in log scale. The linear growth of model
performance with exponential increase in observed tasks and model size is a promising trend. Evidently, the
performance gain from more instances is limited.
Testing Encoding →
Training Encoding ↓
Task ID Def Pos (1) Def
+ Pos (1) Pos (2)  Def
+ Pos (2) 
Def 
+ Pos (2) 
+ Neg (2) 
Def
+ Pos (2) 
+ Neg (2) 
+ Expl
Pos (4) Def 
+ Pos (4) Average
Task ID 21.2 33.3 16.8 30.9 23.0 33.7 33.9 31.6 26.0 36.4 33.9
Def 17.3 45.0 31.1 43.8 36.4 46.4 44.2 44.3 38.0 46.0 39.9
Pos (1) 10.9 22.1 43.9 47.8 46.6 49.2 46.2 43.4 46.6 49.5 43.1
Def + Pos (1) 11.1 42.2 43.8 52.4 47.4 53.3 53.1 51.8 47.8 53.7 44.5
Pos (2)  12.7 22.4 47.1 50.2 49.3 52.3 50.6 46.7 49.8 52.4 45.0
Def + Pos (2) 12.4 42.1 44.5 52.4 49.0 54.3 53.5 52.7 50.3 54.8 46.4
Def + Pos (2) + Neg (2)  14.0 42.3 43.6 51.8 48.6 53.5 54.3 50.2 49.6 53.8 45.9
Def + Pos (2) + Neg (2) + Expl 15.4 42.0 43.8 50.7 47.6 51.9 52.5 52.6 48.6 52.2 44.3
Pos (4) 11.0 23.9 45.6 49.8 49.0 51.7 49.5 47.5 49.8 51.3 44.5
Definition  + Pos (4) 11.0 42.4 44.3 51.9 48.7 53.7 53.4 50.6 50.5 53.5 46.0
Table 4: Performance (ROUGE-L) of models trained and evaluated with various encodings. Diagonal numbers
(underlined) represent performances of models trained and evaluated with the same instruction encoding. Each
encoding is a combination of the elements in the instructions (Fig. 1). Task ID is a short string composed of
dataset name and task category; Def represents the task definition; Pos (k) represents k positive examples; Neg
(k) represents k negative examples; Expl represents explanation. These results (a) show the gains from various
instructional elements, and (b) indicate surprising reliability of the models to various input encoding. A model
trained with definition and positive examples (i.e., the last row) remains robust for different encodings.
Tuning larger models with instructions consis-
tently lead to gains. We study the effect of model
scaling by initializing Tk-INSTRUCT from differ-
ent sizes of pretrained T5 checkpoints, including
the small, base, large, xl and xxl sizes (Fig. 5c). We
found that increasing the model sizes consistently
bring significant improvement (log-linearly with
parameter size). This finding contradicts the claim
in Xu et al. (2022) that “model size has little im-
pact on performance with an extremely large num-
ber of tasks.” Combining Fig. 5(a) and Fig. 5(c),
one can create a correspondence between model
size and task size. For example, a T5-large model
trained with 757 tasks can achieve comparable per-
formance (48.0 ROUGE-L) to the T5-3B model
trained with 128 tasks (48.4 ROUGE-L), indicating
that increasing the diversity of training tasks is an
alternative to scaling model sizes.
7.2 Instructing with Different Elements
We evaluate the performance of Tk-INSTRUCT un-
der different instructional elements.
Benefit of different instructional elements. As
shown in Fig. 1, SUP-NATINST provides multiple
elements for instructing a task. We train multiple
models with different combinations of these ele-
ments. The diagonal cells of Table 4 show the
performance of our models when trained and eval-
uated on a particular instruction encoding. Based
on the diagonal numbers, including the task defi-
nition consistently helps the model to generalize
better. Moreover, combining the task definition
with positive demonstration examples yields fur-
ther improvement. However, adding more demon-
stration examples is negligible. Negative examples
help a little bit; explanations decrease performance,
which is consistent with the observations of Mishra
et al. (2022b) and Lampinen et al. (2022) when
5092

the model is not large enough. Future work can
explore whether more powerful models can benefit
from these elements.
Generalization to different input encodings. We
further investigate whether a model trained on a par-
ticular encoding can generalize to other encodings.
This can be read from the non-diagonal cells of
Table 4. The negative result here is that definition-
only models cannot generalize to example-only
test encodings; and similarly, example-only models
cannot generalize to definition-only test encodings.
However, models trained on encodings that con-
tain both definition and examples are surprisingly
robust across different encoding variations.
8 Conclusion
We construct a large-scale benchmark consisting
of a diverse set of NLP tasks and their instructions.
This benchmark can serve as a rich playground for
training or evaluation of models that can generalize
to unseen tasks by following instructions. Further-
more, we train Tk-I NSTRUCT using this data, and
demonstrate its capability to perform unseen tasks
to a surprising extent. We provide extensive anal-
ysis to understand the important factors for such
generalization. We hope our data and model will fa-
cilitate future work towards more general-purpose
models.
9 Limitations
While the presented data offers a notable variety
(e.g., diverse task types), its underlying distribu-
tions suffer from skews which should be addressed
in future work (see Appendix F). On language di-
versity, the proposed benchmark is biased toward
English. On output diversity, the collected tasks
are generally still skewed to short responses, which
might reflect the distribution of the available tasks
in the field. This under-representation of the long-
tail of tasks poses a challenge for building general-
purpose models in the future. We hope future work
addresses such distributional imbalances. More-
over, we see natural extensions of the instruction-
following setup here in the context of other modali-
ties such as vision or speech.
Automatic evaluation of models’ performance
is another challenge, considering the diverse set of
tasks in our benchmark, and many of them being
open-ended generation tasks. We use ROUGE-L as
an aggregated metric in this paper and find it as a
good proxy for the overall performance of the mod-
els, aligning well with human evaluation. However,
there are specific tasks for which ROUGE-L might
not serve as an effective proxy of quality (such
as rewriting tasks or error correction tasks where
copying the input can result in a high ROUGE-L
score). We hope these issues will be addressed
with the development of more powerful evaluation
metrics for text generation.
In terms of computing power, we have experi-
mented with models that were accessible to us and
have made the resulting models publicly available.
We also acknowledge that there are larger models
that we were not able to train due to the limitations
of our computational budget.
Acknowledgments
We thank the anonymous reviewers, our colleagues
from AI2 and UWNLP, especially Matthew Peters
for his encouraging conversations that motivated
this project. We also thank the student contributors
of Arizona State University’s CSE 576 “Topics
in NLP” course and all other contributors to our
data repository. All experiments were run on AI2’s
Beaker GPU clusters and Google’s research TPUs.
This work was supported in part by ONR MURI
N00014-18-1-2670, ONR N00014-18-1-2826, and
DARPA MCS N66001-19-2-4031 grants.
References
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao,
Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon-
glei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni,
et al. 2022. ExT5: Towards Extreme Multi-Task
Scaling for Transfer Learning. In International Con-
ference on Learning Representations (ICLR).
Stephen H Bach, Victor Sanh, Zheng-Xin Yong, Al-
bert Webson, Colin Raffel, Nihal V Nayak, Abheesht
Sharma, Taewoon Kim, M Saiful Bari, Thibault
Fevry, et al. 2022. PromptSource: An Integrated
Development Environment and Repository for Nat-
ural Language Prompts. In Annual Meeting of the
Association for Computational Linguistics (ACL) -
System Demonstrations.
Michele Banko and Eric Brill. 2001. Scaling to Very
Very Large Corpora for Natural Language Disam-
biguation. In Annual Meeting of the Association for
Computational Linguistics (ACL).
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebas-
tian Riedel, and Pontus Stenetorp. 2020. Beat the ai:
Investigating adversarial human annotation for read-
ing comprehension. Transactions of the Association
for Computational Linguistics (TACL), 8:662–678.
5093