PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization — part 3/6

Cleaned-text code points [27837, 41509). Bibliography omitted here; source.md includes it.
We ﬁne-tune all of the four models on different
subsets with 10 and 100 examples, and the results
are shown in Figure 5. (hyperparameter settings
in Appx. §D.1) Since R-1, -2, and -L show the
same trend, we only present the average of the
three metrics in the ﬁgure for brevity (full ROUGE
scores can be found in Appx. Table 8) To show the
generality, all the results of few-shot experiments
are the average over 5 runs on different subsets
(shared by all the models).
The result of each run is obtained by the ‘best’
model chosen based on the ROUGE scores on a
randomly sampled few-shot validation set with the
same number of examples as the training set, which
is similar with Zhang et al. (2020). Note that their
reported best models have been selected based on
the whole validation set which may give PEGA -
SUS some advantage. Nevertheless, we argue that
sampling few-shot validation sets as we do here
is closer to real few-shot scenarios (Bragg et al.,
2021).
Our model outperforms all baselines on all of
the datasets with 10 and 100 examples demonstrat-
ing the beneﬁts of our pretraining strategy and in-
put structure. Comparing the performances of our
model with the different number of training data
fed in, our model converges faster than other mod-
els with as few as 10 data examples.
4.5 Fully Supervised Evaluation
To show the advantage of our pretrained model
when there is abundant training data, we also train
the model with the full training set (hyperparame-
ter settings can be found in Appx. §D.2). Table 3
shows the performance comparison with previous
state-of-the-art10, along with the results of previous
SOTA. We observe thatPRIMERA achieves state-
of-the-art results on Multi-News, WCEP, and arXiv,
while slightly underperforming the prior work on
Multi-XScience (R-1). One possible explanation
is that in Multi-XScience clusters have less over-
lapping information than in the corpus on which
PRIMERA was pretrained. In particular, the source
documents in this dataset are the abstracts of all the
publications cited in the related work paragraphs,
which might be less similar to each other and the
target related work(i.e., their summary) . PRIMERA
9We re-evaluate the generated summaries of the models
from Lu et al. (2020) for Multi-XScience, as we use a different
version of ROUGE.
10Due to the lack of computational resources, we do not
train the model on Wikisum.
Datasets Previous SOTA P RIMERA
R-1 R-2 R-L R-1 R-2 R-L
Multi-News 49.2 19.6 24.5 49.9 21.1 25.9
Multi-XScience 33.9 6.8 18.2 31.9 7.4 18.0
WCEP 35.4 15.1 25.6 46.1 25.2 37.9
arXiv 46.6 19.6 41.8 47.6 20.8 42.6
Table 3: Fully supervised results. Previous SOTA are
from Pasunuru et al. (2021) for Multi-News, Lu et al.
(2020) for Multi-XScience11, Hokamp et al. (2020) for
WCEP, and Beltagy et al. (2020) for arXiv.
0 10 100
10.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5AVG ROUGE
(a) Ablation on Struct. & Pre-train
LED
LED+struct.
Ours(LED+struct.+pre-train)
0 10 100
21
22
23
24
25
26
27
(b) Entity Pyramid V.S. PEGASUS
Entity Pyramid
PEGASUS masking strategy
Number of Training Examples
Figure 6: Ablation study with the few-shot setting on
the Multi-News dataset regarding to (a) input Structure
(<doc-sep> tokens between documents and global
attention on them) and pretraining, (b) pretraining us-
ing PEGASUS vs our approach.
outperforms the LED model (State-of-the-art) on
the arXiv dataset while using a sequence length 4x
shorter (4K in PRIMERA v.s. 16K in LED), further
showing that the pretraining and input structure of
our model not only works for multi-document sum-
marization, but can be also effective for summariz-
ing single documents having multiple sections.
5 Ablation Study
We conduct ablation studies on the Multi-News
dataset in few-shot setting, to validate the contribu-
tion of each component in our pretrained models.
Input structure: In Figure 6 (a) we observe the
effectiveness of both pretraining and the input struc-
ture (<doc-sep> tokens between documents and
global attention on them).
Sentence masking strategy: To isolate the effect
of our proposed pretraining approach, we compare
with a model with exactly the same architecture
when pretrained on the same amount of data but
using the PEGASUS (Zhang et al., 2020) masking
strategy instead of ours. In other words, we keep
all the other settings the same (e.g., data, length
limit of input and output, pretraining dataset, in-
put structure, as well as the separators) and only
modify the pretraining masking strategy. We run
the same experiments under zero-/few-shot scenar-
5251

ios on the Multi-News dataset as in §4.2, and the
results are shown in Figure 6 (b). The model pre-
trained with our Entity Pyramid strategy shows a
clear improvement under few-shot scenarios.
6 Human Evaluation
We also conduct human evaluations to validate
the effectiveness of PRIMERA on DUC2007 and
TAC2008 (Dang and Owczarzak, 2008) datasets
in the few-shot setting (10/10/20 examples for
train/valid/test). Both datasets consist of clusters of
news articles, and DUC2007 contains longer inputs
(25 v.s. 10 documents/cluster) and summaries (250
v.s. 100 words). Since the goal of our method is to
enable the model to better aggregate information
across documents, we evaluate the content quality
of the generated summaries following the original
Pyramid human evaluation framework (Nenkova
and Passonneau, 2004). In addition, we also evalu-
ate the ﬂuency of generated summaries following
the DUC guidelines.12
Settings Three annotators13 are hired to do both
Pyramid Evaluation and Fluency evaluation, they
harmonize the standards on one of the examples.
Speciﬁcally, for each data example, we provide
three anonymized system generated summaries,
along with a list of SCUs. The annotators are asked
to ﬁnd all the covered SCUs for each summary,
and score the ﬂuency in terms of Grammaticality,
Referential clarity and Structure & Coherence, ac-
cording to DUC human evaluation guidelines, with
a scale 1-5 (worst to best). They are also suggested
to make comparison between three generated sum-
maries into consideration when scoring the ﬂuency.
To control for the ordering effect of the given sum-
maries, we re-order the three summaries for each
data example, and ensure the chance of their ap-
pearance in different order is the same (e.g. BART
appears as summary A for 7 times, B for 7 times
and C for 6 times for both datasets). The instruction
for human annotation can be found in Figure 7 and
Figure 8 in the appendix. Annotators were aware
that annotations will be used solely for computing
aggregate human evaluation metrics and reporting
in the scientiﬁc paper.
Compared Models We compare our model with
LED and PEGASUS in human evaluations. Be-
12https://www-nlpir.nist.gov/projects/
duc/duc2007/quality-questions.txt
13We recruited expert annotators with payment above aver-
age of the participants’ demographics.
Model DUC2007(20) TAC2008(20)
Sr R P F Sr R P F
PEGASUS 6.0 2.5 2.4 2.4 8.7 9.1 9.4 9.1
LED 9.6 3.9 4.0 3.8 6.9 7.1 10.8 8.4
PRIMERA 12.5 5.1 5.0 5.0 8.5 8.9 10.0 9.3
Table 4: Pyramid Evaluation results: Raw scores Sr,
(R)ecall, (P)recision and (F)-1 score. For readability,
Recall, Precision and F-1 scores are multiplied by 100.
Model DUC2007(20) TAC2008(20)
Gram. Ref. Str.&Coh. Gram. Ref. Str.&Coh.
PEGASUS 4.45 4.35 1.95 4.40 4.20 3.20
LED 4.35 4.50 3.20 3.10 3.80 2.55
PRIMERA 4.70 4.65 3.70 4.40 4.45 4.10
Table 5: The results of Fluency Evaluation on two
datasets, in terms of the Grammaticality , Referential
clarity and Structure & Coherence.
cause PEGASUS is a task-speciﬁc model for ab-
stractive summarization, and LED has the same
architecture and length limits as our model with the
parameters inherited from BART, which is more
comparable with our model than vanilla BART.
Pyramid Evaluation Both TAC and DUC
datasets include SCU (Summary Content Unit) an-
notations and weights identiﬁed by experienced
annotators. We then ask 3 annotators to make a
binary decision whether each SCU is covered in a
candidate summary. Following Nenkova and Pas-
sonneau (2004), the raw score of each summary is
then computed by the sum of weights of the cov-
ered SCUs, i.e. Sr = ∑
SCU wiI(SCU i), where
I(SCU i) is an indicator function on whetherSCU i
is covered by the current summary, and wi is the
weight of SCU i. In the original pyramid evalua-
tion, the ﬁnal score is computed by the ratio of Sr
to the maximum possible weights with the same
number of SCUs as in the generated summaries.
However, the total number of SCUs of generated
summaries is not available in the simpliﬁed anno-
tations in our design. To take consideration of the
length of generated summaries and make a fair com-
parison, instead, we compute Recall, Precision and
F-1 score regarding lengths of both gold references
and system generated summaries as
R= Sr
len(gold) ; P= Sr
len(sys) ; F1= 2· R· P
(R + P )
Fluency Evaluation Fluency results can be
found in Table 5, and PRIMERA has the best perfor-
mance on both datasets in terms of all aspects. Only
5252

for Grammaticality PRIMERA’s top performance
is matched by PEGASUS.
7 Related Work
Neural Multi-Document Summarization
These models can be categorized into two classes,
graph-based models (Yasunaga et al., 2017; Liao
et al., 2018; Li et al., 2020; Pasunuru et al., 2021)
and hierarchical models (Liu and Lapata, 2019a;
Fabbri et al., 2019; Jin et al., 2020). Graph-based
models often require auxiliary information (e.g.,
AMR, discourse structure) to build an input graph,
making them reliant on auxiliary models and less
general. Hierarchical models are another class
of models for multi-document summarization,
examples of which include multi-head pooling
and inter-paragraph attention (Liu and Lapata,
2019a), MMR-based attention (Fabbri et al.,
2019; Mao et al., 2020), and attention across
representations of different granularity (words,
sentences, and documents) (Jin et al., 2020). Prior
work has also shown the advantages of customized
optimization in multi-document summarization
(e.g., RL; Su et al., 2021). Such models are often
dataset-speciﬁc and difﬁcult to develop and adapt
to other datasets or tasks.
Pretrained Models for Summarization Pre-
trained language models have been successfully
applied to summarization, e.g., BERTSUM (Liu
and Lapata, 2019b), BART (Lewis et al., 2020),
T5 (Raffel et al., 2020). Instead of regular language
modeling objectives, PEGASUS (Zhang et al., 2020)
introduced a pretraining objective with a focus on
summarization, using Gap Sentence Generation,
where the model is tasked to generate summary-
worthy sentences, and Zou et al. (2020) proposed
different pretraining objectives to reinstate the orig-
inal document, speciﬁcally for summarization task
as well. Contemporaneous work by Rothe et al.
(2021) argued that task-speciﬁc pretraining does
not always help for summarization, however, their
experiments are limited to single-document sum-
marization datasets. Pretraining on the titles of
HTMLs has been recently shown to be useful for
few-shot short-length single-document summariza-
tion as well (Aghajanyan et al., 2021). Goodwin
et al. (2020) evaluate three state-of-the-art models
(BART,PEGASUS , T5) on several multi-document
summarization datasets with low-resource settings,
showing that abstractive multi-document summa-
rization remains challenging. Efﬁcient pretrained
transformers (e.g., Longformer (Beltagy et al.,
2020) and BigBird (Zaheer et al., 2020) that can
process long sequences have been also proven suc-
cessful in summarization, typically by the ability to
process long inputs, connecting information across
the entire sequence. CDLM (Caciularu et al., 2021)
is a follow-up work for pretraining the Longformer
model in a cross-document setting using global at-
tention on masked tokens during pretraining. How-
ever, this model only addresses encoder-speciﬁc
tasks and it is not suitable for generation. In this
work, we show how efﬁcient transformers can be
pretrained using a task-inspired pretraining objec-
tive for multi-document summarization. Our pro-
posed method is also related to the PMI-based to-
ken masking Levine et al. (2020) which improves
over random token masking outside summariza-
tion.
8 Conclusion and Future Work
In this paper, we present PRIMERA a pre-trained
model for multi-document summarization. Unlike
prior work, PRIMERA minimizes dataset-speciﬁc
modeling by using a Longformer model pretrained
with a novel entity-based sentence masking ob-
jective. The pretraining objective is designed to
help the model connect and aggregate informa-
tion across input documents. PRIMERA outper-
forms prior state-of-the-art pre-trained and dataset-
speciﬁc models on 6 summarization datasets from
3 different domains, on zero, few-shot, and full
ﬁne-tuning setting. PRIMERA ’s top performance is
also revealed by human evaluation.
In zero-shot setting, we can only control the
output length of generated summaries at inference
time by specifying a length limit during decoding.
Exploring a controllable generator in which the de-
sired length can be injected as part of the input is a
natural future direction. Besides the summarization
task, we would like to explore using PRIMERA for
other generation tasks with multiple documents as
input, like multi-hop question answering.
Ethics Concern
While there is limited risk associated with our work,
similar to existing state-of-the-art generation mod-
els, there is no guarantee that our model will always
generate factual content. Therefore, caution must
be exercised when the model is deployed in prac-
tical settings. Factuality is an open problem in
existing generation models.
5253