PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization — part 2/6

Cleaned-text code points [14370, 27837). Bibliography omitted here; source.md includes it.

0.200Doc 1 Sent 2
Entity 1
Entity 2 Entity 3
Entity 4 Entity 5
Entity 6 Entity 7 Entity 8 frequency: 1
frequency: 2
frequency: 3
frequency: 4Doc 1: {Entity 1, Entity 2, Entity 6}
Doc 2: {Entity 1, Entity 2, Entity 3,  
     Entity 5, Entity 7}
Doc 3: {Entity 1, Entity 2, Entity 3,   
     Entity 4, Entity 5}
Doc 4: {Entity 1, Entity 3, Entity 4,  
     Entity 8}
Entity 1
Entity 4
Doc 1 Sent 1
Doc 2 Sent 10
...
Sentences with
Entity 1
Cluster
ROUGE
0.107
0.142
...
Extract Entities Entity Pyramid Estimation Sentence Selection
Entity 2
Entity 3
Entity 5
Doc 4 Sent 2 0.110
Doc 2 Sent 5
Sentences with
Entity 2
Cluster
ROUGE
0.130
Doc 3 Sent 3 0.015
Sentences with
Entity 3
Cluster
ROUGE
...
...
Sentences
to mask out
Doc 1: Sent 2,
    Sent 6, ...
Doc 2: Sent 10, ...
Doc 3: ...
Doc 4: /
Figure 4: The Entity Pyramid Strategy to select salient sentences for masking. Pyramid entity is based on the
frequency of entities in the documents. The most representative sentence are chosen based on Cluster ROUGE for
each entity with frequency > 1, e.g. Sentence 10 in Document 2 for Entity 1.
Algorithm 1 Entity Pyramid Sentence Selection
Input: Document cluster
Input: List of entities w/ frequency > 1. N length of the list
Input: m number of sentences to select
Output: List of sentences to mask
1: E← sort entities by frequency, descending
2: selected = []
3: for i← 1 to|E| do
4: SentCand← all sentences in the cluster containing
E[i]
5: cur_sent = arg max s∈SentCand Score(s)
6: selected.append(cur_sent)
7: if|selected| == m then
8: Break
9: end if
10: end for
11: Return selected
framework, we use the entity frequency in the clus-
ter as a proxy for saliency. Concretely, as shown in
Fig. 4, we have the following three steps to select
salient sentences in our masking strategy:
1. Entity Extraction. We extract named entities
using SpaCy (Honnibal et al., 2020).3
2. Entity Pyramid Estimation. We then build an
Entity Pyramid for estimating the salience of en-
tities based on their document frequency, i.e. the
number of documents each entity appears in.
3. Sentence Selection. Similar to the Pyramid eval-
uation framework, we identify salient sentences
with respect to the cluster of related documents. Al-
gorithm 1 shows the sentence selection procedure.
As we aim to select the entities better representing
the whole cluster instead of a single document, we
first remove all entities from the Pyramid that ap-
pear only in one document. Next, we iteratively se-
lect entities from top of the pyramid to bottom (i.e.,
3Note that entity information is only used at pretraining
time. This is unlike some prior work (e.g. Pasunuru et al.
(2021)) that utilize additional information (like named entities,
coref, discourse, or AMR) at fine-tuning and inference time.
highest to lowest frequency), and then select sen-
tences in the document that include the entity as the
initial candidate set. Finally, within this candidate
set, we find the most representative sentences to the
cluster by measuring the content overlap of the sen-
tence w.r.t documents other than the one it appears
in. This final step supports the goal of our pre-
training objective, namely to reconstruct sentences
that can be recovered using information from other
documents in the cluster, which encourages the
model to better connect and aggregate information
across multiple documents. Following Zhang et al.
(2020) we use ROUGE scores (Lin, 2004) as a
proxy for content overlap. For each sentence si,
we specifically define a Cluster ROUGE score as
Score(si) = ∑
{docj∈C,si⁄∈ docj} ROUGE (si, docj)
Where C is the cluster of related documents.
Note that different from the importance heuristic
defined in PEGASUS (Zhang et al., 2020), Entity
Pyramid strategy favors sentences that are repre-
sentative of more documents in the cluster than
the exact matching between fewer documents (See
Figure 3 for a qualitative example.) . The benefit
of our strategy is shown in an ablation study (§5).
3 Experiment Goals
We aim to answer the following questions:
• Q1: How does PRIMERA perform, compared
with existing pre-trained generation models in zero-
and few-shot settings? See §4.2.
• Q2: How does PRIMERA perform, compared
with current state-of-the-art models, in the fully
supervised setting? See §4.5.
• Q3: How much is the contribution of each compo-
nent in PRIMERA , i.e. input structure, pretraining,
and masking strategy? See §5.
• Q4: What is the effect of our entity pyramid
5248

Dataset #Examples #Doc/C Lensrc Lensumm
Newshead (2020) 360K 3.5 1734 -
Multi-News (2019) 56K 2.8 1793 217
Multi-Xscience (2020) 40K 4.4 700 105
Wikisum* (2018) 1.5M 40 2238 113
WCEP-10 (2020) 10K 9.1 3866 28
DUC2004 (2005) 50 10 5882 115
arXiv (2018) 214K 5.5 6021 272
Table 1: The statistics of all the datasets we explore in
this paper. *We use subsets of Wikisum (10/100, 3200)
for few-shot training and testing only.
strategy, compared with the strategy used in PEGA-
SUS? See §5.
• Q5: Is PRIMERA able to capture salient informa-
tion and generate fluent summaries? See §6.
With these goals, we explore the effectiveness of
PRIMERA quantitatively on multi-document sum-
marization benchmarks, verify the improvements
by comparing PRIMERA with multiple existing pre-
trained models and SOTA models, and further vali-
date the contribution of each component with care-
fully controlled ablations. An additional human
evaluation is conducted to show PRIMERA is able
to capture salient information and generate more
fluent summaries.
4 Experiments
4.1 Experimental Setup
Implementation Details We use the
Longformer-Encoder-Decoder (LED) (Belt-
agy et al., 2020) large as our model initialization,
The length limits of input and output are 4096 and
1024, respectively, with sliding window size as
w = 512 for local attention in the input. (More
implementation details of pretraining process can
be found in Appx §A)
Pretraining corpus For pretraining, our goal is
to use a large resource where each instance is a
set of related documents without any ground-truth
summaries. The Newshead dataset (Gu et al., 2020)
(row 1, Table 1) is an ideal choice; it is a relatively
large dataset, where every news event is associated
with multiple news articles.
Evaluation Datasets We evaluate our approach
on wide variety of multi-document summarization
datasets plus one single document dataset from
various domains (News, Wikipedia, and Scientific
literature). See Table 1 for dataset statistics and
Appx. §B for details of each dataset.
Evaluation metrics Following previous
works (Zhang et al., 2020), we use ROUGE
scores (R-1, -2, and -L), which are the standard
evaluation metrics, to evaluate the downstream
task of multi-document summarization.4 For better
readability, we use A VG ROUGE scores (R-1, -2,
and -L) for evaluation in the few-shot setting.
4.2 Zero- and Few-shot Evaluation
Many existing works in adapting pretrained models
for summarization require large amounts of fine-
tuning data, which is often impractical for new
domains. In contrast, since our pretraining strategy
is mainly designed for multi-document summariza-
tion, we expect that our approach can quickly adapt
to new datasets without the need for significant
fine-tuning data. To test this hypothesis, we first
provide evaluation results in zero and few-shot set-
tings where the model is provided with no, or only
a few (10 and 100) training examples. Obtaining
such a small number of examples should be viable
in practice for new datasets.
Comparison To better show the utility of our pre-
trained models, we compare with three state-of-the-
art pretrained generation models: BART (Lewis
et al., 2020) 5, PEGASUS (Zhang et al., 2020)
and Longformer-Encoder-Decoder(LED) (Beltagy
et al., 2020). These pretrained models have been
shown to outperform dataset-specific models in
summarization (Lewis et al., 2020; Zhang et al.,
2020), and because of pretraining, they are ex-
pected to also work well in the few-shot settings.
As there is no prior work doing few-shot and zero-
shot evaluations on all the datasets we consider,
and also the results in the few-shot setting might
be influenced by sampling variability (especially
with only 10 examples) (Bragg et al., 2021), we
run the same experiments for the compared mod-
els five times with different random seeds (shared
with all the models), with the publicly available
checkpoints .6
Similar to Pasunuru et al. (2021), the inputs of
all the models are the concatenations of the docu-
ments within the clusters (in the same order), each
document is truncated based on the input length
limit divided by the total number of documents so
4We use https://github.com/google-research/google-
research/tree/master/rouge with default stemmer settings.
5Pilot experiments comparing BART and T5 showed
BART to outperform T5 on the few-shot evaluation of Multi-
News (with A VG ROUGE of 23.5/26.4 (T5) v.s. 25.2/26.7
(BART) for 10/100 training examples, respectively). Thus, we
are using BART as one of the baselines.
6Checkpoints from https://huggingface.co/models
5249

Models Multi-News(256) Multi-XSci(128) WCEP(50) WikiSum(128) arXiv(300) DUC2004 (128)
R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L R-1 R-2 R-L
PEGASUS ⋆(Zhang et al., 2020) 36.5 10.5 18.7 - - - - - - - - - 28.1 6.6 17.7 - - -
PEGASUS (our run) 32.0 10.1 16.7 27.6 4.6 15.3 33.2 12.7 23.8 24.6 5.5 15.0 29.5 7.9 17.1 32.7 7.4 17.6
BART (our run) 27.3 6.2 15.1 18.9 2.6 12.3 20.2 5.7 15.3 21.6 5.5 15.0 29.2 7.5 16.9 24.1 4.0 15.3
LED (our run) 17.3 3.7 10.4 14.6 1.9 9.9 18.8 5.4 14.7 10.5 2.4 8.6 15.0 3.1 10.8 16.6 3.0 12.0
PRIMERA (our model) 42.0 13.6 20.8 29.1 4.6 15.7 28.0 10.3 20.9 28.0 8.0 18.0 34.6 9.4 18.3 35.1 7.2 17.9
Table 2: Zero-shot results. The models in the first block use the full-length attention (O(n2)) and are pretrained on
the single document datasets. The numbers in the parenthesis following each dataset indicate the output length limit
set for inference. P EGASUS ⋆ means results taken exactly from PEGASUS (Zhang et al., 2020), where available.
0 10 100
10
15
20
25AVG ROUGE
Multi-News
0 10 100
8
10
12
14
16
18
Multi-Xscience
0 10 100
5
10
15
20
25
Wikisum
0 10 100
10
20
30
WCEP
0 10 100
10
15
20
25
arXiv
0 20
10
15
20
DUC2004
Ours
BART
PEGASUS
LED
Number of Training Examples
Figure 5: The A VG ROUGE scores (R-1, R-2 and R-L) of the pretrained models with 0, 10 and 100 training data
with variance. All the results of few-shot experiments (10 and 100) are obtained by the average of 5 random runs
(with std, and the same set of seeds shared by all the models). Note that DUC2004 only has 50 examples, we use
20/10/20 for train/valid/test in the few-shot experiments.
that all documents are represented in the input. 7
To preserve the same format as the correspond-
ing pretrained models, we set the length limit of
output for BART and PEGASUS exactly as their
pretrained settings on all of the datasets (except
for the zero-shot experiments, the details can be
found in Sec.4.3). Regarding length limit of in-
puts, we tune the baselines by experimenting with
512, 1024, 4096 on Multi-News dataset in few-shot
setting (10 data examples), and the model with
length limit 512(PEGASUS )/1024(BART) achieves
the best performance, thus we use this setting (de-
tailed experiment results for different input lengths
can be found in Appx. §C.1). We use the same
length limit as our model for the LED model, i.e.
4096/1024 for input and output respectively, for all
the datasets.
4.3 Zero-Shot Results
For zero-shot8 abstractive summarization experi-
ments, since the models have not been trained on
the downstream datasets, the lengths of generated
summaries mostly depend on the pretrained set-
tings. Thus to better control the length of gener-
ated summaries and for a fair comparison between
all models, following Zhu et al. (2021), we set the
7Pilot experiments show simple truncation results in infe-
rior performance, which is in line with Pasunuru et al. (2021).
8For clarity, by zero-shot we mean using the pretrained
model directly without any additional supervision.
length limit of the output at inference time to the av-
erage length of gold summaries.9 Exploring other
approaches to controlling length at inference time
(e.g., Wu et al., 2021) is an orthogonal direction,
which we leave for future work.
Table 2 shows the performance comparison
among all the models. Results indicate that our
model achieves substantial improvements com-
pared with all the three baselines on most of the
datasets. As our model is pretrained on clusters of
documents with longer input and output, the benefit
is stronger on the dataset with longer summaries,
e.g. Multi-News and arXiv. Comparing PEGASUS
and BART models, as the objective of PEGASUS
is designed mainly for summarization tasks, not
surprisingly it has relatively better performances
across different datasets. Interestingly, LED un-
derperforms other models, plausibly since part of
the positional embeddings (1k to 4k) are not pre-
trained. Encouragingly, our model performs the
best, demonstrating the benefits of our pretraining
strategy for multi-document summarization.
4.4 Few Shot Evaluation
Compared with the strict zero-shot scenario, few-
shot experiments are closer to the practical scenar-
ios, as it is arguably affordable to label dozens of
examples for almost any application.
9In practice, it is reasonable to assume knowing the approx-
imate length of the expected summary for a given task/domain.
5250