PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization — part 5/6

Cleaned-text code points [56341, 67696). Bibliography omitted here; source.md includes it.

Length Limit BART P EGASUS
R-1 R-2 R-L R-1 R-2 R-L
512 - - - 39.0 12.1 20.3
1024 42.3 13.7 19.7 37.6 10.7 18.8
4096 37.9 11.0 17.5 34.9 8.7 17.6
Table 6: The ROUGE score (R-1/R-2/R-3) for pre-
trained models (BART and P EGASUS ) with different
input length limit in few-shot setting (10 data example)
on the multi-news dataset. The results are the average
over 5 runs on different subsets (the same seeds shared
with all the other models in this paper).
Longformer Encoder-Decoder (LED) (Beltagy
et al., 2020) is the initial state of our model before
pretraining. The parameters of LED are inherited
from the BART model, and to enable the model
to deal with longer input, the position embeddings
are repeatedly copied from BART’s 1K position
embeddings. It is different from our model with re-
spect to both pretraining and input structure (docu-
ment separators and global attentions), with global
attention on the (<s>) token only and no document
separators.
C.1 Detailed Experiment for Input Length
Limit
We run an experiment to select the proper length
limit for compared pretrained models, i.e. BART
and PEGASUS . Specifically, we train both models
with different input length limits (512/1024/4096)
in the few-shot setting (with 10 data examples) on
the multi-news dataset. Similar as the few-shot
experiments described in §4.2, we train each model
with each specific input length limit for 5 times
on different subsets, which are shared by all the
models. As shown in Table 6, BART with length
limit 1024 performs the best and PEGASUS with
length limit 512 performs the best, thus in all our
experiments, we use 1024 as the input length limit
for BART and 512 for PEGASUS .
D Hyperparameters in Few-shot and
Full Supervised Experiments
D.1 Few-shot Experiments
We use Adam as the optimizer with linear sched-
uled learning rate 3e− 5 for BART, LED and our
model, and use the default optimization settings of
the few-shot experiments from Zhang et al. (2020),
i.e. AdaFactor optimizer with scheduled learning
rate 5e− 4. For all the experiments with 10 exam-
ples, the batch size is 10, the models are trained
for 200 steps, with warm-up as 20 steps. For the
experiments with 100 examples, we use the same
batch size, with the total step and warm-up step set
to be 1000 and 100, respectively.
D.2 Fully Supervised Experiments
We use Adam as the optimizer with linear sched-
uled learning rate 3e− 5, and batch size as 16 for
all the datasets in the full supervised experiments.
The number of steps and warm-up steps are set
based on the size of the datasets. The details can
be found in Table 7
Dataset Total Steps Warmup Steps
Multi-News 25k 2.5k
Multi-XScience 20k 2k
WCEP 5k .5k
arXiv 40k 4k
Table 7: Details of total steps and warm-up steps used
in the Full Supervised experiments.
E Detailed Results in Few-shot Setting
The exact ROUGE scores in Figure 5 are shown in
Table 8.
Model 0 Examples 10 Examples 100 Examples
Multi-News
PEGASUS 31.97/10.06/16.74 39.02/12.10/20.32 42.99/13.50/21.10
BART 26.10/8.98/13.06 42.30/13.74/19.71 44.23/14.77/21.02
LED 16.60/4.78/9.05 38.86/12.48/18.82 44.45/14.85/21.16
Ours 39.09/13.91/19.19 44.02/15.54/22.03 46.01/16.76/22.91
Multi-Science
PEGASUS 27.33/4.77/15.04 28.14/4.68/15.49 28.01/4.09/15.89
BART 15.21/3.49/8.61 27.80/4.74/14.90 31.17/5.32/16.45
LED 11.79/2.47/6.86 26.57/4.05/15.36 29.46/4.85/16.32
Ours 26.90/4.98/14.09 28.36/4.73/15.29 31.25/5.43/16.84
Wikisum
PEGASUS 23.67/5.37/14.17 23.44/6.44/16.21 28.50/9.83/21.33
BART 15.80/4.60/9.13 28.95/9.88/20.80 32.97/13.81/25.01
LED 8.70/2.34/5.78 26.53/9.30/19.95 34.15/16.03/26.75
Ours 17.79/5.02/10.90 31.10/13.26/23.39 36.05/17.85/27.81
WCEP
PEGASUS 27.69/10.85/20.03 35.60/14.84/26.84 42.09/19.93/33.04
BART 7.11/3.41/5.32 37.46/15.82/28.70 41.34/19.19/32.58
LED 5.69/2.19/4.32 36.29/15.04/27.80 41.83/19.46/32.92
Ours 13.50/5.30/10.11 38.97/17.55/30.64 42.96/20.53/33.87
arXiv
PEGASUS 29.76/7.94/17.27 33.10/8.52/19.40 36.38/9.55/20.83
BART 23.26/7.57/12.01 32.53/8.70/17.98 37.62/10.78/20.99
LED 13.94/3.76/8.35 36.51/11.16/20.68 41.00/13.74/22.34
Ours 29.14/8.64/15.82 41.13/13.81/23.02 43.42/15.85/24.07
Table 8: Detailed ROUGE scores (R-1/R-2/R-L) on all
the datasets in the few-shot setting (corresponds to Fig-
ure 5)
5257

F Detailed Analysis on Fully Supervised
Experiments
To show the advantage of our pre-trained model
when there is sufficient data, we also train the
model with the full training set, and the results
can be found in Table 9-12 15, along with the re-
sults from previous works. Differently from the
zero-/few-shot experiments, here we report the
state-of-the-art results on different datasets, as they
were presented in the corresponding original pa-
pers. Since we use the same train/valid/test set as
in those prior works, we can perform a fair com-
parison , without re-running all those extremely
time-consuming experiments .
Overall, our model achieves state-of-the-art on
Multi-News (see Table 9 , WCEP dataset (see Ta-
ble 11) and arXiv dataset (see Table 12).
Models ROUGE-1 ROUGE-2 ROUGE-L
PEGASUS (Zhang et al., 2020) 47.52 18.72 24.91
BART-Long-Graph (Pasunuru et al., 2021) 49.03 19.04 24.04
BART-Long-Graph(1000) (Pasunuru et al., 2021) 49.24 18.99 23.97
BART-Long(1000) (Pasunuru et al., 2021) 49.15 19.50 24.47
Ours 49.94 21.05 25.85
Table 9: ROUGE scores of the previous models and our
fully supervised model on the Multi-News dataset. The
results of P EGASUS is from Zhang et al. (2020), and
the other results are from Pasunuru et al. (2021)
Multi-News The experiment results on Multi-
News dataset can be found in Table 9. Specifically,
the PEGASUS model (Zhang et al., 2020) is pre-
trained on a large-scale single-document dataset
with the Gap Sentence Generation objective, which
is the same as ours, but with a different mask-
ing strategy, BART-Long (Pasunuru et al., 2021)
uses the same model structure as ours , and BART-
Long-Graph (Pasunuru et al., 2021) additionally
has discourse graph injected. Comparing the re-
sults with the BART-Long model, our model is
around 1 ROUGE point higher, which may result
from either better model structure or pre-training.
Interestingly, in one of the ablation studies in Pa-
sunuru et al. (2021), they find that the BART-Long
model achieves its best performance with the length
limit of 1000, and no further improvement is found
when the length limit is greater than that. Thus we
may conclude the gap between the performances is
mainly from our design on the model, i.e. the doc-
ument separators, proper global attention as well
as the pre-training on a multi-document dataset.
15Due to the lack of computational resources, we do not
train the model on Wikisum.
Models R1 R2 RL*
LEAD 27.46 4.57 -
BERTABS 31.56 5.02 -
BART 32.83 6.36 -
SCIBERTABS 32.12 5.59 -
SOTA(Pointer Generator) 34.11 6.76 18.2
LEAD(ours) 26.49 4.26 14.70
Ours 31.93 7.37 18.02
Table 10: ROUGE scores of the previous models
and our fully supervised model on the Multi-Xscience
dataset. All the results are from Lu et al. (2020). * The
ROUGE-L is not comparable as we have different set-
tings on the settings of evaluation, see the gap between
LEAD and LEAD(ours).
Models R1 R2 RL
BERTREG (Gholipour Ghalandari et al., 2020) 35.0 13.5 25.5
SUBMODULAR+ABS(Gholipour Ghalandari et al., 2020) 30.6 10.1 21.4
DynE (Hokamp et al., 2020) 35.4 15.1 25.6
Ours 46.08 25.21 37.86
Table 11: ROUGE scores of the previous models and
our fully supervised model on the WCEP dataset.
WCEP As for the WCEP dataset, BERTREG
(Gholipour Ghalandari et al., 2020) is a Regression-
based sentence ranking system with BERT em-
bedding, which is used as extractive summariza-
tion method, while Submodular+Abs is a simple
two-step abstractive summarization model with a
submodular-based extractive summarizer followed
by a bottom-up abstractive summarizer (Gehrmann
et al., 2018). DynE is a BART-based abstractive
approach, which is to ensemble multiple input, al-
lowing single document summarization models to
be directly leveraged on the multi-document sum-
marization task. Our model outperforms all the
models by a large margin, including the SOTA
model DynE, and it may indicate that the plain
structure is more effective than purely ensembling
the output of single documents.
arXiv In addition to the experiments on multi-
document summarization datasets, we also com-
pare our fully supervised model with previous
works on the arXiv dataset, with each section
treated as a single document. All the models to
be compared with are based on pre-trained mod-
els, and Bigbird- PEGASUS and LED utilize the
pre-training of PEGASUS (Zaheer et al., 2020) and
BART (Lewis et al., 2020), respectively. However,
both Bigbird and LED apply more efficient atten-
tions, which make the models able to take longer
5258

Models R1 R2 RL
PEGASUS (1K) 44.21 16.95 38.83
Bigbird-PEGASUS (3k) 46.63 19.02 41.77
LED(4K) 44.40 17.94 39.76
LED(16K) 46.63 19.62 41.83
Ours(4k) 47.58 20.75 42.57
Table 12: ROUGE scores of the previous models and
our fully supervised model on the arXiv dataset. The re-
sult of P EGASUS and BigBird-PEGASUS are from (Za-
heer et al., 2020), and the results of LED are from (Belt-
agy et al., 2020). The number in the parenthesis indi-
cates the length limit of the input.
input (3k for BigBird, 4K and 16k for LED). Our
model has a better performance than all the models,
including LED(16K), which allows for the input
4 times longer than ours. It is worth mentioning
that LED(4K) has the same structure as our model,
with the same length limit of the input, and with
the pre-training on multi-document datasets, our
model is more than 3 ROUGE point better than it,
which shows that the strategy not only works for
multi-document summarization but can also effec-
tively improve single-document summarization for
long documents.
G Examples of Generated Summaries
We show an example (from Multi-News) of gener-
ated summaries by PRIMERA and compared mod-
els trained with different number of examples in
Table 13. And we show an example from DUC2007
(which is one of the examples used for human eval-
uation) with generated summaries by PRIMERA
and two compared models in Table 14, with all
the models trained on 10 data examples from
DUC2007.
H Software and Licenses
Our code is licensed under Apache License 2.0.
Our framework dependencies are:
• HuggingFace Datasets16, Apache 2.0
• NLTK 17, Apache 2.0
• Numpy18, BSD 3-Clause "New" or "Revised"
• Spacy19, MIT
16https://github.com/huggingface/
datasets/blob/master/LICENSE
17https://github.com/nltk/nltk
18https://github.com/numpy/numpy/blob/
main/LICENSE.txt
19https://github.com/explosion/spaCy/
• Transformers20, Apache 2.0
• Pytorch21, Misc
• Pytorch Lightning 22,Apache 2.0
• Longformer23, Apache 2.0
• ROUGE 24, Apache 2.0
I Annotation Instructions for Human
Evaluation
Figure 7 and Figure 8 shows the annotation instruc-
tion for human annotators.
blob/master/LICENSE
20https://github.com/huggingface/
transformers/blob/master/LICENSE
21https://github.com/pytorch/pytorch/
blob/master/LICENSE
22https://github.com/PyTorchLightning/
pytorch-lightning/blob/master/LICENSE
23https://github.com/allenai/
longformer/blob/master/LICENSE
24https://github.com/google-research/
google-research/tree/master/rouge
5259

Figure 7: Annotation instruction for human annotators.
5260

Figure 8: Annotation instruction for human annotators.
5261