PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization — part 4/6

Cleaned-text code points [41509, 56341). Bibliography omitted here; source.md includes it.
References
Armen Aghajanyan, Dmytro Okhonko, Mike Lewis,
Mandar Joshi, Hu Xu, Gargi Ghosh, and Luke
Zettlemoyer. 2021. HTLM: hyper-text pre-training
and prompting of language models. CoRR,
abs/2107.06955.
Iz Beltagy, Matthew E. Peters, and Arman Cohan.
2020. Longformer: The long-document transformer.
CoRR, abs/2004.05150.
Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Belt-
agy. 2021. Flex: Unifying evaluation for few-shot
nlp. arXiv preprint arXiv:2107.07170.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Pe-
ters, Arie Cattan, and Ido Dagan. 2021. CDLM:
Cross-document language modeling. In Findings
of the Association for Computational Linguistics:
EMNLP 2021, pages 2648–2662, Punta Cana, Do-
minican Republic. Association for Computational
Linguistics.
Janara Christensen, Mausam, Stephen Soderland, and
Oren Etzioni. 2013. Towards coherent multi-
document summarization. In Proceedings of the
2013 Conference of the North American Chapter of
the Association for Computational Linguistics: Hu-
man Language Technologies, pages 1163–1173, At-
lanta, Georgia. Association for Computational Lin-
guistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim,
Trung Bui, Seokhwan Kim, Walter Chang, and Na-
zli Goharian. 2018. A discourse-aware attention
model for abstractive summarization of long docu-
ments. In Proceedings of the 2018 Conference of
the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies, Volume 2 (Short Papers), pages 615–621,
New Orleans, Louisiana. Association for Computa-
tional Linguistics.
Hoa Trang Dang. 2005. Overview of duc 2005. In Pro-
ceedings of the document understanding conference,
volume 2005, pages 1–12.
Hoa Trang Dang and Karolina Owczarzak. 2008.
Overview of the tac 2008 update summarization task.
Theory and Applications of Categories.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and
Dragomir Radev. 2019. Multi-news: A large-scale
multi-document summarization dataset and abstrac-
tive hierarchical model. In Proceedings of the 57th
Annual Meeting of the Association for Computa-
tional Linguistics, pages 1074–1084, Florence, Italy.
Association for Computational Linguistics.
Sebastian Gehrmann, Yuntian Deng, and Alexander
Rush. 2018. Bottom-up abstractive summarization.
In Proceedings of the 2018 Conference on Em-
pirical Methods in Natural Language Processing ,
pages 4098–4109, Brussels, Belgium. Association
for Computational Linguistics.
Demian Gholipour Ghalandari, Chris Hokamp,
Nghia The Pham, John Glover, and Georgiana Ifrim.
2020. A large-scale multi-document summarization
dataset from the Wikipedia current events portal.
In Proceedings of the 58th Annual Meeting of the
Association for Computational Linguistics , pages
1302–1308, Online. Association for Computational
Linguistics.
Travis Goodwin, Max Savery, and Dina Demner-
Fushman. 2020. Flight of the PEGASUS? compar-
ing transformers on few-shot and zero-shot multi-
document abstractive summarization. In Proceed-
ings of the 28th International Conference on Com-
putational Linguistics, pages 5640–5646, Barcelona,
Spain (Online). International Committee on Compu-
tational Linguistics.
Xiaotao Gu, Yuning Mao, Jiawei Han, Jialu Liu, You
Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi
Zhai, and Nicholas Zukoski. 2020. Generating rep-
resentative headlines for news stories. In Proceed-
ings of The Web Conference 2020, WWW ’20, page
1773–1784, New York, NY , USA. Association for
Computing Machinery.
Chris Hokamp, Demian Gholipour Ghalandari,
Nghia The Pham, and John Glover. 2020. Dyne:
Dynamic ensemble decoding for multi-document
summarization. CoRR, abs/2006.08748.
Matthew Honnibal, Ines Montani, Soﬁe Van Lan-
deghem, and Adriane Boyd. 2020. spaCy:
Industrial-strength Natural Language Processing in
Python.
Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020.
Multi-granularity interaction network for extractive
and abstractive multi-document summarization. In
Proceedings of the 58th Annual Meeting of the Asso-
ciation for Computational Linguistics , pages 6244–
6254, Online. Association for Computational Lin-
guistics.
Dan Jurafsky and James H. Martin. 2009. Speech and
language processing : an introduction to natural
language processing, computational linguistics, and
speech recognition. Pearson Prentice Hall, Upper
Saddle River, N.J.
Yoav Levine, Barak Lenz, Opher Lieber, Omri
Abend, Kevin Leyton-Brown, Moshe Tennenholtz,
and Yoav Shoham. 2020. Pmi-masking: Princi-
pled masking of correlated spans. arXiv preprint
arXiv:2010.01825.
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020. BART: Denoising sequence-to-sequence pre-
training for natural language generation, translation,
and comprehension. In Proceedings of the 58th An-
nual Meeting of the Association for Computational
Linguistics, pages 7871–7880, Online. Association
for Computational Linguistics.
5254

Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng
Wang, and Junping Du. 2020. Leveraging graph
to improve abstractive multi-document summariza-
tion. In Proceedings of the 58th Annual Meeting
of the Association for Computational Linguistics ,
pages 6232–6243, Online. Association for Compu-
tational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Ab-
stract Meaning Representation for multi-document
summarization. In Proceedings of the 27th Inter-
national Conference on Computational Linguistics ,
pages 1178–1190, Santa Fe, New Mexico, USA. As-
sociation for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto-
matic evaluation of summaries. In Text Summariza-
tion Branches Out , pages 74–81, Barcelona, Spain.
Association for Computational Linguistics.
Peter J. Liu*, Mohammad Saleh*, Etienne Pot, Ben
Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam
Shazeer. 2018. Generating wikipedia by summariz-
ing long sequences. In International Conference on
Learning Representations.
Yang Liu and Mirella Lapata. 2019a. Hierarchical
transformers for multi-document summarization. In
Proceedings of the 57th Annual Meeting of the
Association for Computational Linguistics , pages
5070–5081, Florence, Italy. Association for Compu-
tational Linguistics.
Yang Liu and Mirella Lapata. 2019b. Text summariza-
tion with pretrained encoders. In Proceedings of
the 2019 Conference on Empirical Methods in Nat-
ural Language Processing and the 9th International
Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 3730–3740, Hong Kong,
China. Association for Computational Linguistics.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi-
XScience: A large-scale dataset for extreme multi-
document summarization of scientiﬁc articles. In
Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP),
pages 8068–8074, Online. Association for Computa-
tional Linguistics.
Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and
Jiawei Han. 2020. Multi-document summarization
with maximal marginal relevance-guided reinforce-
ment learning. In Proceedings of the 2020 Con-
ference on Empirical Methods in Natural Language
Processing (EMNLP), pages 1737–1751, Online. As-
sociation for Computational Linguistics.
Ani Nenkova and Rebecca Passonneau. 2004. Evaluat-
ing content selection in summarization: The pyra-
mid method. In Proceedings of the Human Lan-
guage Technology Conference of the North Ameri-
can Chapter of the Association for Computational
Linguistics: HLT-NAACL 2004 , pages 145–152,
Boston, Massachusetts, USA. Association for Com-
putational Linguistics.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal,
Sujith Ravi, and Markus Dreyer. 2021. Efﬁciently
summarizing text and graph encodings of multi-
document clusters. In Proceedings of the 2021 Con-
ference of the North American Chapter of the Asso-
ciation for Computational Linguistics: Human Lan-
guage Technologies, pages 4768–4779, Online. As-
sociation for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather-
ine Lee, Sharan Narang, Michael Matena, Yanqi
Zhou, Wei Li, and Peter J. Liu. 2020. Exploring
the limits of transfer learning with a uniﬁed text-to-
text transformer. Journal of Machine Learning Re-
search, 21(140):1–67.
Sascha Rothe, Joshua Maynez, and Shashi Narayan.
2021. A thorough evaluation of task-speciﬁc pre-
training for summarization. In Proceedings of the
2021 Conference on Empirical Methods in Natural
Language Processing, pages 140–145, Online and
Punta Cana, Dominican Republic. Association for
Computational Linguistics.
Andy Su, Difei Su, John M Mulvey, and H Vincent
Poor. 2021. Pobrl: Optimizing multi-document sum-
marization by blending reinforcement learning poli-
cies. arXiv preprint arXiv:2105.08244.
Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus
Stenetorp, and Caiming Xiong. 2021. Controllable
abstractive dialogue summarization with sketch su-
pervision. In Findings of the Association for Com-
putational Linguistics: ACL-IJCNLP 2021 , pages
5108–5122, Online. Association for Computational
Linguistics.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu,
Ayush Pareek, Krishnan Srinivasan, and Dragomir
Radev. 2017. Graph-based neural multi-document
summarization. In Proceedings of the 21st Confer-
ence on Computational Natural Language Learning
(CoNLL 2017), pages 452–462, Vancouver, Canada.
Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava
Dubey, Joshua Ainslie, Chris Alberti, Santiago On-
tanon, Philip Pham, Anirudh Ravula, Qifan Wang,
Li Yang, and Amr Ahmed. 2020. Big bird: Trans-
formers for longer sequences. In Advances in
Neural Information Processing Systems, volume 33,
pages 17283–17297. Curran Associates, Inc.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and
Peter Liu. 2020. PEGASUS: Pre-training with ex-
tracted gap-sentences for abstractive summarization.
In Proceedings of the 37th International Conference
on Machine Learning , volume 119 of Proceedings
of Machine Learning Research, pages 11328–11339.
PMLR.
Chenguang Zhu, Ziyi Yang, Robert Gmyr, Michael
Zeng, and Xuedong Huang. 2021. Leveraging Lead
Bias for Zero-Shot Abstractive News Summarization,
page 1462–1471. Association for Computing Ma-
chinery, New York, NY , USA.
5255

Yanyan Zou, Xingxing Zhang, Wei Lu, Furu Wei, and
Ming Zhou. 2020. Pre-training for abstractive doc-
ument summarization by reinstating source text. In
Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP),
pages 3646–3660, Online. Association for Computa-
tional Linguistics.
A Implementation details of pre-training
As the multi-document summarization task
has a higher compression ratio, deﬁned as
len(Summary )/len(Input), (e.g. 12% for
Multi-News dataset and 15% for Multi-Xscience
dataset), we use 15% as the ratio of masked sen-
tences for generation. In addition to this 15%
masked sentences, following PEGASUS (Zhang
et al., 2020), we also copy an additional 15% of
the input sentences to the output without masking
them in the input. This allows the model to also
learn to copy information from the source directly
and found to be useful by Zhang et al. (2020).
We pretrain the model for 100K steps, with early
stopping, batch size of 16, Adam optimizer with
a learning rate of 3e−5 following Beltagy et al.
(2020), with 10K warmup steps and linear decay.
The pretraining process takes likely 7 days on 4
A100 GPUs.
As the backbone of PRIMERA is the Longformer
Encoder Decoder model (LED), it has the same
number of parameters with LED (447M).
B Detailed Description on the Evaluation
Datasets
The details of evaluation datasets can be found
below.
Multi-News (Fabbri et al., 2019): A multi-
document dataset with summaries written by pro-
fessional editors from the newser.com.
Wikisum (Liu* et al., 2018) Each summary is a
Wikipedia article, and the source documents are
either citations in the reference section or the Web
Search results of section titles. 14 In our experi-
ments, we use the data crawled by Liu and Lapata
(2019a).
WCEP (Gholipour Ghalandari et al., 2020) is
built based on news events from Wikipedia Current
Events Portal and the references are obtained simi-
lar to Wikisum. There are at most 100 documents
within each cluster in the original dataset, thus we
14Due to the large size of the dataset, we evaluate all the
models on the ﬁrst 3200 data in the test set. And in the few-
shot experiments, we randomly choose few examples (10 or
100) from the training set and validation set.
remove all the duplicates and only keep up to 10
documents for each cluster based on the relevance
score in the original dataset, which is similar to the
WCEP-10 variant in the original paper.
Multi-X-Science (Lu et al., 2020) a multi-
document summarization dataset created from sci-
entiﬁc articles, the summaries are paragraphs of
related work section, while source documents in-
clude the abstracts of the query and referred papers.
DUC benchmarks (Dang, 2005) include multi-
document summarization datasets in the news
domain, with 10-30 documents and 3-4 human-
written summaries per cluster. Since these datasets
are small, we use them primarily for a few-shot
evaluation. We use DUC2003 for training (only
one of the reference summaries for each document
is used for training) and DUC2004 as test.
ArXiv (Cohan et al., 2018) is a single document
summarization dataset in the scientiﬁc paper do-
main. Each document is a scientiﬁc paper, and the
summary is the corresponding abstract. As each
scientiﬁc paper consists of multiple sections, we
treat each section as a separate document within
a cluster in our experiments. This is to evaluate
our model’s effectiveness on summarizing single
documents having multiple sections.
C Details on Compared models
The details of compared models in the zero-/few-
shot setting can be found below.
BART (Lewis et al., 2020) an encoder-decoder
transformer model pretrained on the objective of
reconstructing the corrupted documents in multiple
ways, e.g. Token Deletion, Text Inﬁlling, Sentence
Rotation and etc.
PEGASUS (Zhang et al., 2020) a pretrained
model designed for abstractive summarization as
the downstream task, especially for the single doc-
ument input. It is trained on the objective of Gap
Sentence Generation on C4 (Raffel et al., 2020) and
Hugenews datasets (Note that the pretraining data
size in PEGASUS is magnitudes larger than ours).
As it is only evaluated on one multi-document
summarization dataset (Multi-news), we rerun the
model on all the datasets. To verify the quality
of our reproduction, the average ROUGE scores
of our re-run model vs. (the ones reported on the
paper) with 10 examples and 100 examples fed
are 23.81± 0.79 vs. (24.13) and 25.86± 0.41 vs.
(25.48), with minor differences plausibly resulting
from different samplings.
5256