PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization — part 1/6

Cleaned-text code points [0, 14370). Bibliography omitted here; source.md includes it.

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics
V olume 1: Long Papers, pages 5245 - 5263
May 22-27, 2022 c©2022 Association for Computational Linguistics
PRIMERA : Pyramid-based Masked Sentence Pre-training for
Multi-document Summarization
Wen Xiao†∗ Iz Beltagy‡ Giuseppe Carenini† Arman Cohan‡§
†University of British Columbia, Vancouver, Canada
‡Allen Institute for AI, Seattle, W A, USA
§Paul G. Allen School of Computer Science & Engineering, University of Washington
{xiaowen3,carenini}@cs.ubc.ca, {beltagy,armanc}@allenai.org
Abstract
We introduce P RIMERA , a pre-trained model
for multi-document representation with a fo-
cus on summarization that reduces the need
for dataset-specific architectures and large
amounts of fine-tuning labeled data. PRIMERA
uses our newly proposed pre-training objective
designed to teach the model to connect and ag-
gregate information across documents. It also
uses efficient encoder-decoder transformers to
simplify the processing of concatenated input
documents. With extensive experiments on 6
multi-document summarization datasets from
3 different domains on zero-shot, few-shot
and full-supervised settings, PRIMERA outper-
forms current state-of-the-art dataset-specific
and pre-trained models on most of these set-
tings with large margins.1
1 Introduction
Multi-Document Summarization is the task of
generating a summary from a cluster of re-
lated documents. State-of-the-art approaches to
multi-document summarization are primarily ei-
ther graph-based (Liao et al., 2018; Li et al., 2020;
Pasunuru et al., 2021), leveraging graph neural net-
works to connect information between the docu-
ments, or hierarchical (Liu and Lapata, 2019a;
Fabbri et al., 2019; Jin et al., 2020), building inter-
mediate representations of individual documents
and then aggregating information across. While ef-
fective, these models either require domain-specific
additional information e.g. Abstract Meaning
Representation (Liao et al., 2018), or discourse
graphs (Christensen et al., 2013; Li et al., 2020), or
use dataset-specific, customized architectures, mak-
ing it difficult to leverage pretrained language mod-
els. Simultaneously, recent pretrained language
models (typically encoder-decoder transformers)
∗Work mainly done during an internship at AI2.
1The code and pre-trained models can be found athttps:
//github.com/allenai/PRIMER
General-Purpose  
Pre-trained Models 
T5 BART
BigBird LED
Task-Specific  
Pre-trained Models 
PEGASUS
PRIMERA
Single-doc
Summarization
Multi-doc  
Summarization
+ Pre-training w/ Entity Pyramid 
+ Model Struct for multi-doc input
Figure 1: P RIMERA vs existing pretrained models.
have shown the advantages of pretraining and trans-
fer learning for generation and summarization (Raf-
fel et al., 2020; Lewis et al., 2020; Beltagy et al.,
2020; Zaheer et al., 2020). Yet, existing pretrained
models either use single-document pretraining ob-
jectives or use encoder-only models that do not
work for generation tasks like summarization (e.g.,
CDLM, Caciularu et al., 2021).
Therefore, we argue that these pretrained models
are not necessarily the best fit for multi-document
summarization. Alternatively, we propose a simple
pretraining approach for multi-document summa-
rization, reducing the need for dataset-specific ar-
chitectures and large fine-tuning labeled data (See
Figure 1 to compare with other pretrained mod-
els). Our method is designed to teach the model to
identify and aggregate salient information across
a “cluster” of related documents during pretrain-
ing. Specifically, our approach uses the Gap Sen-
tence Generation objective (GSG) (Zhang et al.,
2020), i.e. masking out several sentences from
the input document, and recovering them in or-
der in the decoder. We propose a novel strategy
for GSG sentence masking which we call, En-
tity Pyramid, inspired by the Pyramid Evaluation
method (Nenkova and Passonneau, 2004). With
Entity Pyramid, we mask salient sentences in the
entire cluster then train the model to generate them,
encouraging it to find important information across
documents and aggregate it in one summary.
We conduct extensive experiments on 6 multi-
document summarization datasets from 3 differ-
ent domains. We show that despite its simplic-
5245

Longformer Encoder Decoder (LED)
Doc 1 Doc 2 Doc 3
Masked Sentence
<s> [sent mask] <doc-sep>Six wildfires were ... Wildfires continued ... [sent mask] <doc-sep> The Buffalo ... </s> 
Wildfires have burned ... . In Colorado 's<s> </s> 
<doc-sep>
w/ Global
Attention
w/ Global
Attention
w/ Global
Attention
Global Attention
Local Attention
...
Sent 1 Sent 2
Tokens with  
Global Attention
Tokens with 
Local Attention
Documents are separated with <doc-sep>
tokens and they are assigned global
attention.  
Other tokens except for <s> have local
attention only. 
Selected sentences are replaced with a
special [sent mask] token 
The model is trained to generate the
masked sentences 
Figure 2: Model Structure of P RIMERA .
ity, PRIMERA achieves superior performance com-
pared with prior state-of-the-art pretrained models,
as well as dataset-specific models in both few-shot
and full fine-tuning settings. PRIMERA performs
particularly strong in zero- and few-shot settings,
significantly outperforming prior state-of-the-art
up to 5 Rouge-1 points with as few as 10 examples.
Our contributions are summarized below:
1. We release PRIMERA , the first pretrained gener-
ation model for multi-document inputs with focus
on summarization.
2. We propose Entity Pyramid, a novel pretraining
strategy that trains the model to select and aggre-
gate salient information from documents.
3. We extensively evaluatePRIMERA on 6 datasets
from 3 different domains for zero-shot, few-shot
and fully-supervised settings. We show that
PRIMERA outperforms current state-of-the-art on
most of these evaluations with large margins.
2 Model
In this section, we discuss our proposed model
PRIMERA , a new pretrained general model for
multi-document summarization. Unlike prior work,
PRIMERA minimizes dataset-specific modeling by
simply concatenating a set of documents and pro-
cessing them with a general efficient encoder-
decoder transformer model (§2.1). The underlying
transformer model is pretrained on an unlabeled
multi-document dataset, with a new entity-based
sentence masking objective to capture the salient in-
formation within a set of related documents (§2.2).
2.1 Model Architecture and Input Structure
Our goal is to minimize dataset-specific modeling
to leverage general pretrained transformer models
for the multi-document task and make it easy to
use in practice. Therefore, to summarize a set of
related documents, we simply concatenate all the
documents in a single long sequence, and process
them with an encoder-decoder transformer model.
Since the concatenated sequence is long, instead of
more standard encoder-decoder transformers like
BART (Lewis et al., 2020) and T5 (Raffel et al.,
2020), we use the Longformer-Encoder-Decoder
(LED) Model (Beltagy et al., 2020), an efficient
transformer model with linear complexity with
respect to the input length. 2 LED uses a sparse
local+global attention mechanism in the encoder
self-attention side while using the full attention on
decoder and cross-attention.
When concatenating, we add special document
separator tokens (<doc-sep>) between the doc-
uments to make the model aware of the document
boundaries (Figure 2). We also assign global at-
tention to these tokens which the model can use
to share information across documents (Caciularu
et al., 2021) (see §5 for ablations of the effective-
ness of this input structure and global attention).
2.2 Pretraining objective
In summarization, task-inspired pretraining ob-
jectives have been shown to provide gains
over general-purpose pretrained transformers
(PEGASUS ; Zhang et al., 2020). In particular, PE-
GASUS introduces Gap Sentence Generation (GSG)
as a pretraining objective where some sentences are
masked in the input and the model is tasked to gen-
erate them. Following PEGASUS , we use the GSG
objective, but introduce a new masking strategy
designed for multi-document summarization. As in
GSG, we select and mask out m summary-like sen-
tences from the input documents we want to sum-
marize, i.e. every selected sentence is replaced by a
2We use LED and not other efficient transformers like
Bigbird-PEGASUS (Zaheer et al., 2020) for two reasons, the
first is that BigBird’s global attention can’t be assigned to indi-
vidual tokens in the middle of the sequence, which is important
for the representation of long documents as shown in Caci-
ularu et al. (2021). Second, because pretrained checkpoints
are available for LED, while BigBird-PEGASUS released the
already fine-tuned checkpoints.
5246

Document #1 Wildfires have burned across tens of thousands of acres of
parched terrain in Colorado, spurring thousands of evacuations ...(0.107)..., res-
idents have sought shelter in middle schools, and local officials fear tourists
usually drawn to the region for the summer may not come.
Document #2 ... In Colorado’s southwest, authorities have shuttered the San
Juan National Forest in southwestern Colorado and residents of more than
2,000 homes were forced to evacuate.(0.187) No homes had been destroyed
... “Under current conditions, one abandoned campfire or spark could cause a
catastrophic wildfire, ..., with human life and property, ” said San Juan National
Forest Fire Staff Officer Richard Bustamante...
Document #3 The Buffalo Fire west of Denver is ... Several wildfires in Col-
orado have prompted thousands of home evacuations ...(0.172)... Nearly 1,400
homes have been evacuated in Summit County, Colorado, ...(0.179)... “Under
current conditions, one abandoned campfire or spark could cause a catastrophic
wildfire, ... , with human life and property,” said Richard Bustamante, SJNF
forest fire staff officer ...
Entities with High Frequency
Colorado, 416, Tuesday, Wildfires, San Juan National Forest,...
Figure 3: An example on sentence selection by Princi-
ple vs our Entity Pyramid strategy. Italic text in red is
the sentence with the highest Principle ROUGE scores,
which is thereby chosen by the Principle Strategy. Most
frequent entity ’Colorado’ is shown with blue, followed
by the Pyramid ROUGE scores in parenthesis. The fi-
nal selected sentence by Entity Pyramid strategy is in
italic. which is a better pseudo-summary than the ones
selected by the Principle strategy.
single token [sent-mask] in the input, and train
the model to generate the concatenation of those
sentences as a “pseudo-summary” (Figure 2). This
is close to abstractive summarization because the
model needs to reconstruct the masked sentences
using the information in the rest of the documents.
The key idea is how to select sentences that
best summarize or represent a set of related in-
put documents (which we also call a “cluster”),
not just a single document as in standard GSG.
Zhang et al. (2020) use three strategies - Random,
Lead (first m sentences), and “Principle”. The
“Principle” method computes sentence salience
score based on ROUGE score of each sentence,
si, w.r.t the rest of the document ( D/{si}), i.e.
Score(si) = ROUGE (si, D/{si}). Intuitively, this
assigns a high score to the sentences that have a
high overlap with the other sentences.
However, we argue that a naive extension of
such strategy to multi-document summarization
would be sub-optimal since multi-document inputs
typically include redundant information, and such
strategy would prefer an exact match between sen-
tences, resulting in a selection of less representative
information.
For instance, Figure 3 shows an example of sen-
tences picked by the Principle strategy (Zhang et al.,
2020) vs our Entity Pyramid approach. The figure
shows a cluster containing three news articles dis-
cussing a wildfire happened in Corolado, and the
pseudo-summary of this cluster should be related
to the location, time and consequence of the wild-
fire, but with the Principle strategy, the non-salient
sentences quoting the words from an officer are
assigned the highest score, as the exact same sen-
tence appeared in two out of the three articles. In
comparison, instead of the quoted words, our strat-
egy selects the most representative sentences in the
cluster with high frequency entities.
To address this limitation, we propose a new
masking strategy inspired by the Pyramid Evalua-
tion framework (Nenkova and Passonneau, 2004)
which was originally developed for evaluating sum-
maries with multiple human written references.
Our strategy aims to select sentences that best rep-
resent the entire cluster of input documents.
2.2.1 Entity Pyramid Masking
Pyramid Evaluation The Pyramid Evaluation
method (Nenkova and Passonneau, 2004) is based
on the intuition that relevance of a unit of informa-
tion can be determined by the number of references
(i.e. gold standard) summaries that include it. The
unit of information is called Summary Content Unit
(SCU); words or phrases that represent single facts.
These SCUs are first identified by human annota-
tors in each reference summary, and they receive a
score proportional to the number of reference sum-
maries that contain them. A Pyramid Score for a
candidate summary is then the normalized mean
of the scores of the SCUs that it contains. One
advantage of the Pyramid method is that it directly
assesses the content quality.
Entity Pyramid Masking Inspired by how con-
tent saliency is measured in the Pyramid Evalua-
tion, we hypothesize that a similar idea could be
applied in multi-document summarization to iden-
tify salient sentences for masking. Specifically, for
a cluster with multiple related documents, the more
documents an SCU appears in, the more salient that
information should be to the cluster. Therefore, it
should be considered for inclusion in the pseudo-
summary in our masked sentence generation objec-
tive. However, SCUs in the original Pyramid Eval-
uation are human-annotated, which is not feasible
for large scale pretraining. As a proxy, we explore
leveraging information expressed as named entities,
since they are key building blocks in extracting in-
formation from text about events/objects and the
relationships between their participants/parts (Ju-
rafsky and Martin, 2009). Following the Pyramid
5247