PELMS: Pre-training for Effective Low-Shot Multi-Document Summarization
Joseph J. Peper Wenzhao Qiu Lu Wang
We investigate pre-training techniques for abstractive multi-document
summarization (MDS), which is much less studied than summarizing single
documents. Though recent work has demonstrated the effectiveness of
highlighting information salience for pre-training strategy design, it
struggles to generate abstractive and reflective summaries, which are critical
properties for MDS. To this end, we present PELMS, a pre-trained model that
uses objectives based on semantic coherence heuristics and faithfulness
constraints with un-labeled multi-document inputs, to promote the generation of
concise, fluent, and faithful summaries. To support the training of PELMS, we
compile MultiPT, a multi-document pre-training corpus containing over 93
million documents to form more than 3 million unlabeled topic-centric document
clusters, covering diverse genres such as product reviews, news, and general
knowledge. We perform extensive evaluation of PELMS in low-shot settings on a
wide range of MDS datasets. Our approach consistently outperforms competitive
comparisons with respect to overall informativeness, abstractiveness,
coherence, and faithfulness.