Absformer: Transformer-based Model for Unsupervised Multi-Document
  Abstractive Summarization
Mohamed Trabelsi Huseyin Uzunalioglu
Multi-document summarization (MDS) refers to the task of summarizing the text
in multiple documents into a concise summary. The generated summary can save
the time of reading many documents by providing the important content in the
form of a few sentences. Abstractive MDS aims to generate a coherent and fluent
summary for multiple documents using natural language generation techniques. In
this paper, we consider the unsupervised abstractive MDS setting where there
are only documents with no groundtruh summaries provided, and we propose
Absformer, a new Transformer-based method for unsupervised abstractive summary
generation. Our method consists of a first step where we pretrain a
Transformer-based encoder using the masked language modeling (MLM) objective as
the pretraining task in order to cluster the documents into semantically
similar groups; and a second step where we train a Transformer-based decoder to
generate abstractive summaries for the clusters of documents. To our knowledge,
we are the first to successfully incorporate a Transformer-based model to solve
the unsupervised abstractive MDS task. We evaluate our approach using three
real-world datasets from different domains, and we demonstrate both substantial
improvements in terms of evaluation metrics over state-of-the-art
abstractive-based methods, and generalization to datasets from different
domains.