Generating EDU Extracts for Plan-Guided Summary Re-Ranking
Griffin Adams Alexander R. Fabbri Faisal Ladhak Kathleen McKeown Noémie Elhadad
Two-step approaches, in which summary candidates are generated-then-reranked
to return a single summary, can improve ROUGE scores over the standard
single-step approach. Yet, standard decoding methods (i.e., beam search,
nucleus sampling, and diverse beam search) produce candidates with redundant,
and often low quality, content. In this paper, we design a novel method to
generate candidates for re-ranking that addresses these issues. We ground each
candidate abstract on its own unique content plan and generate distinct
plan-guided abstracts using a model's top beam. More concretely, a standard
language model (a BART LM) auto-regressively generates elemental discourse unit
(EDU) content plans with an extractive copy mechanism. The top K beams from the
content plan generator are then used to guide a separate LM, which produces a
single abstractive candidate for each distinct plan. We apply an existing
re-ranker (BRIO) to abstractive candidates generated from our method, as well
as baseline decoding methods. We show large relevance improvements over
previously published methods on widely used single document news article
corpora, with ROUGE-2 F1 gains of 0.88, 2.01, and 0.38 on CNN / Dailymail, NYT,
and Xsum, respectively. A human evaluation on CNN / DM validates these results.
Similarly, on 1k samples from CNN / DM, we show that prompting GPT-3 to follow
EDU plans outperforms sampling-based methods by 1.05 ROUGE-2 F1 points. Code to
generate and realize plans is available at
https://github.com/griff4692/edu-sum.