On the Role of Summary Content Units in Text Summarization Evaluation
Marcel Nawrath Agnieszka Nowak Tristan Ratz Danilo C. Walenta Juri Opitz Leonardo F. R. Ribeiro João Sedoc Daniel Deutsch Simon Mille Yixin Liu Lining Zhang Sebastian Gehrmann Saad Mahamood Miruna Clinciu Khyathi Chandu Yufang Hou
At the heart of the Pyramid evaluation method for text summarization lie
human written summary content units (SCUs). These SCUs are concise sentences
that decompose a summary into small facts. Such SCUs can be used to judge the
quality of a candidate summary, possibly partially automated via natural
language inference (NLI) systems. Interestingly, with the aim to fully automate
the Pyramid evaluation, Zhang and Bansal (2021) show that SCUs can be
approximated by automatically generated semantic role triplets (STUs). However,
several questions currently lack answers, in particular: i) Are there other
ways of approximating SCUs that can offer advantages? ii) Under which
conditions are SCUs (or their approximations) offering the most value? In this
work, we examine two novel strategies to approximate SCUs: generating SCU
approximations from AMR meaning representations (SMUs) and from large language
models (SGUs), respectively. We find that while STUs and SMUs are competitive,
the best approximation quality is achieved by SGUs. We also show through a
simple sentence-decomposition baseline (SSUs) that SCUs (and their
approximations) offer the most value when ranking short summaries, but may not
help as much when ranking systems or longer summaries.