GUMsley: Evaluating Entity Salience in Summarization for 12 English
  Genres
Jessica Lin Amir Zeldes
As NLP models become increasingly capable of understanding documents in terms
of coherent entities rather than strings, obtaining the most salient entities
for each document is not only an important end task in itself but also vital
for Information Retrieval (IR) and other downstream applications such as
controllable summarization. In this paper, we present and evaluate GUMsley, the
first entity salience dataset covering all named and non-named salient entities
for 12 genres of English text, aligned with entity types, Wikification links
and full coreference resolution annotations. We promote a strict definition of
salience using human summaries and demonstrate high inter-annotator agreement
for salience based on whether a source entity is mentioned in the summary. Our
evaluation shows poor performance by pre-trained SOTA summarization models and
zero-shot LLM prompting in capturing salient entities in generated summaries.
We also show that predicting or providing salient entities to several model
architectures enhances performance and helps derive higher-quality summaries by
alleviating the entity hallucination problem in existing abstractive
summarization.