ROUGE-K: Do Your Summaries Have Keywords?
Sotaro Takeshita Simone Paolo Ponzetto Kai Eckert
Keywords, that is, content-relevant words in summaries play an important role
in efficient information conveyance, making it critical to assess if
system-generated summaries contain such informative words during evaluation.
However, existing evaluation metrics for extreme summarization models do not
pay explicit attention to keywords in summaries, leaving developers ignorant of
their presence. To address this issue, we present a keyword-oriented evaluation
metric, dubbed ROUGE-K, which provides a quantitative answer to the question of
-- \textit{How well do summaries include keywords?} Through the lens of this
keyword-aware metric, we surprisingly find that a current strong baseline model
often misses essential information in their summaries. Our analysis reveals
that human annotators indeed find the summaries with more keywords to be more
relevant to the source documents. This is an important yet previously
overlooked aspect in evaluating summarization systems. Finally, to enhance
keyword inclusion, we propose four approaches for incorporating word importance
into a transformer-based model and experimentally show that it enables guiding
models to include more keywords while keeping the overall quality. Our code is
released at https://github.com/sobamchan/rougek.