Redundancy Aware Multi-Reference Based Gainwise Evaluation of Extractive
  Summarization
Mousumi Akter Shubhra Kanti Karmaker Santu
While very popular for evaluating extractive summarization task, the ROUGE
metric has long been criticized for its lack of semantic awareness and its
ignorance about the ranking quality of the summarizer. Thanks to previous
research that has addressed these issues by proposing a gain-based automated
metric called Sem-nCG, which is both rank and semantic aware. However, Sem-nCG
does not consider the amount of redundancy present in a model-generated summary
and currently does not support evaluation with multiple reference summaries.
Unfortunately, addressing both these limitations simultaneously is not trivial.
Therefore, in this paper, we propose a redundancy-aware Sem-nCG metric and
demonstrate how this new metric can be used to evaluate model summaries against
multiple references. We also explore different ways of incorporating redundancy
into the original metric through extensive experiments. Experimental results
demonstrate that the new redundancy-aware metric exhibits a higher correlation
with human judgments than the original Sem-nCG metric for both single and
multiple reference scenarios.