Automated Metrics for Medical Multi-Document Summarization Disagree with
  Human Evaluations
Lucy Lu Wang Yulia Otmakhova Jay DeYoung Thinh Hung Truong Bailey E. Kuehl Erin Bransom Byron C. Wallace
Evaluating multi-document summarization (MDS) quality is difficult. This is
especially true in the case of MDS for biomedical literature reviews, where
models must synthesize contradicting evidence reported across different
documents. Prior work has shown that rather than performing the task, models
may exploit shortcuts that are difficult to detect using standard n-gram
similarity metrics such as ROUGE. Better automated evaluation metrics are
needed, but few resources exist to assess metrics when they are proposed.
Therefore, we introduce a dataset of human-assessed summary quality facets and
pairwise preferences to encourage and support the development of better
automated evaluation methods for literature review MDS. We take advantage of
community submissions to the Multi-document Summarization for Literature Review
(MSLR) shared task to compile a diverse and representative sample of generated
summaries. We analyze how automated summarization evaluation metrics correlate
with lexical features of generated summaries, to other automated metrics
including several we propose in this work, and to aspects of human-assessed
summary quality. We find that not only do automated metrics fail to capture
aspects of quality as assessed by humans, in many cases the system rankings
produced by these metrics are anti-correlated with rankings according to human
annotators.