Is Summary Useful or Not? An Extrinsic Human Evaluation of Text
  Summaries on Downstream Tasks
Xiao Pu Mingqi Gao Xiaojun Wan
Research on automated text summarization relies heavily on human and
automatic evaluation. While recent work on human evaluation mainly adopted
intrinsic evaluation methods, judging the generic quality of text summaries,
e.g. informativeness and coherence, our work focuses on evaluating the
usefulness of text summaries with extrinsic methods. We carefully design three
different downstream tasks for extrinsic human evaluation of summaries, i.e.,
question answering, text classification and text similarity assessment. We
carry out experiments using system rankings and user behavior data to evaluate
the performance of different summarization models. We find summaries are
particularly useful in tasks that rely on an overall judgment of the text,
while being less effective for question answering tasks. The results show that
summaries generated by fine-tuned models lead to higher consistency in
usefulness across all three tasks, as rankings of fine-tuned summarization
systems are close across downstream tasks according to the proposed extrinsic
metrics. Summaries generated by models in the zero-shot setting, however, are
found to be biased towards the text classification and similarity assessment
tasks, due to its general and less detailed summary style. We further evaluate
the correlation of 14 intrinsic automatic metrics with human criteria and show
that intrinsic automatic metrics perform well in evaluating the usefulness of
summaries in the question-answering task, but are less effective in the other
two tasks. This highlights the limitations of relying solely on intrinsic
automatic metrics in evaluating the performance and usefulness of summaries.