UMSE: Unified Multi-scenario Summarization Evaluation
Shen Gao Zhitao Yao Chongyang Tao Xiuying Chen Pengjie Ren Zhaochun Ren Zhumin Chen
Summarization quality evaluation is a non-trivial task in text summarization.
Contemporary methods can be mainly categorized into two scenarios: (1)
reference-based: evaluating with human-labeled reference summary; (2)
reference-free: evaluating the summary consistency of the document. Recent
studies mainly focus on one of these scenarios and explore training neural
models built on PLMs to align with human criteria. However, the models from
different scenarios are optimized individually, which may result in sub-optimal
performance since they neglect the shared knowledge across different scenarios.
Besides, designing individual models for each scenario caused inconvenience to
the user. Inspired by this, we propose Unified Multi-scenario Summarization
Evaluation Model (UMSE). More specifically, we propose a perturbed prefix
tuning method to share cross-scenario knowledge between scenarios and use a
self-supervised training paradigm to optimize the model without extra human
labeling. Our UMSE is the first unified summarization evaluation framework
engaged with the ability to be used in three evaluation scenarios. Experimental
results across three typical scenarios on the benchmark dataset SummEval
indicate that our UMSE can achieve comparable performance with several existing
strong methods which are specifically designed for each scenario.