Reference Matters: Benchmarking Factual Error Correction for Dialogue
  Summarization with Fine-grained Evaluation Framework
Mingqi Gao Xiaojun Wan Jia Su Zhefeng Wang Baoxing Huai
Factuality is important to dialogue summarization. Factual error correction
(FEC) of model-generated summaries is one way to improve factuality. Current
FEC evaluation that relies on factuality metrics is not reliable and detailed
enough. To address this problem, we are the first to manually annotate a FEC
dataset for dialogue summarization containing 4000 items and propose FERRANTI,
a fine-grained evaluation framework based on reference correction that
automatically evaluates the performance of FEC models on different error
categories. Using this evaluation framework, we conduct sufficient experiments
with FEC approaches under a variety of settings and find the best training
modes and significant differences in the performance of the existing approaches
on different factual error categories.