Fine-grained and Explainable Factuality Evaluation for Multimodal
  Summarization
Liqiang Jing Jingxuan Zuo Yue Zhang
Multimodal summarization aims to generate a concise summary based on the
input text and image. However, the existing methods potentially suffer from
unfactual output. To evaluate the factuality of multimodal summarization
models, we propose two fine-grained and explainable evaluation frameworks
(FALLACIOUS) for different application scenarios, i.e. reference-based
factuality evaluation framework and reference-free factuality evaluation
framework. Notably, the reference-free factuality evaluation framework doesn't
need ground truth and hence it has a wider application scenario. To evaluate
the effectiveness of the proposed frameworks, we compute the correlation
between our frameworks and the other metrics. The experimental results show the
effectiveness of our proposed method. We will release our code and dataset via
github.