LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive
  Summarisation
Jennifer A Bishop Qianqian Xie Sophia Ananiadou
Maintaining factual consistency is a critical issue in abstractive text
summarisation, however, it cannot be assessed by traditional automatic metrics
used for evaluating text summarisation, such as ROUGE scoring. Recent efforts
have been devoted to developing improved metrics for measuring factual
consistency using pre-trained language models, but these metrics have
restrictive token limits, and are therefore not suitable for evaluating long
document text summarisation. Moreover, there is limited research evaluating
whether existing automatic evaluation metrics are fit for purpose when applied
to long document data sets. In this work, we evaluate the efficacy of automatic
metrics at assessing factual consistency in long document text summarisation
and propose a new evaluation framework LongDocFACTScore. This framework allows
metrics to be extended to any length document. This framework outperforms
existing state-of-the-art metrics in its ability to correlate with human
measures of factuality when used to evaluate long document summarisation data
sets. Furthermore, we show LongDocFACTScore has performance comparable to
state-of-the-art metrics when evaluated against human measures of factual
consistency on short document data sets. We make our code and annotated data
publicly available: https://github.com/jbshp/LongDocFACTScore.