Identifying Factual Inconsistency in Summaries: Towards Effective
  Utilization of Large Language Model
Liyan Xu Zhenlin Su Mo Yu Jin Xu Jinho D. Choi Jie Zhou Fei Liu
Factual inconsistency poses a significant hurdle for the commercial
deployment of abstractive summarizers. Under this Large Language Model (LLM)
era, this work focuses around two important questions: what is the best way to
leverage LLM for factual inconsistency detection, and how could we distill a
smaller LLM with both high efficiency and efficacy? Three zero-shot paradigms
are firstly proposed and evaluated across five diverse datasets: direct
inference on the entire summary or each summary window; entity verification
through question generation and answering. Experiments suggest that LLM itself
is capable to resolve this task train-free under the proper paradigm design,
surpassing strong trained baselines by 2.8% on average. To further promote
practical utility, we then propose training strategies aimed at distilling
smaller open-source LLM that learns to score the entire summary at once with
high accuracy, which outperforms the zero-shot approaches by much larger LLM,
serving as an effective and efficient ready-to-use scorer.