Hybrid Long Document Summarization using C2F-FAR and ChatGPT: A
  Practical Study
Guang Lu Sylvia B. Larcher Tu Tran
Text summarization is a downstream natural language processing (NLP) task
that challenges the understanding and generation capabilities of language
models. Considerable progress has been made in automatically summarizing short
texts, such as news articles, often leading to satisfactory results. However,
summarizing long documents remains a major challenge. This is due to the
complex contextual information in the text and the lack of open-source
benchmarking datasets and evaluation frameworks that can be used to develop and
test model performance. In this work, we use ChatGPT, the latest breakthrough
in the field of large language models (LLMs), together with the extractive
summarization model C2F-FAR (Coarse-to-Fine Facet-Aware Ranking) to propose a
hybrid extraction and summarization pipeline for long documents such as
business articles and books. We work with the world-renowned company
getAbstract AG and leverage their expertise and experience in professional book
summarization. A practical study has shown that machine-generated summaries can
perform at least as well as human-written summaries when evaluated using
current automated evaluation metrics. However, a closer examination of the
texts generated by ChatGPT through human evaluations has shown that there are
still critical issues in terms of text coherence, faithfulness, and style.
Overall, our results show that the use of ChatGPT is a very promising but not
yet mature approach for summarizing long documents and can at best serve as an
inspiration for human editors. We anticipate that our work will inform NLP
researchers about the extent to which ChatGPT's capabilities for summarizing
long documents overlap with practitioners' needs. Further work is needed to
test the proposed hybrid summarization pipeline, in particular involving GPT-4,
and to propose a new evaluation framework tailored to the task of summarizing
long documents.