Can GPT models Follow Human Summarization Guidelines? Evaluating ChatGPT
  and GPT-4 for Dialogue Summarization
Yongxin Zhou Fabien Ringeval François Portet
This study explores the capabilities of prompt-driven Large Language Models
(LLMs) like ChatGPT and GPT-4 in adhering to human guidelines for dialogue
summarization. Experiments employed DialogSum (English social conversations)
and DECODA (French call center interactions), testing various prompts:
including prompts from existing literature and those from human summarization
guidelines, as well as a two-step prompt approach. Our findings indicate that
GPT models often produce lengthy summaries and deviate from human summarization
guidelines. However, using human guidelines as an intermediate step shows
promise, outperforming direct word-length constraint prompts in some cases. The
results reveal that GPT models exhibit unique stylistic tendencies in their
summaries. While BERTScores did not dramatically decrease for GPT outputs
suggesting semantic similarity to human references and specialised pre-trained
models, ROUGE scores reveal grammatical and lexical disparities between
GPT-generated and human-written summaries. These findings shed light on the
capabilities and limitations of GPT models in following human instructions for
dialogue summarization.