Element-aware Summarization with Large Language Models: Expert-aligned
  Evaluation and Chain-of-Thought Method
Yiming Wang Zhuosheng Zhang Rui Wang
Automatic summarization generates concise summaries that contain key ideas of
source documents. As the most mainstream datasets for the news sub-domain,
CNN/DailyMail and BBC XSum have been widely used for performance benchmarking.
However, the reference summaries of those datasets turn out to be noisy, mainly
in terms of factual hallucination and information redundancy. To address this
challenge, we first annotate new expert-writing Element-aware test sets
following the "Lasswell Communication Model" proposed by Lasswell (1948),
allowing reference summaries to focus on more fine-grained news elements
objectively and comprehensively. Utilizing the new test sets, we observe the
surprising zero-shot summary ability of LLMs, which addresses the issue of the
inconsistent results between human preference and automatic evaluation metrics
of LLMs' zero-shot summaries in prior work. Further, we propose a Summary
Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step
by step, which helps them integrate more fine-grained details of source
documents into the final summaries that correlate with the human writing
mindset. Experimental results show our method outperforms state-of-the-art
fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two
datasets, respectively. Dataset and code are publicly available at
https://github.com/Alsace08/SumCoT.