WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search
  Results with Citations
Haolin Deng Chang Wang Xin Li Dezhang Yuan Junlang Zhan Tianhua Zhou Jin Ma Jun Gao Ruifeng Xu
Enhancing the attribution in large language models (LLMs) is a crucial task.
One feasible approach is to enable LLMs to cite external sources that support
their generations. However, existing datasets and evaluation methods in this
domain still exhibit notable limitations. In this work, we formulate the task
of attributed query-focused summarization (AQFS) and present WebCiteS, a
Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS
derives from real-world user queries and web search results, offering a
valuable resource for model training and evaluation. Prior works in attribution
evaluation do not differentiate between groundedness errors and citation
errors. They also fall short in automatically verifying sentences that draw
partial support from multiple sources. We tackle these issues by developing
detailed metrics and enabling the automatic evaluator to decompose the
sentences into sub-claims for fine-grained verification. Our comprehensive
evaluation of both open-source and proprietary models on WebCiteS highlights
the challenge LLMs face in correctly citing sources, underscoring the necessity
for further improvement. The dataset and code will be open-sourced to
facilitate further research in this crucial field.