Benchmarking Large Language Models in Complex Question Answering
  Attribution using Knowledge Graphs
Nan Hu Jiaoyan Chen Yike Wu Guilin Qi Sheng Bi Tongtong Wu Jeff Z. Pan
The attribution of question answering is to provide citations for supporting
generated statements, and has attracted wide research attention. The current
methods for automatically evaluating the attribution, which are often based on
Large Language Models (LLMs), are still inadequate, particularly in recognizing
subtle differences between attributions, and complex relationships between
citations and statements. To compare these attribution evaluation methods and
develop new ones, we introduce a set of fine-grained categories (i.e.,
supportive, insufficient, contradictory and irrelevant) for measuring the
attribution, and develop a Complex Attributed Question Answering (CAQA)
benchmark by leveraging knowledge graphs (KGs) for automatically generating
attributions of different categories to question-answer pairs. Our analysis
reveals that existing evaluators perform poorly under fine-grained attribution
settings and exhibit weaknesses in complex citation-statement reasoning. Our
CAQA benchmark, validated with human annotations, emerges as a promising tool
for selecting and developing LLM attribution evaluators.