CR-LT-KGQA: A Knowledge Graph Question Answering Dataset Requiring
  Commonsense Reasoning and Long-Tail Knowledge
Willis Guo Armin Toroghi Scott Sanner
Knowledge graph question answering (KGQA) is a well-established field that
seeks to provide factual answers to natural language (NL) questions by
leveraging knowledge graphs (KGs). However, existing KGQA datasets suffer from
two significant limitations: (1) no existing KGQA dataset requires commonsense
reasoning to arrive at an answer and (2) existing KGQA datasets focus on
popular entities for which large language models (LLMs) can directly answer
without hallucinating and without leveraging the KG. In this work, we seek a
novel KGQA dataset that supports commonsense reasoning and focuses on long-tail
entities (e.g., non-mainstream and recent entities) where LLMs frequently
hallucinate, and thus create the need for novel methodologies that leverage the
KG for factual and attributable commonsense inference. We create a novel
Commonsense Reasoning (CR) and Long-Tail (LT) KGQA dataset with two subtasks --
question answering and claim verification -- that address both limitations (1)
and (2). We construct CR-LT-KGQA by building extensions to existing reasoning
datasets StrategyQA and CREAK over Wikidata. While existing KGQA methods are
not applicable due to their lack of commonsense inference support, baseline
evaluation of LLMs on CR-LT KGQA demonstrate a high rate of hallucination.
Thus, CR-LT KGQA poses significant challenges for hallucination-prone LLMs,
hence paving the way for future commonsense KGQA research to provide accurate
and factual answers for long-tail entities in the era of LLMs.