HAGRID: A Human-LLM Collaborative Dataset for Generative
  Information-Seeking with Attribution
Ehsan Kamalloo Aref Jafari Xinyu Zhang Nandan Thakur Jimmy Lin
The rise of large language models (LLMs) had a transformative impact on
search, ushering in a new era of search engines that are capable of generating
search results in natural language text, imbued with citations for supporting
sources. Building generative information-seeking models demands openly
accessible datasets, which currently remain lacking. In this paper, we
introduce a new dataset, HAGRID (Human-in-the-loop Attributable Generative
Retrieval for Information-seeking Dataset) for building end-to-end generative
information-seeking models that are capable of retrieving candidate quotes and
generating attributed explanations. Unlike recent efforts that focus on human
evaluation of black-box proprietary search engines, we built our dataset atop
the English subset of MIRACL, a publicly available information retrieval
dataset. HAGRID is constructed based on human and LLM collaboration. We first
automatically collect attributed explanations that follow an in-context
citation style using an LLM, i.e. GPT-3.5. Next, we ask human annotators to
evaluate the LLM explanations based on two criteria: informativeness and
attributability. HAGRID serves as a catalyst for the development of
information-seeking models with better attribution capabilities.