Evaluating Generative Ad Hoc Information Retrieval
Lukas Gienapp Harrisen Scells Niklas Deckers Janek Bevendorff Shuai Wang Johannes Kiesel Shahbaz Syed Maik Fröbe Guido Zuccon Benno Stein Matthias Hagen Martin Potthast
Recent advances in large language models have enabled the development of
viable generative information retrieval systems. A generative retrieval system
returns a grounded generated text in response to an information need instead of
the traditional document ranking. Quantifying the utility of these types of
responses is essential for evaluating generative retrieval systems. As the
established evaluation methodology for ranking-based ad hoc retrieval may seem
unsuitable for generative retrieval, new approaches for reliable, repeatable,
and reproducible experimentation are required. In this paper, we survey the
relevant information retrieval and natural language processing literature,
identify search tasks and system architectures in generative retrieval, develop
a corresponding user model, and study its operationalization. This theoretical
analysis provides a foundation and new insights for the evaluation of
generative ad hoc retrieval systems.