Query Performance Prediction: From Ad-hoc to Conversational Search
Chuan Meng Negar Arabzadeh Mohammad Aliannejadi Maarten de Rijke
Query performance prediction (QPP) is a core task in information retrieval.
The QPP task is to predict the retrieval quality of a search system for a query
without relevance judgments. Research has shown the effectiveness and
usefulness of QPP for ad-hoc search. Recent years have witnessed considerable
progress in conversational search (CS). Effective QPP could help a CS system to
decide an appropriate action to be taken at the next turn. Despite its
potential, QPP for CS has been little studied. We address this research gap by
reproducing and studying the effectiveness of existing QPP methods in the
context of CS. While the task of passage retrieval remains the same in the two
settings, a user query in CS depends on the conversational history, introducing
novel QPP challenges. In particular, we seek to explore to what extent findings
from QPP methods for ad-hoc search generalize to three CS settings: (i)
estimating the retrieval quality of different query rewriting-based retrieval
methods, (ii) estimating the retrieval quality of a conversational dense
retrieval method, and (iii) estimating the retrieval quality for top ranks vs.
deeper-ranked lists. Our findings can be summarized as follows: (i) supervised
QPP methods distinctly outperform unsupervised counterparts only when a
large-scale training set is available; (ii) point-wise supervised QPP methods
outperform their list-wise counterparts in most cases; and (iii) retrieval
score-based unsupervised QPP methods show high effectiveness in assessing the
conversational dense retrieval method, ConvDR.