Desiderata for the Context Use of Question Answering Systems
Sagi Shaier Lawrence E Hunter Katharina von der Wense
Prior work has uncovered a set of common problems in state-of-the-art
context-based question answering (QA) systems: a lack of attention to the
context when the latter conflicts with a model's parametric knowledge, little
robustness to noise, and a lack of consistency with their answers. However,
most prior work focus on one or two of those problems in isolation, which makes
it difficult to see trends across them. We aim to close this gap, by first
outlining a set of -- previously discussed as well as novel -- desiderata for
QA models. We then survey relevant analysis and methods papers to provide an
overview of the state of the field. The second part of our work presents
experiments where we evaluate 15 QA systems on 5 datasets according to all
desiderata at once. We find many novel trends, including (1) systems that are
less susceptible to noise are not necessarily more consistent with their
answers when given irrelevant context; (2) most systems that are more
susceptible to noise are more likely to correctly answer according to a context
that conflicts with their parametric knowledge; and (3) the combination of
conflicting knowledge and noise can reduce system performance by up to 96%. As
such, our desiderata help increase our understanding of how these models work
and reveal potential avenues for improvements.