On Monotonic Aggregation for Open-domain QA
Sang-eun Han Yeonseok Jeong Seung-won Hwang Kyungjae Lee
Question answering (QA) is a critical task for speech-based retrieval from
knowledge sources, by sifting only the answers without requiring to read
supporting documents. Specifically, open-domain QA aims to answer user
questions on unrestricted knowledge sources. Ideally, adding a source should
not decrease the accuracy, but we find this property (denoted as
"monotonicity") does not hold for current state-of-the-art methods. We identify
the cause, and based on that we propose Judge-Specialist framework. Our
framework consists of (1) specialist retrievers/readers to cover individual
sources, and (2) judge, a dedicated language model to select the final answer.
Our experiments show that our framework not only ensures monotonicity, but also
outperforms state-of-the-art multi-source QA methods on Natural Questions.
Additionally, we show that our models robustly preserve the monotonicity
against noise from speech recognition. We publicly release our code and
setting.