A Knowledge Plug-and-Play Test Bed for Open-domain Dialogue Generation
Xiangci Li Linfeng Song Lifeng Jin Haitao Mi Jessica Ouyang Dong Yu
Knowledge-based, open-domain dialogue generation aims to build chit-chat
systems that talk to humans using mined support knowledge. Many types and
sources of knowledge have previously been shown to be useful as support
knowledge. Even in the era of large language models, response generation
grounded in knowledge retrieved from additional up-to-date sources remains a
practically important approach. While prior work using single-source knowledge
has shown a clear positive correlation between the performances of knowledge
selection and response generation, there are no existing multi-source datasets
for evaluating support knowledge retrieval. Further, prior work has assumed
that the knowledge sources available at test time are the same as during
training. This unrealistic assumption unnecessarily handicaps models, as new
knowledge sources can become available after a model is trained. In this paper,
we present a high-quality benchmark named multi-source Wizard of Wikipedia
(Ms.WoW) for evaluating multi-source dialogue knowledge selection and response
generation. Unlike existing datasets, it contains clean support knowledge,
grounded at the utterance level and partitioned into multiple knowledge
sources. We further propose a new challenge, dialogue knowledge plug-and-play,
which aims to test an already trained dialogue model on using new support
knowledge from previously unseen sources in a zero-shot fashion.