OmniDialog: An Omnipotent Pre-training Model for Task-Oriented Dialogue
  System
Mingtao Yang See-Kiong Ng Jinlan Fu
Pre-trained conversation models (PCMs) have demonstrated remarkable results
in task-oriented dialogue (TOD) systems. Many PCMs focus predominantly on
dialogue management tasks like dialogue state tracking, dialogue generation
tasks like response generation, or both. However, the existing PCMs seldom
consider dialogue comprehension tasks, such as dialogue question answering and
summarization tasks. These tasks allow PCMs to glean dialogue context from
various angles. This observation naturally raises the question: Can the
performance of downstream dialogue tasks be enhanced if a PCM is pre-trained on
dialogue management, generation, and comprehension tasks?
  To investigate this, we proposed an Omnipotent Dialogue pre-training model
(OmniDialog). It unifies these three dialogue tasks into a monolithic framework
by multi-task learning, fostering inter-task communication. The pre-training
corpus of OmniDialog spans $\mathbf{7}$ dialogue-focused tasks, drawing from
$\mathbf{15}$ datasets and encompassing over $\mathbf{3.2}$ million dialogue
utterances. To our knowledge, OmniDialog is a pioneering PCM pre-trained across
dialogue management, generation, and comprehension domains. We evaluated its
performance across four tasks: dialogue summarization, end-to-end dialogue
modeling, dialogue state tracking, and intent classification. The results
underscore its efficacy in domain transfer learning, low-resource, and
full-dataset scenarios. Furthermore, to glean a nuanced understanding of
OmniDialog's strengths and potential pitfalls, we designed a fine-grained
analysis framework for dialogue-centric tasks. Experimental results show that
the OmniDialog is good at hard samples, such as long dialogues and lengthy
responses.