MDDial: A Multi-turn Differential Diagnosis Dialogue Dataset with
  Reliability Evaluation
Srija Macherla Man Luo Mihir Parmar Chitta Baral
Dialogue systems for Automatic Differential Diagnosis (ADD) have a wide range
of real-life applications. These dialogue systems are promising for providing
easy access and reducing medical costs. Building end-to-end ADD dialogue
systems requires dialogue training datasets. However, to the best of our
knowledge, there is no publicly available ADD dialogue dataset in English
(although non-English datasets exist). Driven by this, we introduce MDDial, the
first differential diagnosis dialogue dataset in English which can aid to build
and evaluate end-to-end ADD dialogue systems. Additionally, earlier studies
present the accuracy of diagnosis and symptoms either individually or as a
combined weighted score. This method overlooks the connection between the
symptoms and the diagnosis. We introduce a unified score for the ADD system
that takes into account the interplay between symptoms and diagnosis. This
score also indicates the system's reliability. To the end, we train two
moderate-size of language models on MDDial. Our experiments suggest that while
these language models can perform well on many natural language understanding
tasks, including dialogue tasks in the general domain, they struggle to relate
relevant symptoms and disease and thus have poor performance on MDDial. MDDial
will be released publicly to aid the study of ADD dialogue research.