DiQAD: A Benchmark Dataset for End-to-End Open-domain Dialogue
  Assessment
Yukun Zhao Lingyong Yan Weiwei Sun Chong Meng Shuaiqiang Wang Zhicong Cheng Zhaochun Ren Dawei Yin
Dialogue assessment plays a critical role in the development of open-domain
dialogue systems. Existing work are uncapable of providing an end-to-end and
human-epistemic assessment dataset, while they only provide sub-metrics like
coherence or the dialogues are conversed between annotators far from real user
settings. In this paper, we release a large-scale dialogue quality assessment
dataset (DiQAD), for automatically assessing open-domain dialogue quality.
Specifically, we (1) establish the assessment criteria based on the dimensions
conforming to human judgements on dialogue qualities, and (2) annotate
large-scale dialogues that conversed between real users based on these
annotation criteria, which contains around 100,000 dialogues. We conduct
several experiments and report the performances of the baselines as the
benchmark on DiQAD. The dataset is openly accessible at
https://github.com/yukunZhao/Dataset_Dialogue_quality_evaluation.