How to Choose How to Choose Your Chatbot: A Massively Multi-System
  MultiReference Data Set for Dialog Metric Evaluation
Huda Khayrallah Zuhaib Akhtar Edward Cohen João Sedoc
We release MMSMR, a Massively Multi-System MultiReference dataset to enable
future work on metrics and evaluation for dialog. Automatic metrics for
dialogue evaluation should be robust proxies for human judgments; however, the
verification of robustness is currently far from satisfactory. To quantify the
robustness correlation and understand what is necessary in a test set, we
create and release an 8-reference dialog dataset by extending single-reference
evaluation sets and introduce this new language learning conversation dataset.
We then train 1750 systems and evaluate them on our novel test set and the
DailyDialog dataset. We release the novel test set, and model hyper parameters,
inference outputs, and metric scores for each system on a variety of datasets.