MISMATCH: Fine-grained Evaluation of Machine-generated Text with
  Mismatch Error Types
Keerthiram Murugesan Sarathkrishna Swaminathan Soham Dan Subhajit Chaudhury Chulaka Gunasekara Maxwell Crouse Diwakar Mahajan Ibrahim Abdelaziz Achille Fokoue Pavan Kapanipathi Salim Roukos Alexander Gray
With the growing interest in large language models, the need for evaluating
the quality of machine text compared to reference (typically human-generated)
text has become focal attention. Most recent works focus either on
task-specific evaluation metrics or study the properties of machine-generated
text captured by the existing metrics. In this work, we propose a new
evaluation scheme to model human judgments in 7 NLP tasks, based on the
fine-grained mismatches between a pair of texts. Inspired by the recent efforts
in several NLP tasks for fine-grained evaluation, we introduce a set of 13
mismatch error types such as spatial/geographic errors, entity errors, etc, to
guide the model for better prediction of human judgments. We propose a neural
framework for evaluating machine texts that uses these mismatch error types as
auxiliary tasks and re-purposes the existing single-number evaluation metrics
as additional scalar features, in addition to textual features extracted from
the machine and reference texts. Our experiments reveal key insights about the
existing metrics via the mismatch errors. We show that the mismatch errors
between the sentence pairs on the held-out datasets from 7 NLP tasks align well
with the human evaluation.