Granular Change Accuracy: A More Accurate Performance Metric for
  Dialogue State Tracking
Taha Aksu Nancy F. Chen
Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit
three primary limitations. They: i) erroneously presume a uniform distribution
of slots throughout the dialog, ii) neglect to assign partial scores for
individual turns, iii) frequently overestimate or underestimate performance by
repeatedly counting the models' successful or failed predictions. To address
these shortcomings, we introduce a novel metric: Granular Change Accuracy
(GCA). GCA focuses on evaluating the predicted changes in dialogue state over
the entire dialogue history. Benchmarking reveals that GCA effectively reduces
biases arising from distribution uniformity and the positioning of errors
across turns, resulting in a more precise evaluation. Notably, we find that
these biases are particularly pronounced when evaluating few-shot or zero-shot
trained models, becoming even more evident as the model's error rate increases.
Hence, GCA offers significant promise, particularly for assessing models
trained with limited resources. Our GCA implementation is a useful addition to
the pool of DST metrics.