Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Tom Kocmi Vilém Zouhar Christian Federmann Matt Post
Ten years ago a single metric, BLEU, governed progress in machine translation
research. For better or worse, there is no such consensus today, and
consequently it is difficult for researchers to develop and retain the kinds of
heuristic intuitions about metric deltas that drove earlier research and
deployment decisions. This paper investigates the "dynamic range" of a number
of modern metrics in an effort to provide a collective understanding of the
meaning of differences in scores both within and among metrics; in other words,
we ask what point difference X in metric Y is required between two systems for
humans to notice? We conduct our evaluation on a new large dataset, ToShip23,
using it to discover deltas at which metrics achieve system-level differences
that are meaningful to humans, which we measure by pairwise system accuracy. We
additionally show that this method of establishing delta-accuracy is more
stable than the standard use of statistical p-values in regards to testset
size. Where data size permits, we also explore the effect of metric deltas and
accuracy across finer-grained features such as translation direction, domain,
and system closeness.