The Inside Story: Towards Better Understanding of Machine Translation
  Neural Evaluation Metrics
Ricardo Rei Nuno M. Guerreiro Marcos Treviso Luisa Coheur Alon Lavie André F. T. Martins
Neural metrics for machine translation evaluation, such as COMET, exhibit
significant improvements in their correlation with human judgments, as compared
to traditional metrics based on lexical overlap, such as BLEU. Yet, neural
metrics are, to a great extent, "black boxes" returning a single sentence-level
score without transparency about the decision-making process. In this work, we
develop and compare several neural explainability methods and demonstrate their
effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our
study reveals that these metrics leverage token-level information that can be
directly attributed to translation errors, as assessed through comparison of
token-level neural saliency maps with Multidimensional Quality Metrics (MQM)
annotations and with synthetically-generated critical translation errors. To
ease future research, we release our code at:
https://github.com/Unbabel/COMET/tree/explainable-metrics.