Towards Explainable Evaluation Metrics for Machine Translation
Christoph Leiter Piyawat Lertvittayakumjorn Marina Fomicheva Wei Zhao Yang Gao Steffen Eger
Unlike classical lexical overlap metrics such as BLEU, most current
evaluation metrics for machine translation (for example, COMET or BERTScore)
are based on black-box large language models. They often achieve strong
correlations with human judgments, but recent research indicates that the
lower-quality classical metrics remain dominant, one of the potential reasons
being that their decision processes are more transparent. To foster more
widespread acceptance of novel high-quality metrics, explainability thus
becomes crucial. In this concept paper, we identify key properties as well as
key goals of explainable machine translation metrics and provide a
comprehensive synthesis of recent techniques, relating them to our established
goals and properties. In this context, we also discuss the latest
state-of-the-art approaches to explainable metrics based on generative models
such as ChatGPT and GPT4. Finally, we contribute a vision of next-generation
approaches, including natural language explanations. We hope that our work can
help catalyze and guide future research on explainable evaluation metrics and,
mediately, also contribute to better and more transparent machine translation
systems.