BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust
  Machine Translation Evaluation
Taisiya Glushkova Chrysoula Zerva André F. T. Martins
Although neural-based machine translation evaluation metrics, such as COMET
or BLEURT, have achieved strong correlations with human judgements, they are
sometimes unreliable in detecting certain phenomena that can be considered as
critical errors, such as deviations in entities and numbers. In contrast,
traditional evaluation metrics, such as BLEU or chrF, which measure lexical or
character overlap between translation hypotheses and human references, have
lower correlations with human judgements but are sensitive to such deviations.
In this paper, we investigate several ways of combining the two approaches in
order to increase robustness of state-of-the-art evaluation methods to
translations with critical errors. We show that by using additional information
during training, such as sentence-level features and word-level tags, the
trained metrics improve their capability to penalize translations with specific
troublesome phenomena, which leads to gains in correlation with human judgments
and on recent challenge sets on several language pairs.