Robustness Tests for Automatic Machine Translation Metrics with
  Adversarial Attacks
Yichen Huang Timothy Baldwin
We investigate MT evaluation metric performance on adversarially-synthesized
texts, to shed light on metric robustness. We experiment with word- and
character-level attacks on three popular machine translation metrics:
BERTScore, BLEURT, and COMET. Our human experiments validate that automatic
metrics tend to overpenalize adversarially-degraded translations. We also
identify inconsistencies in BERTScore ratings, where it judges the original
sentence and the adversarially-degraded one as similar, while judging the
degraded translation as notably worse than the original with respect to the
reference. We identify patterns of brittleness that motivate more robust metric
development.