Breeding Machine Translations: Evolutionary approach to survive and
  thrive in the world of automated evaluation
Josef Jon Ondřej Bojar
We propose a genetic algorithm (GA) based method for modifying n-best lists
produced by a machine translation (MT) system. Our method offers an innovative
approach to improving MT quality and identifying weaknesses in evaluation
metrics. Using common GA operations (mutation and crossover) on a list of
hypotheses in combination with a fitness function (an arbitrary MT metric), we
obtain novel and diverse outputs with high metric scores. With a combination of
multiple MT metrics as the fitness function, the proposed method leads to an
increase in translation quality as measured by other held-out automatic
metrics. With a single metric (including popular ones such as COMET) as the
fitness function, we find blind spots and flaws in the metric. This allows for
an automated search for adversarial examples in an arbitrary metric, without
prior assumptions on the form of such example. As a demonstration of the
method, we create datasets of adversarial examples and use them to show that
reference-free COMET is substantially less robust than the reference-based
version.