ACES: Translation Accuracy Challenge Sets at WMT 2023
Chantal Amrhein Nikita Moghe Liane Guillou
We benchmark the performance of segmentlevel metrics submitted to WMT 2023
using the ACES Challenge Set (Amrhein et al., 2022). The challenge set consists
of 36K examples representing challenges from 68 phenomena and covering 146
language pairs. The phenomena range from simple perturbations at the
word/character level to more complex errors based on discourse and real-world
knowledge. For each metric, we provide a detailed profile of performance over a
range of error categories as well as an overall ACES-Score for quick
comparison. We also measure the incremental performance of the metrics
submitted to both WMT 2023 and 2022. We find that 1) there is no clear winner
among the metrics submitted to WMT 2023, and 2) performance change between the
2023 and 2022 versions of the metrics is highly variable. Our recommendations
are similar to those from WMT 2022. Metric developers should focus on: building
ensembles of metrics from different design families, developing metrics that
pay more attention to the source and rely less on surface-level overlap, and
carefully determining the influence of multilingual embeddings on MT
evaluation.