Identifying Context-Dependent Translations for Evaluation Set Production
Rachel Wicks Matt Post
A major impediment to the transition to context-aware machine translation is
the absence of good evaluation metrics and test sets. Sentences that require
context to be translated correctly are rare in test sets, reducing the utility
of standard corpus-level metrics such as COMET or BLEU. On the other hand,
datasets that annotate such sentences are also rare, small in scale, and
available for only a few languages. To address this, we modernize, generalize,
and extend previous annotation pipelines to produce CTXPRO, a tool that
identifies subsets of parallel documents containing sentences that require
context to correctly translate five phenomena: gender, formality, and animacy
for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input
to the pipeline is a set of hand-crafted, per-language, linguistically-informed
rules that select contextual sentence pairs using coreference, part-of-speech,
and morphological features provided by state-of-the-art tools. We apply this
pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT,
and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its
performance using both overlap with previous work and its ability to
discriminate a contextual MT system from a sentence-based one. We release the
CTXPRO pipeline and data as open source.