Discourse Centric Evaluation of Machine Translation with a Densely
  Annotated Parallel Corpus
Yuchen Eleanor Jiang Tianyu Liu Shuming Ma Dongdong Zhang Mrinmaya Sachan Ryan Cotterell
Several recent papers claim human parity at sentence-level Machine
Translation (MT), especially in high-resource languages. Thus, in response, the
MT community has, in part, shifted its focus to document-level translation.
Translating documents requires a deeper understanding of the structure and
meaning of text, which is often captured by various kinds of discourse
phenomena such as consistency, coherence, and cohesion. However, this renders
conventional sentence-level MT evaluation benchmarks inadequate for evaluating
the performance of context-aware MT systems. This paper presents a new dataset
with rich discourse annotations, built upon the large-scale parallel corpus BWB
introduced in Jiang et al. (2022). The new BWB annotation introduces four extra
evaluation aspects, i.e., entity, terminology, coreference, and quotation,
covering 15,095 entity mentions in both languages. Using these annotations, we
systematically investigate the similarities and differences between the
discourse structures of source and target languages, and the challenges they
pose to MT. We discover that MT outputs differ fundamentally from human
translations in terms of their latent discourse structures. This gives us a new
perspective on the challenges and opportunities in document-level MT. We make
our resource publicly available to spur future research in document-level MT
and the generalization to other language translation tasks.