Towards General Error Diagnosis via Behavioral Testing in Machine
  Translation
Junjie Wu Lemao Liu Dit-Yan Yeung
Behavioral testing offers a crucial means of diagnosing linguistic errors and
assessing capabilities of NLP models. However, applying behavioral testing to
machine translation (MT) systems is challenging as it generally requires human
efforts to craft references for evaluating the translation quality of such
systems on newly generated test cases. Existing works in behavioral testing of
MT systems circumvent this by evaluating translation quality without
references, but this restricts diagnosis to specific types of errors, such as
incorrect translation of single numeric or currency words. In order to diagnose
general errors, this paper proposes a new Bilingual Translation Pair Generation
based Behavior Testing (BTPGBT) framework for conducting behavioral testing of
MT systems. The core idea of BTPGBT is to employ a novel bilingual translation
pair generation (BTPG) approach that automates the construction of high-quality
test cases and their pseudoreferences. Experimental results on various MT
systems demonstrate that BTPGBT could provide comprehensive and accurate
behavioral testing results for general error diagnosis, which further leads to
several insightful findings. Our code and data are available at https:
//github.com/wujunjie1998/BTPGBT.