Can ChatGPT Rival Neural Machine Translation? A Comparative Study
Zhaokun Jiang Ziyin Zhang
Inspired by the increasing interest in leveraging large language models for
translation, this paper evaluates the capabilities of large language models
(LLMs) represented by ChatGPT in comparison to the mainstream neural machine
translation (NMT) engines in translating Chinese diplomatic texts into English.
Specifically, we examine the translation quality of ChatGPT and NMT engines as
measured by four automated metrics and human evaluation based on an
error-typology and six analytic rubrics. Our findings show that automated
metrics yield similar results for ChatGPT under different prompts and NMT
systems, while human annotators tend to assign noticeably higher scores to
ChatGPT when it is provided an example or contextual information about the
translation task. Pairwise correlation between automated metrics and dimensions
of human evaluation produces weak and non-significant results, suggesting the
divergence between the two methods of translation quality assessment. These
findings provide valuable insights into the potential of ChatGPT as a capable
machine translator, and the influence of prompt engineering on its performance.