ChatGPT MT: Competitive for High- (but not Low-) Resource Languages
Nathaniel R. Robinson Perez Ogayo David R. Mortensen Graham Neubig
Large language models (LLMs) implicitly learn to perform a range of language
tasks, including machine translation (MT). Previous studies explore aspects of
LLMs' MT capabilities. However, there exist a wide variety of languages for
which recent LLM MT performance has never before been evaluated. Without
published experimental evidence on the matter, it is difficult for speakers of
the world's diverse languages to know how and whether they can use LLMs for
their languages. We present the first experimental evidence for an expansive
set of 204 languages, along with MT cost analysis, using the FLORES-200
benchmark. Trends reveal that GPT models approach or exceed traditional MT
model performance for some high-resource languages (HRLs) but consistently lag
for low-resource languages (LRLs), under-performing traditional MT for 84.1% of
languages we covered. Our analysis reveals that a language's resource level is
the most important feature in determining ChatGPT's relative ability to
translate it, and suggests that ChatGPT is especially disadvantaged for LRLs
and African languages.