Did Translation Models Get More Robust Without Anyone Even Noticing?
Ben Peters André F. T. Martins
Neural machine translation (MT) models achieve strong results across a
variety of settings, but it is widely believed that they are highly sensitive
to "noisy" inputs, such as spelling errors, abbreviations, and other formatting
issues. In this paper, we revisit this insight in light of recent multilingual
MT models and large language models (LLMs) applied to machine translation.
Somewhat surprisingly, we show through controlled experiments that these models
are far more robust to many kinds of noise than previous models, even when they
perform similarly on clean data. This is notable because, even though LLMs have
more parameters and more complex training processes than past models, none of
the open ones we consider use any techniques specifically designed to encourage
robustness. Next, we show that similar trends hold for social media translation
experiments -- LLMs are more robust to social media text. We include an
analysis of the circumstances in which source correction techniques can be used
to mitigate the effects of noise. Altogether, we show that robustness to many
types of noise has increased.