Character-based neural machine translation (NMT) models alleviate out-of-vocabulary issues, learn morphology, and move us closer to completely end-to-end translation systems.   Unfortunately, they are also very brittle and easily falter when presented with noisy data. In this paper, we confront NMT models with synthetic and natural sources of noise. We find that state-of-the-art models fail to translate even moderately noisy texts that humans have no trouble comprehending. We explore two approaches to increase model robustness: structure-invariant word representations and robust training on noisy texts. We find that a model based on a character convolutional neural network is able to simultaneously learn representations robust to multiple kinds of noise. Humans have surprisingly robust language processing systems that can easily overcome typos, misspellings, and the complete omission of letters when reading BID34 . A particularly extreme and comical exploitation of our robustness came years ago in the form of a popular meme: "Aoccdrnig to a rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae." A person's ability to read this text comes as no surprise to the psychology literature. BID38 found that this robustness extends to audio as well. They experimented with playing parts of audio transcripts backwards and found that it did not affect comprehension. BID35 found that in noisier settings reading comprehension only slowed by 11%. BID27 found that the common case of swapping letters could often go unnoticed by the reader. The exact mechanisms and limitations of our understanding system are unknown. There is some evidence that we rely on word shape BID26 , that we can switch between whole word recognition and piecing together words from letters BID36 BID33 , and there appears to be no evidence that the first and last letter positions are required to stay constant for comprehension. 1 In stark contrast, neural machine translation (NMT) systems, despite their pervasive use, are immensely brittle. For instance, Google Translate produces the following unintelligible translation for a German version of the above meme: 2 "After being stubbornly defiant, it is clear to kenie Rlloe in which Reiehnfogle is advancing the boulders in a Wrot that is integral to Sahce, as the utterance and the lukewarm boorstbaen stmimt."While typos and noise are not new to NLP, our systems are rarely trained to explicitly address them, as we instead hope that the relevant noise will occur in the training data.Despite these weaknesses, the move to character-based NMT is important. It helps us tackle the long tailed distribution of out-of-vocabulary words in natural language, as well as reduce computation load of dealing with large word embedding matrices. NMT models based on characters and other sub-word units are able to extract stem and morphological information to generalize to unseen words and conjugations. They perform very well in practice on a range of languages BID44 BID53 . In many cases, these models actually discover an impressive amount of morphological information about a language BID2 . Unfortunately, training (and testing) on clean data makes models brittle and, arguably, unfit for broad deployment. Figure 1 shows how the performance of two state-of-the-art NMT systems degrades when translating German to English as a function of the percent of German words modified. Here we show three types of noise: 1) Random permutation of the word, 2) Swapping a pair of adjacent letters, and 3) Natural human errors. We discuss these types of noise and others in depth in section 4.2. The important thing to note is that even small amounts of noise lead to substantial drops in performance. Figure 1: Degradation of Nematus and char2char BID21 performance as noise increases.To address these trends and investigate the effects of noise on NMT, we explore two simple strategies for increasing model robustness: using structure-invariant representations and robust training on noisy data, a form of adversarial training BID49 BID14 . We find that a character CNN representation trained on an ensemble of noise types is robust to all kinds of noise. We shed some light on the model ability to learn robust representations to multiple types of noise, and point to remaining difficulties in handling natural noise. Our goal is two fold: 1) initiate a conversation on robust training and modeling techniques in NMT, and 2) promote the creation of better and more linguistically accurate artificial noise to be applied to new languages and tasks.
CharNMT is brittle This paper investigates the impact of character-level noise on 4 different neural machine translation systems This paper empirically investigates the performance of character-level NMT systems in the face of character-level noise, both synthesized and natural. This paper investigates the impact of noisy input on Machine Translation and tests ways to make NMT models more robust