Machine translation is an important real-world application, and neural network-based AutoRegressive Translation (ART) models have achieved very promising accuracy. Due to the unparallelizable nature of the autoregressive factorization, ART models have to generate tokens one by one during decoding and thus suffer from high inference latency. Recently, Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time. However, they could only achieve inferior accuracy compared with ART models. To improve the accuracy of NART models, in this paper, we propose to leverage the hints from a well-trained ART model to train the NART model. We define two hints for the machine translation task: hints from hidden states and hints from word alignments, and use such hints to regularize the optimization of NART models. Experimental results show that the NART model trained with hints could achieve significantly better translation performance than previous NART models on several tasks. In particular, for the WMT14 En-De and De-En task, we obtain BLEU scores of 25.20 and 29.52 respectively, which largely outperforms the previous non-autoregressive baselines. It is even comparable to a strong LSTM-based ART model (24.60 on WMT14 En-De), but one order of magnitude faster in inference. Neural machine translation has attracted much attention from the research community BID1 BID13 BID5 and has been gradually adopted by industry in the past several years BID22 . Despite the huge variety of model architectures BID1 BID6 BID19 , given a source sentence x = (x 1 , ..., x Tx ) and a target sentence y = (y 1 , ..., y Ty ), most neural machine translation models decompose and estimate the conditional probability P (y|x) in an universal autoregressive manner: P (y|x) = Π Ty t=1 P (y t |y <t , x),where y <t represents the first t − 1 words of y. During inference, given an input sentence, those models generate the translation results sequentially, token by token from left to right. We call all such models AutoRegressive neural machine Translation (ART) models. A state-of-the-art ART model, Transformer BID19 , is shown in the left part of Figure 1 .A well-known limitation of the ART models is that the inference process can hardly be parallelized, and the inference time is linear with respect to the length of the target sequence. As a result, the ART models suffer from long inference time BID22 , which is sometimes unaffordable for industrial applications. Consequently , people start to develop Non-AutoRegressive neural machine Translation (NART) models to speed up the inference process BID7 BID15 . These models use the general encoder-decoder framework: the encoder takes a source sentence x as input and generates a set of contextual embeddings and predicted length T y ; conditioned on the contextual embeddings, the decoder takes a transformed copy of x as input and predicts the target tokens at all the positions independently in parallel according to the following decomposition:P (y|x, T y ) = Π Ty t=1 P (y t |T y , x).While the NART models achieve significant speedup during inference BID7 , their accuracy is considerably lower than their ART counterpart. Most of the previous works attribute the poor performance to this unavoidable conditional independence assumption of the NART model. To tackle this issue , they try to improve the expressiveness and accuracy of the decoder input in different ways: BID7 introduce fertilities from statistical machine translation models into the NART models, BID15 base the decoding process of their proposed model on an iterative refinement process, and take a step further to embed an autoregressive submodule that consists of discrete latent variables into their model. Although such methods provide better expressiveness of decoder inputs and improve the final translation accuracy, the inference speed of these models will be hurt due to the overhead of the introduced modules, which contradicts with the original purpose of introducing the NART models, i.e., to parallelize and speed up neural machine translation models.Different from previous works that develop new submodules for decoder input, we improve the translation model from another perspective. We aim to provide more guided signals during optimization. That is, we do not introduce any new prediction submodule but introduce better regularization. The reason we tackle the problem from this perspective lies in two points: First, the encoder input (source words) contains all semantic information for translation, and the decoder input in the NART model can be considered as a middle layer between input and output. It is not clear how much gain can be achieved by developing a sophisticated submodule for a middle layer in a deep neural network. Second, the encoder-decoder-based NART model is already over-parameterized.We believe that such neural network still has great ability and space to be better optimized if we can provide it with stronger and richer signals, for example, from a much better ART model: Once we have a well-trained ART model, we actually know rich information about the contexts to make the prediction at each time step and the natural word alignments between bilingual sentences. All the information could be invaluable towards the improved training of a NART model.To well leverage an ART model, we use the hint-based training framework BID17 BID3 , in which the information from hidden layers of teacher model (referred as hints) are used to guide the training process of a student model. However, hint-based training was developed for image classification models and it is challenging to define and use hints for translation. First, the translation model is composed of stacked encoder layers, attention layers, and stacked decoder layers. It is not clear how to define hints in such an encoder-decoder framework. Second, the NART and ART models are of different architectures on the decoding stage. It is not obvious how to leverage hints from the teacher to the training of student with a different architecture. We find that directly applying hints used in the classification tasks fails. In this paper, we first investigate the causes of the bad performance of the NART model, and then define hints targeting to solve the problems. According to our empirical study, we find that the hidden states of the NART model differ from the ART model: the positions where the NART model outputs incoherent tokens will have very high hidden states similarity. Also, the attention distributions of the NART model are more ambiguous than those of ART model. Based on these observations, we design two kinds of hints from the hidden states and attention distributions of the ART model, to help the training of the NART model.We have conducted experiments on the widely used WMT14 English-to-German/German-toEnglish (En-De/De-En) task and IWSLT14 German-to-English task. For WMT14 En-De task, our proposed method achieves a BLEU score of 25.20 which significantly outperforms the nonautoregressive baseline models and is even comparable to a strong ART baseline, Google's LSTMbased translation model (24.60 BID22 ). For WMT14 De-En task, we also achieve significant performance gains, reaching 29.52 in terms of BLEU.2 RELATED WORKS 2.1 AUTOREGRESSIVE TRANSLATION Given a sentence x = (x 1 , . . . , x Tx ) from the source language, the straight-forward way for translation is to generate the words in the target language y = (y 1 , . . . , y Ty ) one by one from left to right. This is also known as the autoregressive factorization in which the joint probability is decomposed into a chain of conditional probabilities, as in the Eqn. (1). Deep neural networks are widely used to model such conditional probabilities based on the encoder-decoder framework. The encoder takes the source tokens (x 1 , . . . , x Tx ) as input and encodes x into a set of context states c = (c 1 , . . . , c Tx ). The decoder takes c and subsequence y <t as input and estimates P (y t |y <t , c) according to some parametric function. Non-autoregressive translation (NART) models have suffered from low-quality translation results.In this paper, we proposed to use hints from well-trained autoregressive translation (ART) models to enhance the training of NART models. Our results on WMT14 En-De and De-En significantly outperform previous NART baselines, and achieve comparable accuracy to an LSTM-based ART model, with one order of magnitude faster in inference. In the future, we will focus on designing new architectures and new training methods for NART models to achieve comparable accuracy as the state-of-the-art ART models such as Transformer.
We develop a training algorithm for non-autoregressive machine translation models, achieving comparable accuracy to strong autoregressive baselines, but one order of magnitude faster in inference.   Distills knowledge from intermediary hidden states and attention weights to improve non-autoregressive neural machine translation. Proposes to leverage well trained autoregressive model to inform the hidden states and the word alignment of non-autoregressive Neural Machine Translation models.