Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English–German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English–Romanian. Neural network based models outperform traditional statistical models for machine translation (MT) BID0 BID9 . However, state-of-the-art neural models are much slower than statistical MT approaches at inference time BID16 . Both model families use autoregressive decoders that operate one step at a time: they generate each token conditioned on the sequence of tokens previously generated. This process is not parallelizable, and, in the case of neural MT models, it is particularly slow because a computationally intensive neural network is used to generate each token.While several recently proposed models avoid recurrence at train time by leveraging convolutions BID6 BID4 or self-attention BID14 as more-parallelizable alternatives to recurrent neural networks (RNNs), use of autoregressive decoding makes it impossible to take full advantage of parallelism during inference.We introduce a non-autoregressive translation model based on the Transformer network BID14 . We modify the encoder of the original Transformer network by adding a module that predicts fertilities, sequences of numbers that form an important component of many traditional machine translation models BID1 . These fertilities are supervised during training and provide the decoder at inference time with a globally consistent plan on which to condition its simultaneously computed outputs.
We introduce the first NMT model with fully parallel decoding, reducing inference latency by 10x. This work proposes non-autoregressive decoder for the encoder-decoder framework in which the decision of generating a word does not depends on the prior decision of generated words This paper describes an approach to decode non-autoregressively for neural machine translation with the possibility of more parallel decoding which can result in a significant speed-up. Proposes the introduction of a set of latent variables to represent the fertility of each source word to make the target sentence generation non-autoregressive