Most state-of-the-art neural machine translation systems, despite being different
 in architectural skeletons (e.g., recurrence, convolutional), share an indispensable
 feature: the Attention. However, most existing attention methods are token-based
 and ignore the importance of phrasal alignments, the key ingredient for the success
 of phrase-based statistical machine translation. In this paper, we propose
 novel phrase-based attention methods to model n-grams of tokens as attention
 entities. We incorporate our phrase-based attentions into the recently proposed
 Transformer network, and demonstrate that our approach yields improvements of
 1.3 BLEU for English-to-German and 0.5 BLEU for German-to-English translation
 tasks, and 1.75 and 1.35 BLEU points in English-to-Russian and Russian-to-English translation tasks 
 on WMT newstest2014 using WMT’16 training data.
 Neural Machine Translation (NMT) has established breakthroughs in many different translation tasks, and has quickly become the standard approach to machine translation. NMT offers a simple encoder-decoder architecture that is trained end-to-end. Most NMT models (except a few like BID5 and BID4 ) possess attention mechanisms to perform alignments of the target tokens to the source tokens. The attention module plays a role analogous to the word alignment model in Statistical Machine Translation or SMT BID9 . In fact, the Transformer network introduced recently by BID19 achieves state-of-the-art performance in both speed and BLEU scores BID12 by using only attention modules.On the other hand, phrasal interpretation is an important aspect for many language processing tasks, and forms the basis of Phrase-Based Machine Translation BID9 . Phrasal alignments BID10 can model one-to-one, one-to-many, many-to-one, and many-to-many relations between source and target tokens, and use local context for translation. They are also robust to non-compositional phrases. Despite the advantages, the concept of phrasal attentions has largely been neglected in NMT, as most NMT models generate translations token-by-token autoregressively, and use the token-based attention method which is order invariant. Therefore, the intuition of phrase-based translation is vague in existing NMT systems that solely depend on the underlying neural architectures (recurrent, convolutional, or self-attention) to incorporate contextual information. However, the information aggregation strategies employed by the underlying neural architectures provide context-relevant clues only to represent the current token, and do not explicitly model phrasal alignments. We argue that having an explicit inductive bias for phrases and phrasal alignments is necessary for NMT to exploit the strong correlation between source and target phrases.In this paper, we propose phrase-based attention methods for phrase-level alignments in NMT. Specifically, we propose two novel phrase-based attentions, namely CONVKV and QUERYK, designed to assign attention scores directly to phrases in the source and compute phrase-level attention vector for the target. We also introduce three new attention structures, which apply these methods to conduct phrasal alignments. Our homogeneous and heterogeneous attention structures perform token-to-token and token-to-phrase mappings, while the interleaved heterogeneous attention structure models all token-to-token, token-to-phrase, phrase-to-token, and phrase-to-phrase alignments.To show the effectiveness of our approach, we apply our phrase-based attention methods to all multi-head attention layers of the Transformer. Our experiments on WMT'14 translation tasks show improvements of up to 1.3 and 0.5 BLEU points for English-to-German and German-to-English respectively, and up to 1.75 and 1.35 BLEU points for English-to-Russian and Russian-to-English respectively, compared to the baseline Transformer network trained in identical settings.
Phrase-based attention mechanisms to assign attention on phrases, achieving token-to-phrase, phrase-to-token, phrase-to-phrase attention alignments, in addition to existing token-to-token attentions. Paper presents an attention mechanism that computes a weighted sum over not only single tokens but ngrams(phrases).