Leveraging Diverse Modeling Contexts with Collaborating Learning for
  Neural Machine Translation
Yusheng Liao Yanfeng Wang Yu Wang
Autoregressive (AR) and Non-autoregressive (NAR) models are two types of
generative models for Neural Machine Translation (NMT). AR models predict
tokens in a word-by-word manner and can effectively capture the distribution of
real translations. NAR models predict tokens by extracting bidirectional
contextual information which can improve the inference speed but they suffer
from performance degradation. Previous works utilized AR models to enhance NAR
models by reducing the training data's complexity or incorporating the global
information into AR models by virtue of NAR models. However, those investigated
methods only take advantage of the contextual information of a single type of
model while neglecting the diversity in the contextual information that can be
provided by different types of models. In this paper, we propose a novel
generic collaborative learning method, DCMCL, where AR and NAR models are
treated as collaborators instead of teachers and students. To hierarchically
leverage the bilateral contextual information, token-level mutual learning and
sequence-level contrastive learning are adopted between AR and NAR models.
Extensive experiments on four widely used benchmarks show that the proposed
DCMCL method can simultaneously improve both AR and NAR models with up to 1.38
and 2.98 BLEU scores respectively, and can also outperform the current
best-unified model with up to 0.97 BLEU scores for both AR and NAR decoding.