Bridging the Gap between Synthetic and Authentic Images for Multimodal
  Machine Translation
Wenyu Guo Qingkai Fang Dong Yu Yang Feng
Multimodal machine translation (MMT) simultaneously takes the source sentence
and a relevant image as input for translation. Since there is no paired image
available for the input sentence in most cases, recent studies suggest
utilizing powerful text-to-image generation models to provide image inputs.
Nevertheless, synthetic images generated by these models often follow different
distributions compared to authentic images. Consequently, using authentic
images for training and synthetic images for inference can introduce a
distribution shift, resulting in performance degradation during inference. To
tackle this challenge, in this paper, we feed synthetic and authentic images to
the MMT model, respectively. Then we minimize the gap between the synthetic and
authentic images by drawing close the input image representations of the
Transformer Encoder and the output distributions of the Transformer Decoder.
Therefore, we mitigate the distribution disparity introduced by the synthetic
images during inference, thereby freeing the authentic images from the
inference process.Experimental results show that our approach achieves
state-of-the-art performance on the Multi30K En-De and En-Fr datasets, while
remaining independent of authentic images during inference.