CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for
  Multimodal Machine Translation
Devaansh Gupta Siddhant Kharbanda Jiawei Zhou Wanhua Li Hanspeter Pfister Donglai Wei
There has been a growing interest in developing multimodal machine
translation (MMT) systems that enhance neural machine translation (NMT) with
visual knowledge. This problem setup involves using images as auxiliary
information during training, and more recently, eliminating their use during
inference. Towards this end, previous works face a challenge in training
powerful MMT models from scratch due to the scarcity of annotated multilingual
vision-language data, especially for low-resource languages. Simultaneously,
there has been an influx of multilingual pre-trained models for NMT and
multimodal pre-trained models for vision-language tasks, primarily in English,
which have shown exceptional generalisation ability. However, these are not
directly applicable to MMT since they do not provide aligned multimodal
multilingual features for generative tasks. To alleviate this issue, instead of
designing complex modules for MMT, we propose CLIPTrans, which simply adapts
the independently pre-trained multimodal M-CLIP and the multilingual mBART. In
order to align their embedding spaces, mBART is conditioned on the M-CLIP
features by a prefix sequence generated through a lightweight mapping network.
We train this in a two-stage pipeline which warms up the model with image
captioning before the actual translation task. Through experiments, we
demonstrate the merits of this framework and consequently push forward the
state-of-the-art across standard benchmarks by an average of +2.67 BLEU. The
code can be found at www.github.com/devaansh100/CLIPTrans.