Adding Multimodal Capabilities to a Text-only Translation Model
Vipin Vijayan Braeden Bowen Scott Grigsby Timothy Anderson Jeremy Gwinnup
While most current work in multimodal machine translation (MMT) uses the
Multi30k dataset for training and evaluation, we find that the resulting models
overfit to the Multi30k dataset to an extreme degree. Consequently, these
models perform very badly when evaluated against typical text-only testing sets
such as the WMT newstest datasets. In order to perform well on both Multi30k
and typical text-only datasets, we use a performant text-only machine
translation (MT) model as the starting point of our MMT model. We add
vision-text adapter layers connected via gating mechanisms to the MT model, and
incrementally transform the MT model into an MMT model by 1) pre-training using
vision-based masking of the source text and 2) fine-tuning on Multi30k.