Exploring the Necessity of Visual Modality in Multimodal Machine
  Translation using Authentic Datasets
Zi Long Zhenhao Tang Xianghua Fu Jian Chen Shilong Hou Jinze Lyu
Recent research in the field of multimodal machine translation (MMT) has
indicated that the visual modality is either dispensable or offers only
marginal advantages. However, most of these conclusions are drawn from the
analysis of experimental results based on a limited set of bilingual
sentence-image pairs, such as Multi30k. In these kinds of datasets, the content
of one bilingual parallel sentence pair must be well represented by a manually
annotated image, which is different from the real-world translation scenario.
In this work, we adhere to the universal multimodal machine translation
framework proposed by Tang et al. (2022). This approach allows us to delve into
the impact of the visual modality on translation efficacy by leveraging
real-world translation datasets. Through a comprehensive exploration via
probing tasks, we find that the visual modality proves advantageous for the
majority of authentic translation datasets. Notably, the translation
performance primarily hinges on the alignment and coherence between textual and
visual contents. Furthermore, our results suggest that visual information
serves a supplementary role in multimodal translation and can be substituted.