Translation-Enhanced Multilingual Text-to-Image Generation
Yaoyiran Li Ching-Yun Chang Stephen Rawls Ivan Vulić Anna Korhonen
Research on text-to-image generation (TTI) still predominantly focuses on the
English language due to the lack of annotated image-caption data in other
languages; in the long run, this might widen inequitable access to TTI
technology. In this work, we thus investigate multilingual TTI (termed mTTI)
and the current potential of neural machine translation (NMT) to bootstrap mTTI
systems. We provide two key contributions. 1) Relying on a multilingual
multi-modal encoder, we provide a systematic empirical study of standard
methods used in cross-lingual NLP when applied to mTTI: Translate Train,
Translate Test, and Zero-Shot Transfer. 2) We propose Ensemble Adapter (EnsAd),
a novel parameter-efficient approach that learns to weigh and consolidate the
multilingual text knowledge within the mTTI framework, mitigating the language
gap and thus improving mTTI performance. Our evaluations on standard mTTI
datasets COCO-CN, Multi30K Task2, and LAION-5B demonstrate the potential of
translation-enhanced mTTI systems and also validate the benefits of the
proposed EnsAd which derives consistent gains across all datasets. Further
investigations on model variants, ablation studies, and qualitative analyses
provide additional insights on the inner workings of the proposed mTTI
approaches.