Training on Synthetic Data Beats Real Data in Multimodal Relation
  Extraction
Zilin Du Haoxin Li Xu Guo Boyang Li
The task of multimodal relation extraction has attracted significant research
attention, but progress is constrained by the scarcity of available training
data. One natural thought is to extend existing datasets with cross-modal
generative models. In this paper, we consider a novel problem setting, where
only unimodal data, either text or image, are available during training. We aim
to train a multimodal classifier from synthetic data that perform well on real
multimodal test data. However, training with synthetic data suffers from two
obstacles: lack of data diversity and label information loss. To alleviate the
issues, we propose Mutual Information-aware Multimodal Iterated Relational dAta
GEneration (MI2RAGE), which applies Chained Cross-modal Generation (CCG) to
promote diversity in the generated data and exploits a teacher network to
select valuable training samples with high mutual information with the
ground-truth labels. Comparing our method to direct training on synthetic data,
we observed a significant improvement of 24.06% F1 with synthetic text and
26.42% F1 with synthetic images. Notably, our best model trained on completely
synthetic images outperforms prior state-of-the-art models trained on real
multimodal data by a margin of 3.76% in F1. Our codebase will be made available
upon acceptance.