Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis
Xuming Hu Zhijiang Guo Zhiyang Teng Irwin King Philip S. Yu
Multimodal relation extraction (MRE) is the task of identifying the semantic
relationships between two entities based on the context of the sentence image
pair. Existing retrieval-augmented approaches mainly focused on modeling the
retrieved textual knowledge, but this may not be able to accurately identify
complex relations. To improve the prediction, this research proposes to
retrieve textual and visual evidence based on the object, sentence, and whole
image. We further develop a novel approach to synthesize the object-level,
image-level, and sentence-level information for better reasoning between the
same and different modalities. Extensive experiments and analyses show that the
proposed method is able to effectively select and compare evidence across
modalities and significantly outperforms state-of-the-art models.