Towards Better Multi-modal Keyphrase Generation via Visual Entity
  Enhancement and Multi-granularity Image Noise Filtering
Yifan Dong Suhang Wu Fandong Meng Jie Zhou Xiaoli Wang Jianxin Lin Jinsong Su
Multi-modal keyphrase generation aims to produce a set of keyphrases that
represent the core points of the input text-image pair. In this regard,
dominant methods mainly focus on multi-modal fusion for keyphrase generation.
Nevertheless, there are still two main drawbacks: 1) only a limited number of
sources, such as image captions, can be utilized to provide auxiliary
information. However, they may not be sufficient for the subsequent keyphrase
generation. 2) the input text and image are often not perfectly matched, and
thus the image may introduce noise into the model. To address these
limitations, in this paper, we propose a novel multi-modal keyphrase generation
model, which not only enriches the model input with external knowledge, but
also effectively filters image noise. First, we introduce external visual
entities of the image as the supplementary input to the model, which benefits
the cross-modal semantic alignment for keyphrase generation. Second, we
simultaneously calculate an image-text matching score and image region-text
correlation scores to perform multi-granularity image noise filtering.
Particularly, we introduce the correlation scores between image regions and
ground-truth keyphrases to refine the calculation of the previously-mentioned
correlation scores. To demonstrate the effectiveness of our model, we conduct
several groups of experiments on the benchmark dataset.
  Experimental results and in-depth analyses show that our model achieves the
state-of-the-art performance. Our code is available on
https://github.com/DeepLearnXMU/MM-MKP.