Multi-source Semantic Graph-based Multimodal Sarcasm Explanation
  Generation
Liqiang Jing Xuemeng Song Kun Ouyang Mengzhao Jia Liqiang Nie
Multimodal Sarcasm Explanation (MuSE) is a new yet challenging task, which
aims to generate a natural language sentence for a multimodal social post (an
image as well as its caption) to explain why it contains sarcasm. Although the
existing pioneer study has achieved great success with the BART backbone, it
overlooks the gap between the visual feature space and the decoder semantic
space, the object-level metadata of the image, as well as the potential
external knowledge. To solve these limitations, in this work, we propose a
novel mulTi-source sEmantic grAph-based Multimodal sarcasm explanation scheme,
named TEAM. In particular, TEAM extracts the object-level semantic meta-data
instead of the traditional global visual features from the input image.
Meanwhile, TEAM resorts to ConceptNet to obtain the external related knowledge
concepts for the input text and the extracted object meta-data. Thereafter,
TEAM introduces a multi-source semantic graph that comprehensively characterize
the multi-source (i.e., caption, object meta-data, external knowledge) semantic
relations to facilitate the sarcasm reasoning. Extensive experiments on a
public released dataset MORE verify the superiority of our model over
cutting-edge methods.