A Quantum Probability Driven Framework for Joint Multi-Modal Sarcasm,
  Sentiment and Emotion Analysis
Yaochen Liu Yazhou Zhang Dawei Song
Sarcasm, sentiment, and emotion are three typical kinds of spontaneous
affective responses of humans to external events and they are tightly
intertwined with each other. Such events may be expressed in multiple
modalities (e.g., linguistic, visual and acoustic), e.g., multi-modal
conversations. Joint analysis of humans' multi-modal sarcasm, sentiment, and
emotion is an important yet challenging topic, as it is a complex cognitive
process involving both cross-modality interaction and cross-affection
correlation. From the probability theory perspective, cross-affection
correlation also means that the judgments on sarcasm, sentiment, and emotion
are incompatible. However, this exposed phenomenon cannot be sufficiently
modelled by classical probability theory due to its assumption of
compatibility. Neither do the existing approaches take it into consideration.
In view of the recent success of quantum probability (QP) in modeling human
cognition, particularly contextual incompatible decision making, we take the
first step towards introducing QP into joint multi-modal sarcasm, sentiment,
and emotion analysis. Specifically, we propose a QUantum probabIlity driven
multi-modal sarcasm, sEntiment and emoTion analysis framework, termed QUIET.
Extensive experiments on two datasets and the results show that the
effectiveness and advantages of QUIET in comparison with a wide range of the
state-of-the-art baselines. We also show the great potential of QP in
multi-affect analysis.