Improving Contextual Congruence Across Modalities for Effective
  Multimodal Marketing using Knowledge-infused Learning
Trilok Padhi Ugur Kursuncu Yaman Kumar Valerie L. Shalin Lane Peterson Fronczek
The prevalence of smart devices with the ability to capture moments in
multiple modalities has enabled users to experience multimodal information
online. However, large Language (LLMs) and Vision models (LVMs) are still
limited in capturing holistic meaning with cross-modal semantic relationships.
Without explicit, common sense knowledge (e.g., as a knowledge graph), Visual
Language Models (VLMs) only learn implicit representations by capturing
high-level patterns in vast corpora, missing essential contextual cross-modal
cues. In this work, we design a framework to couple explicit commonsense
knowledge in the form of knowledge graphs with large VLMs to improve the
performance of a downstream task, predicting the effectiveness of multi-modal
marketing campaigns. While the marketing application provides a compelling
metric for assessing our methods, our approach enables the early detection of
likely persuasive multi-modal campaigns and the assessment and augmentation of
marketing theory.