MM-AU:Towards Multimodal Understanding of Advertisement Videos
Digbalay Bose Rajat Hebbar Tiantian Feng Krishna Somandepalli Anfeng Xu Shrikanth Narayanan
Advertisement videos (ads) play an integral part in the domain of Internet
e-commerce as they amplify the reach of particular products to a broad audience
or can serve as a medium to raise awareness about specific issues through
concise narrative structures. The narrative structures of advertisements
involve several elements like reasoning about the broad content (topic and the
underlying message) and examining fine-grained details involving the transition
of perceived tone due to the specific sequence of events and interaction among
characters. In this work, to facilitate the understanding of advertisements
along the three important dimensions of topic categorization, perceived tone
transition, and social message detection, we introduce a multimodal
multilingual benchmark called MM-AU composed of over 8.4K videos (147 hours)
curated from multiple web sources. We explore multiple zero-shot reasoning
baselines through the application of large language models on the ads
transcripts. Further, we demonstrate that leveraging signals from multiple
modalities, including audio, video, and text, in multimodal transformer-based
supervised models leads to improved performance compared to unimodal
approaches.