Sequential Compositional Generalization in Multimodal Models
Semih Yagcioglu Osman Batur İnce Aykut Erdem Erkut Erdem Desmond Elliott Deniz Yuret
The rise of large-scale multimodal models has paved the pathway for
groundbreaking advances in generative modeling and reasoning, unlocking
transformative applications in a variety of complex tasks. However, a pressing
question that remains is their genuine capability for stronger forms of
generalization, which has been largely underexplored in the multimodal setting.
Our study aims to address this by examining sequential compositional
generalization using \textsc{CompAct} (\underline{Comp}ositional
\underline{Act}ivities)\footnote{Project Page:
\url{http://cyberiada.github.io/CompAct}}, a carefully constructed,
perceptually grounded dataset set within a rich backdrop of egocentric kitchen
activity videos. Each instance in our dataset is represented with a combination
of raw video footage, naturally occurring sound, and crowd-sourced step-by-step
descriptions. More importantly, our setup ensures that the individual concepts
are consistently distributed across training and evaluation sets, while their
compositions are novel in the evaluation set. We conduct a comprehensive
assessment of several unimodal and multimodal models. Our findings reveal that
bi-modal and tri-modal models exhibit a clear edge over their text-only
counterparts. This highlights the importance of multimodality while charting a
trajectory for future research in this domain.