Language Plays a Pivotal Role in the Object-Attribute Compositional
  Generalization of CLIP
Reza Abbasi Mohammad Samiei Mohammad Hossein Rohban Mahdieh Soleymani Baghshah
Vision-language models, such as CLIP, have shown promising
Out-of-Distribution (OoD) generalization under various types of distribution
shifts. Recent studies attempted to investigate the leading cause of this
capability. In this work, we follow the same path, but focus on a specific type
of OoD data - images with novel compositions of attribute-object pairs - and
study whether such models can successfully classify those images into
composition classes. We carefully designed an authentic image test dataset
called ImageNet-AO, consisting of attributes for objects that are unlikely
encountered in the CLIP training sets. We found that CLIPs trained with large
datasets such as OpenAI CLIP, LAION-400M, and LAION-2B show orders-of-magnitude
improvement in effective compositional OoD generalization compared to both
supervised models and CLIPs trained with smaller datasets, such as CC-12M and
YFCC-15M. Our results provide evidence that the scale and diversity of training
data and language supervision play a key role in unlocking the compositional
generalization abilities of vision-language models.