Two Effects, One Trigger: On the Modality Gap, Object Bias, and
  Information Imbalance in Contrastive Vision-Language Representation Learning
Simon Schrodi David T. Hoffmann Max Argus Volker Fischer Thomas Brox
Contrastive vision-language models like CLIP have gained popularity for their
versatile applicable learned representations in various downstream tasks.
Despite their successes in some tasks, like zero-shot image recognition, they
also perform surprisingly poor on other tasks, like attribute detection.
Previous work has attributed these challenges to the modality gap, a separation
of image and text in the shared representation space, and a bias towards
objects over other factors, such as attributes. In this work we investigate
both phenomena. We find that only a few embedding dimensions drive the modality
gap. Further, we propose a measure for object bias and find that object bias
does not lead to worse performance on other concepts, such as attributes. But
what leads to the emergence of the modality gap and object bias? To answer this
question we carefully designed an experimental setting which allows us to
control the amount of shared information between the modalities. This revealed
that the driving factor behind both, the modality gap and the object bias, is
the information imbalance between images and captions.