Enhancing Conceptual Understanding in Multimodal Contrastive Learning
  through Hard Negative Samples
Philipp J. Rösch Norbert Oswald Michaela Geierhos Jindřich Libovický
Current multimodal models leveraging contrastive learning often face
limitations in developing fine-grained conceptual understanding. This is due to
random negative samples during pretraining, causing almost exclusively very
dissimilar concepts to be compared in the loss function. Consequently, the
models struggle with fine-grained semantic differences. To address this
problem, we introduce a novel pretraining method incorporating synthetic hard
negative text examples. The hard negatives permute terms corresponding to
visual concepts, leading to a more fine-grained visual and textual concept
alignment. Further, we introduce InpaintCOCO, a new challenging dataset for
assessing the fine-grained alignment of colors, objects, and sizes in
vision-language models. We created the dataset using generative inpainting from
COCO images by changing the visual concepts so that the images no longer match
their original captions. Our results show significant improvements in
fine-grained concept understanding across a wide range of vision-language
datasets, including our InpaintCOCO dataset.