Learn "No" to Say "Yes" Better: Improving Vision-Language Models via
  Negations
Jaisidh Singh Ishaan Shrivastava Mayank Vatsa Richa Singh Aparna Bharati
Existing vision-language models (VLMs) treat text descriptions as a unit,
confusing individual concepts in a prompt and impairing visual semantic
matching and reasoning. An important aspect of reasoning in logic and language
is negations. This paper highlights the limitations of popular VLMs such as
CLIP, at understanding the implications of negations, i.e., the effect of the
word "not" in a given prompt. To enable evaluation of VLMs on fluent prompts
with negations, we present CC-Neg, a dataset containing 228,246 images, true
captions and their corresponding negated captions. Using CC-Neg along with
modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework,
has an improved understanding of negations. This training paradigm improves
CoN-CLIP's ability to encode semantics reliably, resulting in 3.85% average
gain in top-1 accuracy for zero-shot image classification across 8 datasets.
Further, CoN-CLIP outperforms CLIP on challenging compositionality benchmarks
such as SugarCREPE by 4.4%, showcasing emergent compositional understanding of
objects, relations, and attributes in text. Overall, our work addresses a
crucial limitation of VLMs by introducing a dataset and framework that
strengthens semantic associations between images and text, demonstrating
improved large-scale foundation models with significantly reduced computational
cost, promoting efficiency and accessibility.