Augmenters at SemEval-2023 Task 1: Enhancing CLIP in Handling
  Compositionality and Ambiguity for Zero-Shot Visual WSD through Prompt
  Augmentation and Text-To-Image Diffusion
Jie S. Li Yow-Ting Shiue Yong-Siang Shih Jonas Geiping
This paper describes our zero-shot approaches for the Visual Word Sense
Disambiguation (VWSD) Task in English. Our preliminary study shows that the
simple approach of matching candidate images with the phrase using CLIP suffers
from the many-to-many nature of image-text pairs. We find that the CLIP text
encoder may have limited abilities in capturing the compositionality in natural
language. Conversely, the descriptive focus of the phrase varies from instance
to instance. We address these issues in our two systems, Augment-CLIP and
Stable Diffusion Sampling (SD Sampling). Augment-CLIP augments the text prompt
by generating sentences that contain the context phrase with the help of large
language models (LLMs). We further explore CLIP models in other languages, as
the an ambiguous word may be translated into an unambiguous one in the other
language. SD Sampling uses text-to-image Stable Diffusion to generate multiple
images from the given phrase, increasing the likelihood that a subset of images
match the one that paired with the text.