Sentence-level Prompts Benefit Composed Image Retrieval
Yang Bai Xinxing Xu Yong Liu Salman Khan Fahad Khan Wangmeng Zuo Rick Siow Mong Goh Chun-Mei Feng
Composed image retrieval (CIR) is the task of retrieving specific images by
using a query that involves both a reference image and a relative caption. Most
existing CIR models adopt the late-fusion strategy to combine visual and
language features. Besides, several approaches have also been suggested to
generate a pseudo-word token from the reference image, which is further
integrated into the relative caption for CIR. However, these pseudo-word-based
prompting methods have limitations when target image encompasses complex
changes on reference image, e.g., object removal and attribute modification. In
this work, we demonstrate that learning an appropriate sentence-level prompt
for the relative caption (SPRC) is sufficient for achieving effective composed
image retrieval. Instead of relying on pseudo-word-based prompts, we propose to
leverage pretrained V-L models, e.g., BLIP-2, to generate sentence-level
prompts. By concatenating the learned sentence-level prompt with the relative
caption, one can readily use existing text-based image retrieval models to
enhance CIR performance. Furthermore, we introduce both image-text contrastive
loss and text prompt alignment loss to enforce the learning of suitable
sentence-level prompts. Experiments show that our proposed method performs
favorably against the state-of-the-art CIR methods on the Fashion-IQ and CIRR
datasets. The source code and pretrained model are publicly available at
https://github.com/chunmeifeng/SPRC