RankCLIP: Ranking-Consistent Language-Image Pretraining
Yiming Zhang Zhuokai Zhao Zhaorun Chen Zhili Feng Zenghui Ding Yining Sun
Among the ever-evolving development of vision-language models, contrastive
language-image pretraining (CLIP) has set new benchmarks in many downstream
tasks such as zero-shot classifications by leveraging self-supervised
contrastive learning on large amounts of text-image pairs. However, its
dependency on rigid one-to-one mappings overlooks the complex and often
multifaceted relationships between and within texts and images. To this end, we
introduce RankCLIP, a novel pretraining method that extends beyond the rigid
one-to-one matching framework of CLIP and its variants. By leveraging both
in-modal and cross-modal ranking consistency, RankCLIP improves the alignment
process, enabling it to capture the nuanced many-to-many relationships between
and within each modality. Through comprehensive experiments, we demonstrate the
enhanced capability of RankCLIP to effectively improve performance across
various downstream tasks, notably achieving significant gains in zero-shot
classifications over state-of-the-art methods, underscoring the potential of
RankCLIP in further advancing vision-language pretraining.