Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking
Tianyu Zhu Myong Chol Jung Jesse Clark
Contrastive learning has gained widespread adoption for retrieval tasks due
to its minimal requirement for manual annotations. However, popular contrastive
frameworks typically learn from binary relevance, making them ineffective at
incorporating direct fine-grained rankings. In this paper, we curate a
large-scale dataset featuring detailed relevance scores for each query-document
pair to facilitate future research and evaluation. Subsequently, we propose
Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking (GCL),
which is designed to learn from fine-grained rankings beyond binary relevance
scores. Our results show that GCL achieves a 94.5% increase in NDCG@10 for
in-domain and 26.3 to 48.8% increases for cold-start evaluations, all relative
to the CLIP baseline and involving ground truth rankings.