Heterogeneous Contrastive Learning for Foundation Models and Beyond
Lecheng Zheng Baoyu Jing Zihao Li Hanghang Tong Jingrui He
In the era of big data and Artificial Intelligence, an emerging paradigm is
to utilize contrastive self-supervised learning to model large-scale
heterogeneous data. Many existing foundation models benefit from the
generalization capability of contrastive self-supervised learning by learning
compact and high-quality representations without relying on any label
information. Amidst the explosive advancements in foundation models across
multiple domains, including natural language processing and computer vision, a
thorough survey on heterogeneous contrastive learning for the foundation model
is urgently needed. In response, this survey critically evaluates the current
landscape of heterogeneous contrastive learning for foundation models,
highlighting the open challenges and future trends of contrastive learning. In
particular, we first present how the recent advanced contrastive learning-based
methods deal with view heterogeneity and how contrastive learning is applied to
train and fine-tune the multi-view foundation models. Then, we move to
contrastive learning methods for task heterogeneity, including pretraining
tasks and downstream tasks, and show how different tasks are combined with
contrastive learning loss for different purposes. Finally, we conclude this
survey by discussing the open challenges and shedding light on the future
directions of contrastive learning.