On the Generalization of Multi-modal Contrastive Learning
Qi Zhang Yifei Wang Yisen Wang
Multi-modal contrastive learning (MMCL) has recently garnered considerable
interest due to its superior performance in visual tasks, achieved by embedding
multi-modal data, such as visual-language pairs. However, there still lack
theoretical understandings of how MMCL extracts useful visual representation
from multi-modal pairs, and particularly, how MMCL outperforms previous
approaches like self-supervised contrastive learning (SSCL). In this paper, by
drawing an intrinsic connection between MMCL and asymmetric matrix
factorization, we establish the first generalization guarantees of MMCL for
visual downstream tasks. Based on this framework, we further unify MMCL and
SSCL by showing that MMCL implicitly performs SSCL with (pseudo) positive pairs
induced by text pairs. Through this unified perspective, we characterize the
advantage of MMCL by showing that text pairs induce more semantically
consistent and diverse positive pairs, which, according to our analysis,
provably benefit downstream generalization. Inspired by this finding, we
propose CLIP-guided resampling methods to significantly improve the downstream
performance of SSCL on ImageNet by leveraging multi-modal information. Code is
available at https://github.com/PKU-ML/CLIP-Help-SimCLR.