Understanding Transferable Representation Learning and Zero-shot
  Transfer in CLIP
Zixiang Chen Yihe Deng Yuanzhi Li Quanquan Gu
Multi-modal learning has become increasingly popular due to its ability to
leverage information from different data sources (e.g., text and images) to
improve the model performance. Recently, CLIP has emerged as an effective
approach that employs vision-language contrastive pretraining to learn joint
image and text representations and exhibits remarkable performance in zero-shot
learning and text-guided natural image generation. Despite the huge practical
success of CLIP, its theoretical understanding remains elusive. In this paper,
we formally study transferrable representation learning underlying CLIP and
demonstrate how features from different modalities get aligned. We also analyze
its zero-shot transfer performance on the downstream tasks. Inspired by our
analysis, we propose a new CLIP-type approach, which achieves better
performance than CLIP and other state-of-the-art methods on benchmark datasets.