Revealing Multimodal Contrastive Representation Learning through Latent
  Partial Causal Models
Yuhang Liu Zhen Zhang Dong Gong Biwei Huang Mingming Gong Anton van den Hengel Kun Zhang Javen Qinfeng Shi
Multimodal contrastive representation learning methods have proven successful
across a range of domains, partly due to their ability to generate meaningful
shared representations of complex phenomena. To enhance the depth of analysis
and understanding of these acquired representations, we introduce a unified
causal model specifically designed for multimodal data. By examining this
model, we show that multimodal contrastive representation learning excels at
identifying latent coupled variables within the proposed unified model, up to
linear or permutation transformations resulting from different assumptions. Our
findings illuminate the potential of pre-trained multimodal models, eg, CLIP,
in learning disentangled representations through a surprisingly simple yet
highly effective tool: linear independent component analysis. Experiments
demonstrate the robustness of our findings, even when the assumptions are
violated, and validate the effectiveness of the proposed method in learning
disentangled representations.