Propensity Score Alignment of Unpaired Multimodal Data
Johnny Xi Jason Hartford
Multimodal representation learning techniques typically rely on paired
samples to learn common representations, but paired samples are challenging to
collect in fields such as biology where measurement devices often destroy the
samples. This paper presents an approach to address the challenge of aligning
unpaired samples across disparate modalities in multimodal representation
learning. We draw an analogy between potential outcomes in causal inference and
potential views in multimodal observations, which allows us to use Rubin's
framework to estimate a common space in which to match samples. Our approach
assumes we collect samples that are experimentally perturbed by treatments, and
uses this to estimate a propensity score from each modality, which encapsulates
all shared information between a latent state and treatment and can be used to
define a distance between samples. We experiment with two alignment techniques
that leverage this distance -- shared nearest neighbours (SNN) and optimal
transport (OT) matching -- and find that OT matching results in significant
improvements over state-of-the-art alignment approaches in both a synthetic
multi-modal setting and in real-world data from NeurIPS Multimodal Single-Cell
Integration Challenge.