RedCore: Relative Advantage Aware Cross-modal Representation Learning
  for Missing Modalities with Imbalanced Missing Rates
Jun Sun Xinxin Zhang Shoukang Han Yu-ping Ruan Taihao Li
Multimodal learning is susceptible to modality missing, which poses a major
obstacle for its practical applications and, thus, invigorates increasing
research interest. In this paper, we investigate two challenging problems: 1)
when modality missing exists in the training data, how to exploit the
incomplete samples while guaranteeing that they are properly supervised? 2)
when the missing rates of different modalities vary, causing or exacerbating
the imbalance among modalities, how to address the imbalance and ensure all
modalities are well-trained? To tackle these two challenges, we first introduce
the variational information bottleneck (VIB) method for the cross-modal
representation learning of missing modalities, which capitalizes on the
available modalities and the labels as supervision. Then, accounting for the
imbalanced missing rates, we define relative advantage to quantify the
advantage of each modality over others. Accordingly, a bi-level optimization
problem is formulated to adaptively regulate the supervision of all modalities
during training. As a whole, the proposed approach features \textbf{Re}lative
a\textbf{d}vantage aware \textbf{C}ross-m\textbf{o}dal \textbf{r}epresentation
l\textbf{e}arning (abbreviated as \textbf{RedCore}) for missing modalities with
imbalanced missing rates. Extensive empirical results demonstrate that RedCore
outperforms competing models in that it exhibits superior robustness against
either large or imbalanced missing rates.