Multi-Modality Guidance Network For Missing Modality Inference
Zhuokai Zhao Harish Palani Tianyi Liu Lena Evans Ruth Toner
Multimodal models have gained significant success in recent years. Standard
multimodal approaches often assume unchanged modalities from training stage to
inference stage. In practice, however, many scenarios fail to satisfy such
assumptions with missing modalities during inference, leading to limitations on
where multimodal models can be applied. While existing methods mitigate the
problem through reconstructing the missing modalities, it increases unnecessary
computational cost, which could be just as critical, especially for large,
deployed systems. To solve the problem from both sides, we propose a novel
guidance network that promotes knowledge sharing during training, taking
advantage of the multimodal representations to train better single-modality
models for inference. Real-life experiment in violence detection shows that our
proposed framework trains single-modality models that significantly outperform
its traditionally trained counterparts while maintaining the same inference
cost.