Multi-modal Learning with Missing Modality via Shared-Specific Feature
  Modelling
Hu Wang Yuanhong Chen Congbo Ma Jodie Avery Louise Hull Gustavo Carneiro
The missing modality issue is critical but non-trivial to be solved by
multi-modal models. Current methods aiming to handle the missing modality
problem in multi-modal tasks, either deal with missing modalities only during
evaluation or train separate models to handle specific missing modality
settings. In addition, these models are designed for specific tasks, so for
example, classification models are not easily adapted to segmentation tasks and
vice versa. In this paper, we propose the Shared-Specific Feature Modelling
(ShaSpec) method that is considerably simpler and more effective than competing
approaches that address the issues above. ShaSpec is designed to take advantage
of all available input modalities during training and evaluation by learning
shared and specific features to better represent the input data. This is
achieved from a strategy that relies on auxiliary tasks based on distribution
alignment and domain classification, in addition to a residual feature fusion
procedure. Also, the design simplicity of ShaSpec enables its easy adaptation
to multiple tasks, such as classification and segmentation. Experiments are
conducted on both medical image segmentation and computer vision
classification, with results indicating that ShaSpec outperforms competing
methods by a large margin. For instance, on BraTS2018, ShaSpec improves the
SOTA by more than 3% for enhancing tumour, 5% for tumour core and 3% for whole
tumour.