Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via
  Projective Networks
José Morano Guilherme Aresta Christoph Grechenig Ursula Schmidt-Erfurth Hrvoje Bogunović
The use of multimodal imaging has led to significant improvements in the
diagnosis and treatment of many diseases. Similar to clinical practice, some
works have demonstrated the benefits of multimodal fusion for automatic
segmentation and classification using deep learning-based methods. However,
current segmentation methods are limited to fusion of modalities with the same
dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the
fusion strategies implemented by classification methods are incompatible with
localization tasks. In this work, we propose a novel deep learning-based
framework for the fusion of multimodal data with heterogeneous dimensionality
(e.g., 3D+2D) that is compatible with localization tasks. The proposed
framework extracts the features of the different modalities and projects them
into the common feature subspace. The projected features are then fused and
further processed to obtain the final prediction. The framework was validated
on the following tasks: segmentation of geographic atrophy (GA), a late-stage
manifestation of age-related macular degeneration, and segmentation of retinal
blood vessels (RBV) in multimodal retinal imaging. Our results show that the
proposed method outperforms the state-of-the-art monomodal methods on GA and
RBV segmentation by up to 3.10% and 4.64% Dice, respectively.