MaxCorrMGNN: A Multi-Graph Neural Network Framework for Generalized
  Multimodal Fusion of Medical Data for Outcome Prediction
Niharika S. D'Souza Hongzhi Wang Andrea Giovannini Antonio Foncubierta-Rodriguez Kristen L. Beck Orest Boyko Tanveer Syeda-Mahmood
With the emergence of multimodal electronic health records, the evidence for
an outcome may be captured across multiple modalities ranging from clinical to
imaging and genomic data. Predicting outcomes effectively requires fusion
frameworks capable of modeling fine-grained and multi-faceted complex
interactions between modality features within and across patients. We develop
an innovative fusion approach called MaxCorr MGNN that models non-linear
modality correlations within and across patients through
Hirschfeld-Gebelein-Renyi maximal correlation (MaxCorr) embeddings, resulting
in a multi-layered graph that preserves the identities of the modalities and
patients. We then design, for the first time, a generalized multi-layered graph
neural network (MGNN) for task-informed reasoning in multi-layered graphs, that
learns the parameters defining patient-modality graph connectivity and message
passing in an end-to-end fashion. We evaluate our model an outcome prediction
task on a Tuberculosis (TB) dataset consistently outperforming several
state-of-the-art neural, graph-based and traditional fusion techniques.