HyperFusion: A Hypernetwork Approach to Multimodal Integration of
  Tabular and Medical Imaging Data for Predictive Modeling
Daniel Duenias Brennan Nichyporuk Tal Arbel Tammy Riklin Raviv
The integration of diverse clinical modalities such as medical imaging and
the tabular data obtained by the patients' Electronic Health Records (EHRs) is
a crucial aspect of modern healthcare. The integrative analysis of multiple
sources can provide a comprehensive understanding of a patient's condition and
can enhance diagnoses and treatment decisions. Deep Neural Networks (DNNs)
consistently showcase outstanding performance in a wide range of multimodal
tasks in the medical domain. However, the complex endeavor of effectively
merging medical imaging with clinical, demographic and genetic information
represented as numerical tabular data remains a highly active and ongoing
research pursuit.
  We present a novel framework based on hypernetworks to fuse clinical imaging
and tabular data by conditioning the image processing on the EHR's values and
measurements. This approach aims to leverage the complementary information
present in these modalities to enhance the accuracy of various medical
applications. We demonstrate the strength and the generality of our method on
two different brain Magnetic Resonance Imaging (MRI) analysis tasks, namely,
brain age prediction conditioned by subject's sex, and multiclass Alzheimer's
Disease (AD) classification conditioned by tabular data. We show that our
framework outperforms both single-modality models and state-of-the-art
MRI-tabular data fusion methods. The code, enclosed to this manuscript will be
made publicly available.