Multimodal Fusion of EHR in Structures and Semantics: Integrating
  Clinical Records and Notes with Hypergraph and LLM
Hejie Cui Xinyu Fang Ran Xu Xuan Kan Joyce C. Ho Carl Yang
Electronic Health Records (EHRs) have become increasingly popular to support
clinical decision-making and healthcare in recent decades. EHRs usually contain
heterogeneous information, such as structural data in tabular form and
unstructured data in textual notes. Different types of information in EHRs can
complement each other and provide a more complete picture of the health status
of a patient. While there has been a lot of research on representation learning
of structured EHR data, the fusion of different types of EHR data (multimodal
fusion) is not well studied. This is mostly because of the complex medical
coding systems used and the noise and redundancy present in the written notes.
In this work, we propose a new framework called MINGLE, which integrates both
structures and semantics in EHR effectively. Our framework uses a two-level
infusion strategy to combine medical concept semantics and clinical note
semantics into hypergraph neural networks, which learn the complex interactions
between different types of data to generate visit representations for
downstream prediction. Experiment results on two EHR datasets, the public
MIMIC-III and private CRADLE, show that MINGLE can effectively improve
predictive performance by 11.83% relatively, enhancing semantic integration as
well as multimodal fusion for structural and textual EHR data.