Global Contrastive Training for Multimodal Electronic Health Records
  with Language Supervision
Yingbo Ma Suraj Kolla Zhenhong Hu Dhruv Kaliraman Victoria Nolan Ziyuan Guan Yuanfang Ren Brooke Armfield Tezcan Ozrazgat-Baslanti Jeremy A. Balch Tyler J. Loftus Parisa Rashidi Azra Bihorac Benjamin Shickel
Modern electronic health records (EHRs) hold immense promise in tracking
personalized patient health trajectories through sequential deep learning,
owing to their extensive breadth, scale, and temporal granularity. Nonetheless,
how to effectively leverage multiple modalities from EHRs poses significant
challenges, given its complex characteristics such as high dimensionality,
multimodality, sparsity, varied recording frequencies, and temporal
irregularities. To this end, this paper introduces a novel multimodal
contrastive learning framework, specifically focusing on medical time series
and clinical notes. To tackle the challenge of sparsity and irregular time
intervals in medical time series, the framework integrates temporal
cross-attention transformers with a dynamic embedding and tokenization scheme
for learning multimodal feature representations. To harness the interconnected
relationships between medical time series and clinical notes, the framework
equips a global contrastive loss, aligning a patient's multimodal feature
representations with the corresponding discharge summaries. Since discharge
summaries uniquely pertain to individual patients and represent a holistic view
of the patient's hospital stay, machine learning models are led to learn
discriminative multimodal features via global contrasting. Extensive
experiments with a real-world EHR dataset demonstrated that our framework
outperformed state-of-the-art approaches on the exemplar task of predicting the
occurrence of nine postoperative complications for more than 120,000 major
inpatient surgeries using multimodal data from UF health system split among
three hospitals (UF Health Gainesville, UF Health Jacksonville, and UF Health
Jacksonville-North).