PECon: Contrastive Pretraining to Enhance Feature Alignment between CT
  and EHR Data for Improved Pulmonary Embolism Diagnosis
Santosh Sanjeev Salwa K. Al Khatib Mai A. Shaaban Ibrahim Almakky Vijay Ram Papineni Mohammad Yaqub
Previous deep learning efforts have focused on improving the performance of
Pulmonary Embolism(PE) diagnosis from Computed Tomography (CT) scans using
Convolutional Neural Networks (CNN). However, the features from CT scans alone
are not always sufficient for the diagnosis of PE. CT scans along with
electronic heath records (EHR) can provide a better insight into the patients
condition and can lead to more accurate PE diagnosis. In this paper, we propose
Pulmonary Embolism Detection using Contrastive Learning (PECon), a supervised
contrastive pretraining strategy that employs both the patients CT scans as
well as the EHR data, aiming to enhance the alignment of feature
representations between the two modalities and leverage information to improve
the PE diagnosis. In order to achieve this, we make use of the class labels and
pull the sample features of the same class together, while pushing away those
of the other class. Results show that the proposed work outperforms the
existing techniques and achieves state-of-the-art performance on the RadFusion
dataset with an F1-score of 0.913, accuracy of 0.90 and an AUROC of 0.943.
Furthermore, we also explore the explainability of our approach in comparison
to other methods. Our code is publicly available at
https://github.com/BioMedIA-MBZUAI/PECon.