Fusing Echocardiography Images and Medical Records for Continuous
  Patient Stratification
Nathan Painchaud Pierre-Yves Courand Pierre-Marc Jodoin Nicolas Duchateau Olivier Bernard
Deep learning now enables automatic and robust extraction of cardiac function
descriptors from echocardiographic sequences, such as ejection fraction or
strain. These descriptors provide fine-grained information that physicians
consider, in conjunction with more global variables from the clinical record,
to assess patients' condition. Drawing on novel transformer models applied to
tabular data (e.g., variables from electronic health records), we propose a
method that considers all descriptors extracted from medical records and
echocardiograms to learn the representation of a difficult-to-characterize
cardiovascular pathology, namely hypertension. Our method first projects each
variable into its own representation space using modality-specific approaches.
These standardized representations of multimodal data are then fed to a
transformer encoder, which learns to merge them into a comprehensive
representation of the patient through a pretext task of predicting a clinical
rating. This pretext task is formulated as an ordinal classification to enforce
a pathological continuum in the representation space. We observe the major
trends along this continuum for a cohort of 239 hypertensive patients to
describe, with unprecedented gradation, the effect of hypertension on a number
of cardiac function descriptors. Our analysis shows that i) pretrained weights
from a foundation model allow to reach good performance (83% accuracy) even
with limited data (less than 200 training samples), ii) trends across the
population are reproducible between trainings, and iii) for descriptors whose
interactions with hypertension are well documented, patterns are consistent
with prior physiological knowledge.