GEMTrans: A General, Echocardiography-based, Multi-Level Transformer
  Framework for Cardiovascular Diagnosis
Masoud Mokhtari Neda Ahmadi Teresa S. M. Tsang Purang Abolmaesumi Renjie Liao
Echocardiography (echo) is an ultrasound imaging modality that is widely used
for various cardiovascular diagnosis tasks. Due to inter-observer variability
in echo-based diagnosis, which arises from the variability in echo image
acquisition and the interpretation of echo images based on clinical experience,
vision-based machine learning (ML) methods have gained popularity to act as
secondary layers of verification. For such safety-critical applications, it is
essential for any proposed ML method to present a level of explainability along
with good accuracy. In addition, such methods must be able to process several
echo videos obtained from various heart views and the interactions among them
to properly produce predictions for a variety of cardiovascular measurements or
interpretation tasks. Prior work lacks explainability or is limited in scope by
focusing on a single cardiovascular task. To remedy this, we propose a General,
Echo-based, Multi-Level Transformer (GEMTrans) framework that provides
explainability, while simultaneously enabling multi-video training where the
inter-play among echo image patches in the same frame, all frames in the same
video, and inter-video relationships are captured based on a downstream task.
We show the flexibility of our framework by considering two critical tasks
including ejection fraction (EF) and aortic stenosis (AS) severity detection.
Our model achieves mean absolute errors of 4.15 and 4.84 for single and
dual-video EF estimation and an accuracy of 96.5 % for AS detection, while
providing informative task-specific attention maps and prototypical
explainability.