Improved Automatic Diabetic Retinopathy Severity Classification Using
  Deep Multimodal Fusion of UWF-CFP and OCTA Images
Mostafa El Habib Daho Yihao Li Rachid Zeghlache Yapo Cedric Atse Hugo Le Boité Sophie Bonnin Deborah Cosette Pierre Deman Laurent Borderie Capucine Lepicard Ramin Tadayoni Béatrice Cochener Pierre-Henri Conze Mathieu Lamard Gwenolé Quellec
Diabetic Retinopathy (DR), a prevalent and severe complication of diabetes,
affects millions of individuals globally, underscoring the need for accurate
and timely diagnosis. Recent advancements in imaging technologies, such as
Ultra-WideField Color Fundus Photography (UWF-CFP) imaging and Optical
Coherence Tomography Angiography (OCTA), provide opportunities for the early
detection of DR but also pose significant challenges given the disparate nature
of the data they produce. This study introduces a novel multimodal approach
that leverages these imaging modalities to notably enhance DR classification.
Our approach integrates 2D UWF-CFP images and 3D high-resolution 6x6 mm$^3$
OCTA (both structure and flow) images using a fusion of ResNet50 and
3D-ResNet50 models, with Squeeze-and-Excitation (SE) blocks to amplify relevant
features. Additionally, to increase the model's generalization capabilities, a
multimodal extension of Manifold Mixup, applied to concatenated multimodal
features, is implemented. Experimental results demonstrate a remarkable
enhancement in DR classification performance with the proposed multimodal
approach compared to methods relying on a single modality only. The methodology
laid out in this work holds substantial promise for facilitating more accurate,
early detection of DR, potentially improving clinical outcomes for patients.