FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in
  Computational Pathology
Yuanzhe Peng Jieming Bian Jie Xu
The fusion of complementary multimodal information is crucial in
computational pathology for accurate diagnostics. However, existing multimodal
learning approaches necessitate access to users' raw data, posing substantial
privacy risks. While Federated Learning (FL) serves as a privacy-preserving
alternative, it falls short in addressing the challenges posed by heterogeneous
(yet possibly overlapped) modalities data across various hospitals. To bridge
this gap, we propose a Federated Multi-Modal (FedMM) learning framework that
federatedly trains multiple single-modal feature extractors to enhance
subsequent classification performance instead of existing FL that aims to train
a unified multimodal fusion model. Any participating hospital, even with
small-scale datasets or limited devices, can leverage these federated trained
extractors to perform local downstream tasks (e.g., classification) while
ensuring data privacy. Through comprehensive evaluations of two publicly
available datasets, we demonstrate that FedMM notably outperforms two baselines
in accuracy and AUC metrics.