Cross-Modal Prototype based Multimodal Federated Learning under Severely
  Missing Modality
Huy Q. Le Chu Myaet Thwal Yu Qiao Ye Lin Tun Minh N. H. Nguyen Choong Seon Hong
Multimodal federated learning (MFL) has emerged as a decentralized machine
learning paradigm, allowing multiple clients with different modalities to
collaborate on training a machine learning model across diverse data sources
without sharing their private data. However, challenges, such as data
heterogeneity and severely missing modalities, pose crucial hindrances to the
robustness of MFL, significantly impacting the performance of global model. The
absence of a modality introduces misalignment during the local training phase,
stemming from zero-filling in the case of clients with missing modalities.
Consequently, achieving robust generalization in global model becomes
imperative, especially when dealing with clients that have incomplete data. In
this paper, we propose Multimodal Federated Cross Prototype Learning (MFCPL), a
novel approach for MFL under severely missing modalities by conducting the
complete prototypes to provide diverse modality knowledge in modality-shared
level with the cross-modal regularization and modality-specific level with
cross-modal contrastive mechanism. Additionally, our approach introduces the
cross-modal alignment to provide regularization for modality-specific features,
thereby enhancing overall performance, particularly in scenarios involving
severely missing modalities. Through extensive experiments on three multimodal
datasets, we demonstrate the effectiveness of MFCPL in mitigating these
challenges and improving the overall performance.