Client-wise Modality Selection for Balanced Multi-modal Federated
  Learning
Yunfeng Fan Wenchao Xu Haozhao Wang Penghui Ruan Song Guo
Selecting proper clients to participate in the iterative federated learning
(FL) rounds is critical to effectively harness a broad range of distributed
datasets. Existing client selection methods simply consider the variability
among FL clients with uni-modal data, however, have yet to consider clients
with multi-modalities. We reveal that traditional client selection scheme in
MFL may suffer from a severe modality-level bias, which impedes the
collaborative exploitation of multi-modal data, leading to insufficient local
data exploration and global aggregation. To tackle this challenge, we propose a
Client-wise Modality Selection scheme for MFL (CMSFed) that can comprehensively
utilize information from each modality via avoiding such client selection bias
caused by modality imbalance. Specifically, in each MFL round, the local data
from different modalities are selectively employed to participate in local
training and aggregation to mitigate potential modality imbalance of the global
model. To approximate the fully aggregated model update in a balanced way, we
introduce a novel local training loss function to enhance the weak modality and
align the divergent feature spaces caused by inconsistent modality adoption
strategies for different clients simultaneously. Then, a modality-level
gradient decoupling method is designed to derive respective submodular
functions to maintain the gradient diversity during the selection progress and
balance MFL according to local modality imbalance in each iteration. Our
extensive experiments showcase the superiority of CMSFed over baselines and its
effectiveness in multi-modal data exploitation.