One-Shot Sequential Federated Learning for Non-IID Data by Enhancing
  Local Model Diversity
Naibo Wang Yuchen Deng Wenjie Feng Shichen Fan Jianwei Yin See-Kiong Ng
Traditional federated learning mainly focuses on parallel settings (PFL),
which can suffer significant communication and computation costs. In contrast,
one-shot and sequential federated learning (SFL) have emerged as innovative
paradigms to alleviate these costs. However, the issue of non-IID (Independent
and Identically Distributed) data persists as a significant challenge in
one-shot and SFL settings, exacerbated by the restricted communication between
clients. In this paper, we improve the one-shot sequential federated learning
for non-IID data by proposing a local model diversity-enhancing strategy.
Specifically, to leverage the potential of local model diversity for improving
model performance, we introduce a local model pool for each client that
comprises diverse models generated during local training, and propose two
distance measurements to further enhance the model diversity and mitigate the
effect of non-IID data. Consequently, our proposed framework can improve the
global model performance while maintaining low communication costs. Extensive
experiments demonstrate that our method exhibits superior performance to
existing one-shot PFL methods and achieves better accuracy compared with
state-of-the-art one-shot SFL methods on both label-skew and domain-shift tasks
(e.g., 6%+ accuracy improvement on the CIFAR-10 dataset).