Not all Minorities are Equal: Empty-Class-Aware Distillation for
  Heterogeneous Federated Learning
Kuangpu Guo Yuhe Ding Jian Liang Ran He Zilei Wang Tieniu Tan
Data heterogeneity, characterized by disparities in local data distribution
across clients, poses a significant challenge in federated learning.
Substantial efforts have been devoted to addressing the heterogeneity in local
label distribution. As minority classes suffer from worse accuracy due to
overfitting on local imbalanced data, prior methods often incorporate
class-balanced learning techniques during local training. Despite the improved
mean accuracy across all classes, we observe that empty classes-referring to
categories absent from a client's data distribution-are still not well
recognized. This paper introduces FedED, a novel approach in heterogeneous
federated learning that integrates both empty-class distillation and logit
suppression simultaneously. Specifically, empty-class distillation leverages
knowledge distillation during local training on each client to retain essential
information related to empty classes from the global model. Moreover, logit
suppression directly penalizes network logits for non-label classes,
effectively addressing misclassifications in minority classes that may be
biased toward majority classes. Extensive experiments validate the efficacy of
FedED, surpassing previous state-of-the-art methods across diverse datasets
with varying degrees of label distribution shift.