Towards Fast and Stable Federated Learning: Confronting Heterogeneity
  via Knowledge Anchor
Jinqian Chen Jihua Zhu Qinghai Zheng
Federated learning encounters a critical challenge of data heterogeneity,
adversely affecting the performance and convergence of the federated model.
Various approaches have been proposed to address this issue, yet their
effectiveness is still limited. Recent studies have revealed that the federated
model suffers severe forgetting in local training, leading to global forgetting
and performance degradation. Although the analysis provides valuable insights,
a comprehensive understanding of the vulnerable classes and their impact
factors is yet to be established. In this paper, we aim to bridge this gap by
systematically analyzing the forgetting degree of each class during local
training across different communication rounds. Our observations are: (1) Both
missing and non-dominant classes suffer similar severe forgetting during local
training, while dominant classes show improvement in performance. (2) When
dynamically reducing the sample size of a dominant class, catastrophic
forgetting occurs abruptly when the proportion of its samples is below a
certain threshold, indicating that the local model struggles to leverage a few
samples of a specific class effectively to prevent forgetting. Motivated by
these findings, we propose a novel and straightforward algorithm called
Federated Knowledge Anchor (FedKA). Assuming that all clients have a single
shared sample for each class, the knowledge anchor is constructed before each
local training stage by extracting shared samples for missing classes and
randomly selecting one sample per class for non-dominant classes. The knowledge
anchor is then utilized to correct the gradient of each mini-batch towards the
direction of preserving the knowledge of the missing and non-dominant classes.
Extensive experimental results demonstrate that our proposed FedKA achieves
fast and stable convergence, significantly improving accuracy on popular
benchmarks.