Local or Global: Selective Knowledge Assimilation for Federated Learning
  with Limited Labels
Yae Jee Cho Gauri Joshi Dimitrios Dimitriadis
Many existing FL methods assume clients with fully-labeled data, while in
realistic settings, clients have limited labels due to the expensive and
laborious process of labeling. Limited labeled local data of the clients often
leads to their local model having poor generalization abilities to their larger
unlabeled local data, such as having class-distribution mismatch with the
unlabeled data. As a result, clients may instead look to benefit from the
global model trained across clients to leverage their unlabeled data, but this
also becomes difficult due to data heterogeneity across clients. In our work,
we propose FedLabel where clients selectively choose the local or global model
to pseudo-label their unlabeled data depending on which is more of an expert of
the data. We further utilize both the local and global models' knowledge via
global-local consistency regularization which minimizes the divergence between
the two models' outputs when they have identical pseudo-labels for the
unlabeled data. Unlike other semi-supervised FL baselines, our method does not
require additional experts other than the local or global model, nor require
additional parameters to be communicated. We also do not assume any
server-labeled data or fully labeled clients. For both cross-device and
cross-silo settings, we show that FedLabel outperforms other semi-supervised FL
baselines by $8$-$24\%$, and even outperforms standard fully supervised FL
baselines ($100\%$ labeled data) with only $5$-$20\%$ of labeled data.