Robust Training of Federated Models with Extremely Label Deficiency
Yonggang Zhang Zhiqin Yang Xinmei Tian Nannan Wang Tongliang Liu Bo Han
Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm
for collaboratively training machine learning models using distributed data
with label deficiency. Advanced FSSL methods predominantly focus on training a
single model on each client. However, this approach could lead to a discrepancy
between the objective functions of labeled and unlabeled data, resulting in
gradient conflicts. To alleviate gradient conflict, we propose a novel
twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by
providing insights from different perspectives of labeled and unlabeled data.
In particular, Twin-sight concurrently trains a supervised model with a
supervised objective function while training an unsupervised model using an
unsupervised objective function. To enhance the synergy between these two
models, Twin-sight introduces a neighbourhood-preserving constraint, which
encourages the preservation of the neighbourhood relationship among data
features extracted by both models. Our comprehensive experiments on four
benchmark datasets provide substantial evidence that Twin-sight can
significantly outperform state-of-the-art methods across various experimental
settings, demonstrating the efficacy of the proposed Twin-sight.