KD-FixMatch: Knowledge Distillation Siamese Neural Networks
Chien-Chih Wang Shaoyuan Xu Jinmiao Fu Yang Liu Bryan Wang
Semi-supervised learning (SSL) has become a crucial approach in deep learning
as a way to address the challenge of limited labeled data. The success of deep
neural networks heavily relies on the availability of large-scale high-quality
labeled data. However, the process of data labeling is time-consuming and
unscalable, leading to shortages in labeled data. SSL aims to tackle this
problem by leveraging additional unlabeled data in the training process. One of
the popular SSL algorithms, FixMatch, trains identical weight-sharing teacher
and student networks simultaneously using a siamese neural network (SNN).
However, it is prone to performance degradation when the pseudo labels are
heavily noisy in the early training stage. We present KD-FixMatch, a novel SSL
algorithm that addresses the limitations of FixMatch by incorporating knowledge
distillation. The algorithm utilizes a combination of sequential and
simultaneous training of SNNs to enhance performance and reduce performance
degradation. Firstly, an outer SNN is trained using labeled and unlabeled data.
After that, the network of the well-trained outer SNN generates pseudo labels
for the unlabeled data, from which a subset of unlabeled data with trusted
pseudo labels is then carefully created through high-confidence sampling and
deep embedding clustering. Finally, an inner SNN is trained with the labeled
data, the unlabeled data, and the subset of unlabeled data with trusted pseudo
labels. Experiments on four public data sets demonstrate that KD-FixMatch
outperforms FixMatch in all cases. Our results indicate that KD-FixMatch has a
better training starting point that leads to improved model performance
compared to FixMatch.