Boosting Semi-Supervised Learning by bridging high and low-confidence
  predictions
Khanh-Binh Nguyen Joon-Sung Yang
Pseudo-labeling is a crucial technique in semi-supervised learning (SSL),
where artificial labels are generated for unlabeled data by a trained model,
allowing for the simultaneous training of labeled and unlabeled data in a
supervised setting. However, several studies have identified three main issues
with pseudo-labeling-based approaches. Firstly, these methods heavily rely on
predictions from the trained model, which may not always be accurate, leading
to a confirmation bias problem. Secondly, the trained model may be overfitted
to easy-to-learn examples, ignoring hard-to-learn ones, resulting in the
\textit{"Matthew effect"} where the already strong become stronger and the weak
weaker. Thirdly, most of the low-confidence predictions of unlabeled data are
discarded due to the use of a high threshold, leading to an underutilization of
unlabeled data during training. To address these issues, we propose a new
method called ReFixMatch, which aims to utilize all of the unlabeled data
during training, thus improving the generalizability of the model and
performance on SSL benchmarks. Notably, ReFixMatch achieves 41.05\% top-1
accuracy with 100k labeled examples on ImageNet, outperforming the baseline
FixMatch and current state-of-the-art methods.