SequenceMatch: Revisiting the design of weak-strong augmentations for
  Semi-supervised learning
Khanh-Binh Nguyen
Semi-supervised learning (SSL) has become popular in recent years because it
allows the training of a model using a large amount of unlabeled data. However,
one issue that many SSL methods face is the confirmation bias, which occurs
when the model is overfitted to the small labeled training dataset and produces
overconfident, incorrect predictions. To address this issue, we propose
SequenceMatch, an efficient SSL method that utilizes multiple data
augmentations. The key element of SequenceMatch is the inclusion of a medium
augmentation for unlabeled data. By taking advantage of different augmentations
and the consistency constraints between each pair of augmented examples,
SequenceMatch helps reduce the divergence between the prediction distribution
of the model for weakly and strongly augmented examples. In addition,
SequenceMatch defines two different consistency constraints for high and
low-confidence predictions. As a result, SequenceMatch is more data-efficient
than ReMixMatch, and more time-efficient than both ReMixMatch ($\times4$) and
CoMatch ($\times2$) while having higher accuracy. Despite its simplicity,
SequenceMatch consistently outperforms prior methods on standard benchmarks,
such as CIFAR-10/100, SVHN, and STL-10. It also surpasses prior
state-of-the-art methods by a large margin on large-scale datasets such as
ImageNet, with a 38.46\% error rate. Code is available at
https://github.com/beandkay/SequenceMatch.