Fast FixMatch: Faster Semi-Supervised Learning with Curriculum Batch
  Size
John Chen Chen Dun Anastasios Kyrillidis
Advances in Semi-Supervised Learning (SSL) have almost entirely closed the
gap between SSL and Supervised Learning at a fraction of the number of labels.
However, recent performance improvements have often come \textit{at the cost of
significantly increased training computation}. To address this, we propose
Curriculum Batch Size (CBS), \textit{an unlabeled batch size curriculum which
exploits the natural training dynamics of deep neural networks.} A small
unlabeled batch size is used in the beginning of training and is gradually
increased to the end of training. A fixed curriculum is used regardless of
dataset, model or number of epochs, and reduced training computations is
demonstrated on all settings. We apply CBS, strong labeled augmentation,
Curriculum Pseudo Labeling (CPL) \citep{FlexMatch} to FixMatch \citep{FixMatch}
and term the new SSL algorithm Fast FixMatch. We perform an ablation study to
show that strong labeled augmentation and/or CPL do not significantly reduce
training computations, but, in synergy with CBS, they achieve optimal
performance. Fast FixMatch also achieves substantially higher data utilization
compared to previous state-of-the-art. Fast FixMatch achieves between
$2.1\times$ - $3.4\times$ reduced training computations on CIFAR-10 with all
but 40, 250 and 4000 labels removed, compared to vanilla FixMatch, while
attaining the same cited state-of-the-art error rate \citep{FixMatch}. Similar
results are achieved for CIFAR-100, SVHN and STL-10. Finally, Fast MixMatch
achieves between $2.6\times$ - $3.3\times$ reduced training computations in
federated SSL tasks and online/streaming learning SSL tasks, which further
demonstrate the generializbility of Fast MixMatch to different scenarios and
tasks.