Enhancing Adversarial Robustness in Low-Label Regime via Adaptively
  Weighted Regularization and Knowledge Distillation
Dongyoon Yang Insung Kong Yongdai Kim
Adversarial robustness is a research area that has recently received a lot of
attention in the quest for trustworthy artificial intelligence. However, recent
works on adversarial robustness have focused on supervised learning where it is
assumed that labeled data is plentiful. In this paper, we investigate
semi-supervised adversarial training where labeled data is scarce. We derive
two upper bounds for the robust risk and propose a regularization term for
unlabeled data motivated by these two upper bounds. Then, we develop a
semi-supervised adversarial training algorithm that combines the proposed
regularization term with knowledge distillation using a semi-supervised teacher
(i.e., a teacher model trained using a semi-supervised learning algorithm). Our
experiments show that our proposed algorithm achieves state-of-the-art
performance with significant margins compared to existing algorithms. In
particular, compared to supervised learning algorithms, performance of our
proposed algorithm is not much worse even when the amount of labeled data is
very small. For example, our algorithm with only 8\% labeled data is comparable
to supervised adversarial training algorithms that use all labeled data, both
in terms of standard and robust accuracies on CIFAR-10.