Distilling Adversarial Robustness Using Heterogeneous Teachers
Jieren Deng Aaron Palmer Rigel Mahmood Ethan Rathbun Jinbo Bi Kaleel Mahmood Derek Aguiar
Achieving resiliency against adversarial attacks is necessary prior to
deploying neural network classifiers in domains where misclassification incurs
substantial costs, e.g., self-driving cars or medical imaging. Recent work has
demonstrated that robustness can be transferred from an adversarially trained
teacher to a student model using knowledge distillation. However, current
methods perform distillation using a single adversarial and vanilla teacher and
consider homogeneous architectures (i.e., residual networks) that are
susceptible to misclassify examples from similar adversarial subspaces. In this
work, we develop a defense framework against adversarial attacks by distilling
adversarial robustness using heterogeneous teachers (DARHT). In DARHT, the
student model explicitly represents teacher logits in a student-teacher feature
map and leverages multiple teachers that exhibit low adversarial example
transferability (i.e., exhibit high performance on dissimilar adversarial
examples). Experiments on classification tasks in both white-box and black-box
scenarios demonstrate that DARHT achieves state-of-the-art clean and robust
accuracies when compared to competing adversarial training and distillation
methods in the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. Comparisons
with homogeneous and heterogeneous teacher sets suggest that leveraging
teachers with low adversarial example transferability increases student model
robustness.