Indirect Gradient Matching for Adversarial Robust Distillation
Hongsin Lee Seungju Cho Changick Kim
Adversarial training significantly improves adversarial robustness, but
superior performance is primarily attained with large models. This substantial
performance gap for smaller models has spurred active research into adversarial
distillation (AD) to mitigate the difference. Existing AD methods leverage the
teacher's logits as a guide. In contrast to these approaches, we aim to
transfer another piece of knowledge from the teacher, the input gradient. In
this paper, we propose a distillation module termed Indirect Gradient
Distillation Module (IGDM) that indirectly matches the student's input gradient
with that of the teacher. We hypothesize that students can better acquire the
teacher's knowledge by matching the input gradient. Leveraging the observation
that adversarial training renders the model locally linear on the input space,
we employ Taylor approximation to effectively align gradients without directly
calculating them. Experimental results show that IGDM seamlessly integrates
with existing AD methods, significantly enhancing the performance of all AD
methods. Particularly, utilizing IGDM on the CIFAR-100 dataset improves the
AutoAttack accuracy from 28.06% to 30.32% with the ResNet-18 model and from
26.18% to 29.52% with the MobileNetV2 model when integrated into the SOTA
method without additional data augmentation. The code will be made available.