Improving Defensive Distillation using Teacher Assistant
Maniratnam Mandal Suna Gao
Adversarial attacks pose a significant threat to the security and safety of
deep neural networks being applied to modern applications. More specifically,
in computer vision-based tasks, experts can use the knowledge of model
architecture to create adversarial samples imperceptible to the human eye.
These attacks can lead to security problems in popular applications such as
self-driving cars, face recognition, etc. Hence, building networks which are
robust to such attacks is highly desirable and essential. Among the various
methods present in literature, defensive distillation has shown promise in
recent years. Using knowledge distillation, researchers have been able to
create models robust against some of those attacks. However, more attacks have
been developed exposing weakness in defensive distillation. In this project, we
derive inspiration from teacher assistant knowledge distillation and propose
that introducing an assistant network can improve the robustness of the
distilled model. Through a series of experiments, we evaluate the distilled
models for different distillation temperatures in terms of accuracy,
sensitivity, and robustness. Our experiments demonstrate that the proposed
hypothesis can improve robustness in most cases. Additionally, we show that
multi-step distillation can further improve robustness with very little impact
on model accuracy.