Adversarial Sparse Teacher: Defense Against Distillation-Based Model
  Stealing Attacks Using Adversarial Examples
Eda Yilmaz Hacer Yalim Keles
Knowledge Distillation (KD) facilitates the transfer of discriminative
capabilities from an advanced teacher model to a simpler student model,
ensuring performance enhancement without compromising accuracy. It is also
exploited for model stealing attacks, where adversaries use KD to mimic the
functionality of a teacher model. Recent developments in this domain have been
influenced by the Stingy Teacher model, which provided empirical analysis
showing that sparse outputs can significantly degrade the performance of
student models. Addressing the risk of intellectual property leakage, our work
introduces an approach to train a teacher model that inherently protects its
logits, influenced by the Nasty Teacher concept. Differing from existing
methods, we incorporate sparse outputs of adversarial examples with standard
training data to strengthen the teacher's defense against student distillation.
Our approach carefully reduces the relative entropy between the original and
adversarially perturbed outputs, allowing the model to produce adversarial
logits with minimal impact on overall performance. The source codes will be
made publicly available soon.