Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL
  Shader Images
Logan Frank Jim Davis
Knowledge distillation (KD) has been a popular and effective method for model
compression. One important assumption of KD is that the original training
dataset is always available. However, this is not always the case due to
privacy concerns and more. In recent years, "data-free" KD has emerged as a
growing research topic which focuses on the scenario of performing KD when no
data is provided. Many methods rely on a generator network to synthesize
examples for distillation (which can be difficult to train) and can frequently
produce images that are visually similar to the original dataset, which raises
questions surrounding whether privacy is completely preserved. In this work, we
propose a new approach to data-free KD that utilizes unnatural OpenGL images,
combined with large amounts of data augmentation and adversarial attacks, to
train a student network. We demonstrate that our approach achieves
state-of-the-art results for a variety of datasets/networks and is more stable
than existing generator-based data-free KD methods. Source code will be
available in the future.