Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge
  Distillation
Rongzhi Zhang Jiaming Shen Tianqi Liu Jialu Liu Michael Bendersky Marc Najork Chao Zhang
Knowledge distillation is a popular technique to transfer knowledge from
large teacher models to a small student model. Typically, the student learns to
imitate the teacher by minimizing the KL divergence of its output distribution
with the teacher's output distribution. In this work, we argue that such a
learning objective is sub-optimal because there exists a discrepancy between
the teacher's output distribution and the ground truth label distribution.
Therefore, forcing the student to blindly imitate the unreliable teacher output
distribution leads to inferior performance. To this end, we propose a novel
knowledge distillation objective PTLoss by first representing the vanilla
KL-based distillation loss function via a Maclaurin series and then perturbing
the leading-order terms in this series. This perturbed loss implicitly
transforms the original teacher into a proxy teacher with a distribution closer
to the ground truth distribution. We establish the theoretical connection
between this "distribution closeness" and the student model generalizability,
which enables us to select the PTLoss's perturbation coefficients in a
principled way. Extensive experiments on five datasets demonstrate PTLoss can
significantly improve the distillation effectiveness for teachers of various
scales.