Lightweight Self-Knowledge Distillation with Multi-source Information
  Fusion
Xucong Wang Pengchao Han Lei Guo
Knowledge Distillation (KD) is a powerful technique for transferring
knowledge between neural network models, where a pre-trained teacher model is
used to facilitate the training of the target student model. However, the
availability of a suitable teacher model is not always guaranteed. To address
this challenge, Self-Knowledge Distillation (SKD) attempts to construct a
teacher model from itself. Existing SKD methods add Auxiliary Classifiers (AC)
to intermediate layers of the model or use the history models and models with
different input data within the same class. However, these methods are
computationally expensive and only capture time-wise and class-wise features of
data. In this paper, we propose a lightweight SKD framework that utilizes
multi-source information to construct a more informative teacher. Specifically,
we introduce a Distillation with Reverse Guidance (DRG) method that considers
different levels of information extracted by the model, including edge, shape,
and detail of the input data, to construct a more informative teacher.
Additionally, we design a Distillation with Shape-wise Regularization (DSR)
method that ensures a consistent shape of ranked model output for all data. We
validate the performance of the proposed DRG, DSR, and their combination
through comprehensive experiments on various datasets and models. Our results
demonstrate the superiority of the proposed methods over baselines (up to
2.87%) and state-of-the-art SKD methods (up to 1.15%), while being
computationally efficient and robust. The code is available at
https://github.com/xucong-parsifal/LightSKD.