DynamicKD: An Effective Knowledge Distillation via Dynamic Entropy
  Correction-Based Distillation for Gap Optimizing
Songling Zhu Ronghua Shang Bo Yuan Weitong Zhang Yangyang Li Licheng Jiao
The knowledge distillation uses a high-performance teacher network to guide
the student network. However, the performance gap between the teacher and
student networks can affect the student's training. This paper proposes a novel
knowledge distillation algorithm based on dynamic entropy correction to reduce
the gap by adjusting the student instead of the teacher. Firstly, the effect of
changing the output entropy (short for output information entropy) in the
student on the distillation loss is analyzed in theory. This paper shows that
correcting the output entropy can reduce the gap. Then, a knowledge
distillation algorithm based on dynamic entropy correction is created, which
can correct the output entropy in real-time with an entropy controller updated
dynamically by the distillation loss. The proposed algorithm is validated on
the CIFAR100 and ImageNet. The comparison with various state-of-the-art
distillation algorithms shows impressive results, especially in the experiment
on the CIFAR100 regarding teacher-student pair resnet32x4-resnet8x4. The
proposed algorithm raises 2.64 points over the traditional distillation
algorithm and 0.87 points over the state-of-the-art algorithm CRD in
classification accuracy, demonstrating its effectiveness and efficiency.