Decoupled Knowledge with Ensemble Learning for Online Distillation
Baitan Shao Ying Chen
Offline distillation is a two-stage pipeline that requires expensive
resources to train a teacher network and then distill the knowledge to a
student for deployment. Online knowledge distillation, on the other hand, is a
one-stage strategy that alleviates the requirement with mutual learning and
collaborative learning. Recent peer collaborative learning (PCL) integrates
online ensemble, collaboration of base networks and temporal mean teacher to
construct effective knowledge. However, the model collapses occasionally in PCL
due to high homogenization between the student and the teacher. In this paper,
the cause of the high homogenization is analyzed and the solution is presented.
A decoupled knowledge for online knowledge distillation is generated by an
independent teacher, separate from the student. Such design can increase the
diversity between the networks and reduce the possibility of model collapse. To
obtain early decoupled knowledge, an initialization scheme for the teacher is
devised, and a 2D geometry-based analysis experiment is conducted under ideal
conditions to showcase the effectiveness of this scheme. Moreover, to improve
the teacher's supervisory resilience, a decaying ensemble scheme is devised. It
assembles the knowledge of the teacher to which a dynamic weight which is large
at the start of the training and gradually decreases with the training process
is assigned. The assembled knowledge serves as a strong teacher during the
early training and the decreased-weight-assembled knowledge can eliminate the
distribution deviation under the potentially overfitted teacher's supervision.
A Monte Carlo-based simulation is conducted to evaluate the convergence.
Extensive experiments on CIFAR-10, CIFAR-100 and TinyImageNet show the
superiority of our method. Ablation studies and further analysis demonstrate
the effectiveness.