RdimKD: Generic Distillation Paradigm by Dimensionality Reduction
Yi Guo Yiqian He Xiaoyang Li Haotong Qin Van Tung Pham Yang Zhang Shouda Liu
Knowledge Distillation (KD) emerges as one of the most promising compression
technologies to run advanced deep neural networks on resource-limited devices.
In order to train a small network (student) under the guidance of a large
network (teacher), the intuitive method is regularizing the feature maps or
logits of the student using the teacher's information. However, existing
methods either over-restrict the student to learn all information from the
teacher, which lead to some bad local minimum, or use various fancy and
elaborate modules to process and align features, which are complex and lack
generality. In this work, we proposed an abstract and general paradigm for the
KD task, referred to as DIMensionality Reduction KD (RdimKD), which solely
relies on dimensionality reduction, with a very minor modification to naive L2
loss. RdimKD straightforwardly utilizes a projection matrix to project both the
teacher's and student's feature maps onto a low-dimensional subspace, which are
then optimized during training. RdimKD achieves the goal in the simplest way
that not only does the student get valuable information from the teacher, but
it also ensures sufficient flexibility to adapt to the student's low-capacity
reality. Our extensive empirical findings indicate the effectiveness of RdimKD
across various learning tasks and diverse network architectures.