Convolutional Neural Networks (CNNs) become deeper and deeper in recent years, making the study of model acceleration imperative. It is a common practice to employ a shallow network, called student, to learn from a deep one, which is termed as teacher. Prior work made many attempts to transfer different types of knowledge from teacher to student, however, there are two problems remaining unsolved. Firstly, the knowledge used by existing methods is highly dependent on task and dataset, limiting their applications. Secondly, there lacks an effective training scheme for the transfer process, leading to degradation of performance. In this work, we argue that feature is the most important knowledge from teacher. It is sufficient for student to just learn good features regardless of the target task. From this discovery, we further present an efficient learning strategy to mimic features stage by stage. Extensive experiments demonstrate the importance of features and show that the proposed approach significantly narrows down the gap between student and teacher, outperforming the state-of-the-art methods.
 Over the past few years, Convolutional Neural Networks (CNNs) have advanced various tasks in computer vision field, such as image classification BID8 , object detection BID17 , semantic segmentation BID3 , etc. However, along with the architecture growing deeper BID12 BID20 BID5 , the great success of CNN is at the cost of large computational power, which can not be afforded by most devices in practice. Some lightweight models are presented by recent work BID7 to reduce the computing cost especially for mobile devices, but the performance drops severely compared with the state-of-the-art methods. Accordingly, it is crucial to balance the trade-off between efficiency and capability of a CNN model.To tackle this problem, knowledge distillation is introduced in BID6 for model acceleration. The core idea is to train shallow networks (student) to mimic deep ones (teacher) following two folds. First, teacher employs a very deep model to achieve satisfying performance by excavating information (knowledge) from labeled data. Second, student learns the knowledge from teacher with a shallow model to speed up without losing much accuracy. Accordingly, the main challenges, corresponding to the above two steps respectively, lie in (1) what kind of knowledge should be transferred to student, and (2) how to transfer the knowledge from teacher to student as much as possible.For the first issue, previous work usually make student to learn from both teacher and original labeled data. There are mainly two problems in doing so. On one hand, it is very sensitive to tasks and datasets. The hyper-parameters, e.g. the loss weights to balance these two objective functions, require careful adjustment, or otherwise, it may cause severe performance degradation. On the other hand, the purposes to learn from teacher and to learn from ground-truth are not always consistent with each other. For example, teacher model may eliminate some label errors during the training process. In this case, trying to minimize the loss to mimic teacher as well as the loss from target task may cause confusions to student. Furthermore, prior work has also characterized various types of knowledge from teacher model for student to learn, such as attention map BID23 , information flow BID22 , etc. However, all types of knowledge are manually defined, which may not fully conform with the information contained in the teacher network. In other words, teacher is trained independently from these handcraft definitions, but is required to guide the student with such knowledge, which may cause some ambiguities. For the second issue, previous approaches do not solve the problem caused by the gaps between the learning abilities of student and teacher. Intuitively, student model has much less parameters compared to teacher, resulting in lower representation capability. Training it from scratch may always lead to poor performance.In this paper, we address these weaknesses by proposing a task independent knowledge transfer approach, where student is trained to mimic features from teacher stage by stage. Here, for simplicity, we do not distinguish between feature and feature map. It has two appealing properties.First, we isolate the knowledge contained in teacher model from the information provided by ground-truth. This goal is achieved with two phases. In the first phase, student learns knowledge by mimicking the output features of teacher, while in the second phase, student is trained with task dependent objection function based on the features from first phase. In this way, student can focus on acquiring information from only one source in each phase, making the transfer process more accurate. Separating these two phases apart also makes our method more generic to various tasks. Besides, we directly treat features as knowledge in this work. Since teacher model just uses features for inference in practice, they are expected to contain the compete information extracted by teacher from training data.Second, instead of training all parameters of student together, we divide the transfer process into different stages and only train a sub-network at one time. Student network has far more limited representation ability than teacher, resulting in the huge difficulty to mimic the final features directly. To alleviate such obstacle, we let the student to learn from teacher gradually. In other words, both teacher network and student network are separated into sequential parts. Then, in each stage, one part of student will be trained to mimic the output of the corresponding part of teacher with all previous parts fixed. In doing so, the gap between learning powers between student and teacher is narrowed down. As long as each stage is well trained, they will finally collaborate to achieve appealing results.To summarize, the contributions of this work are as follows:• We demonstrate the effectiveness of mimicking features directly in task independent knowledge transfer.• We present a stage-by-stage training strategy to learn features accurately and efficiently.• We show experimentally that our approach surpasses the state-of-the-art methods on various tasks with higher performance and stronger stability. This work presents a stage-by-stage knowledge transfer approach by training student to mimic the output features of teacher network gradually. Compared to prior work, our method pays more attention to the information contained in the model, regardless of what task the model is applied for, making it a generic solution for model acceleration. The progressive training strategy helps reduce the learning difficulties of student in each stage, and all stages cooperate together for a better result.Extensive experimental results suggest that our scheme can significantly improve the performance of student model on various tasks with strong stability.
This paper proposes to transfer knowledge from deep model to shallow one by mimicking features stage by stage. Explains a stage by stage knowledge transer method by using different structures of resnets This paper proposes dividing a network into multiple parts and distilling each part sequentially to improve distillation performance in deep teacher networks