Knowledge distillation is a potential solution for model compression. The idea is to make a small student network imitate the target of a large teacher network, then the student network can be competitive to the teacher one. Most previous studies focus on model distillation in the classification task, where they propose different architectures and initializations for the student network. However, only the classification task is not enough, and other related tasks such as regression and retrieval are barely considered. To solve the problem, in this paper, we take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification. By selecting appropriate initializations and targets in the knowledge transfer, the distillation can be easier in non-classification tasks. Experiments on the CelebA and CASIA-WebFace datasets demonstrate that the student network can be competitive to the teacher one in alignment and verification, and even surpasses the teacher network under specific compression rates. In addition, to achieve stronger knowledge transfer, we also use a common initialization trick to improve the distillation performance of classification. Evaluations on the CASIA-Webface and large-scale MS-Celeb-1M datasets show the effectiveness of this simple trick. Since the emergence of Alexnet BID12 , larger and deeper networks have shown to be more powerful BID22 . However, as the network going larger and deeper, it becomes difficult to use it in mobile devices. Therefore, model compression has become necessary in compressing the large network into a small one. In recent years, many compression methods have been proposed, including knowledge distillation BID0 BID8 BID19 , weight quantization BID4 BID17 , weight pruning BID6 BID24 and weight decomposition BID2 BID16 . In this paper, we focus on the knowledge distillation, which is a potential approach for model compression.In knowledge distillation, there is usually a large teacher network and a small student one, and the objective is to make the student network competitive to the teacher one by learning specific targets of the teacher network. Previous studies mainly consider the selection of targets in the classification task, e.g., hidden layers BID15 , logits BID0 BID25 BID20 or soft predictions BID8 BID19 . However, only the distillation of the classification task is not enough, and some common tasks such as regression and retrieval should also be considered. In this paper, we take face recognition as a breaking point that we start with the knowledge distillation in face classification, and consider the distillation on two domain-similar tasks, including face alignment and verification. The objective of face alignment is to locate the key-point locations in each image; while in face verification, we have to determine if two images belong to the same identity.For distillation on non-classification tasks, one intuitive idea is to adopt a similar method as in face classification that trains teacher and student networks from scratch. In this way, the distillation on all tasks will be independent, and this is a possible solution. However, this independence cannot give the best distillation performance. There has been strong evidence that in object detection BID18 , object segmentation BID3 and image retrieval BID30 , they all used the pretrained classification model(on ImageNet) as initialization to boost performance. This success comes from the fact that their domains are similar, which makes them transfer a lot from low-level to high-level representation BID29 . Similarly, face classification, alignment and verification also share the similar domain, thus we propose to transfer the distilled knowledge of classification by taking its teacher and student networks to initialize corresponding networks in alignment and verification.Another problem in knowledge transfer is what targets should be used for distillation? In face classification, the knowledge is distilled from the teacher network by learning its soft-prediction, which has been proved to work well BID8 BID19 . However, in face alignment BID27 and verification BID27 , they have additional task-specific targets. As a result, selecting the classification or task-specific target for distillation remains a problem. One intuitive idea is to measure the relevance of objectives between non-classification and classification tasks. For example, it is not obvious to see the relation between face classification and alignment, but the classification can help a lot in verification. Therefore, it seems reasonable that if the tasks are highly related, the classification target is preferred, or the task-specific target is better.Inspired by the above thoughts, in this paper, we propose the model distillation in face alignment and verification by transferring the distilled knowledge from face classification. With appropriate selection of initializations and targets, we show that the distillation performance of alignment and verification on the CelebA and CASIA-WebFace BID28 datasets can be largely improved, and the student network can even exceed the teacher network under specific compression rates. This knowledge transfer is our main contribution. In addition, we realize that in the proposed method, the knowledge transfer depends on the distillation of classification, thus we use a common initialization trick to further boost the distillation performance of classification. Evaluations on the CASIA-WebFace and large-scale MS-Celeb-1M BID5 datasets show that this simple trick can give the best distillation results in the classification task. In this paper, we take face recognition as a breaking point, and propose the knowledge distillation on two non-classification tasks, including face alignment and verification. We extend the previous distillation framework by transferring the distilled knowledge from face classification to face alignment and verification. By selecting appropriate initializations and targets, the distillation on non-classification tasks can be easier. Besides, we also give some guidelines for target selection on non-classification tasks, and we hope these guidelines can be helpful for more tasks. Experiments on the datasets of CASIA-WebFace, CelebA and large-scale MS-Celeb-1M have demonstrated the effectiveness of the proposed method, which gives the student networks that can be competitive or exceed the teacher network under appropriate compression rates. In addition, we use a common initialization trick to further improve the distillation performance of classification, and this can boost the distillation on non-classification tasks. Experiments on CASIA-WebFace have demonstrated the effectiveness of this simple trick.
We take face recognition as a breaking point and propose model distillation with knowledge transfer from face classification to alignment and verification This paper proposes to transfer the classifier from the model for face classification to the task of alignment and verification. The manuscript presents experiments on distilling knowledge from a face classification model to student models for face alignment and verification.