Cross Architecture Distillation for Face Recognition
Weisong Zhao Xiangyu Zhu Zhixiang He Xiao-Yu Zhang Zhen Lei
Transformers have emerged as the superior choice for face recognition tasks,
but their insufficient platform acceleration hinders their application on
mobile devices. In contrast, Convolutional Neural Networks (CNNs) capitalize on
hardware-compatible acceleration libraries. Consequently, it has become
indispensable to preserve the distillation efficacy when transferring knowledge
from a Transformer-based teacher model to a CNN-based student model, known as
Cross-Architecture Knowledge Distillation (CAKD). Despite its potential, the
deployment of CAKD in face recognition encounters two challenges: 1) the
teacher and student share disparate spatial information for each pixel,
obstructing the alignment of feature space, and 2) the teacher network is not
trained in the role of a teacher, lacking proficiency in handling
distillation-specific knowledge. To surmount these two constraints, 1) we first
introduce a Unified Receptive Fields Mapping module (URFM) that maps pixel
features of the teacher and student into local features with unified receptive
fields, thereby synchronizing the pixel-wise spatial information of teacher and
student. Subsequently, 2) we develop an Adaptable Prompting Teacher network
(APT) that integrates prompts into the teacher, enabling it to manage
distillation-specific knowledge while preserving the model's discriminative
capacity. Extensive experiments on popular face benchmarks and two large-scale
verification sets demonstrate the superiority of our method.