Knowledge Distillation Layer that Lets the Student Decide
Ada Gorgun Yeti Z. Gurbuz A. Aydin Alatan
Typical technique in knowledge distillation (KD) is regularizing the learning
of a limited capacity model (student) by pushing its responses to match a
powerful model's (teacher). Albeit useful especially in the penultimate layer
and beyond, its action on student's feature transform is rather implicit,
limiting its practice in the intermediate layers. To explicitly embed the
teacher's knowledge in feature transform, we propose a learnable KD layer for
the student which improves KD with two distinct abilities: i) learning how to
leverage the teacher's knowledge, enabling to discard nuisance information, and
ii) feeding forward the transferred knowledge deeper. Thus, the student enjoys
the teacher's knowledge during the inference besides training. Formally, we
repurpose 1x1-BN-ReLU-1x1 convolution block to assign a semantic vector to each
local region according to the template (supervised by the teacher) that the
corresponding region of the student matches. To facilitate template learning in
the intermediate layers, we propose a novel form of supervision based on the
teacher's decisions. Through rigorous experimentation, we demonstrate the
effectiveness of our approach on 3 popular classification benchmarks. Code is
available at: https://github.com/adagorgun/letKD-framework