I2CKD : Intra- and Inter-Class Knowledge Distillation for Semantic
  Segmentation
Ayoub Karine Thibault Napoléon Maher Jridi
This paper proposes a new knowledge distillation method tailored for image
semantic segmentation, termed Intra- and Inter-Class Knowledge Distillation
(I2CKD). The focus of this method is on capturing and transferring knowledge
between the intermediate layers of teacher (cumbersome model) and student
(compact model). For knowledge extraction, we exploit class prototypes derived
from feature maps. To facilitate knowledge transfer, we employ a triplet loss
in order to minimize intra-class variances and maximize inter-class variances
between teacher and student prototypes. Consequently, I2CKD enables the student
to better mimic the feature representation of the teacher for each class,
thereby enhancing the segmentation performance of the compact network.
Extensive experiments on three segmentation datasets, i.e., Cityscapes, Pascal
VOC and CamVid, using various teacher-student network pairs demonstrate the
effectiveness of the proposed method.