Weight Averaging Improves Knowledge Distillation under Domain Shift
Valeriy Berezovskiy Nikita Morozov
Knowledge distillation (KD) is a powerful model compression technique broadly
used in practical deep learning applications. It is focused on training a small
student network to mimic a larger teacher network. While it is widely known
that KD can offer an improvement to student generalization in i.i.d setting,
its performance under domain shift, i.e. the performance of student networks on
data from domains unseen during training, has received little attention in the
literature. In this paper we make a step towards bridging the research fields
of knowledge distillation and domain generalization. We show that weight
averaging techniques proposed in domain generalization literature, such as SWAD
and SMA, also improve the performance of knowledge distillation under domain
shift. In addition, we propose a simplistic weight averaging strategy that does
not require evaluation on validation data during training and show that it
performs on par with SWAD and SMA when applied to KD. We name our final
distillation approach Weight-Averaged Knowledge Distillation (WAKD).