Cumulative Spatial Knowledge Distillation for Vision Transformers
Borui Zhao Renjie Song Jiajun Liang
Distilling knowledge from convolutional neural networks (CNNs) is a
double-edged sword for vision transformers (ViTs). It boosts the performance
since the image-friendly local-inductive bias of CNN helps ViT learn faster and
better, but leading to two problems: (1) Network designs of CNN and ViT are
completely different, which leads to different semantic levels of intermediate
features, making spatial-wise knowledge transfer methods (e.g., feature
mimicking) inefficient. (2) Distilling knowledge from CNN limits the network
convergence in the later training period since ViT's capability of integrating
global information is suppressed by CNN's local-inductive-bias supervision. To
this end, we present Cumulative Spatial Knowledge Distillation (CSKD). CSKD
distills spatial-wise knowledge to all patch tokens of ViT from the
corresponding spatial responses of CNN, without introducing intermediate
features. Furthermore, CSKD exploits a Cumulative Knowledge Fusion (CKF)
module, which introduces the global response of CNN and increasingly emphasizes
its importance during the training. Applying CKF leverages CNN's local
inductive bias in the early training period and gives full play to ViT's global
capability in the later one. Extensive experiments and analysis on ImageNet-1k
and downstream datasets demonstrate the superiority of our CSKD. Code will be
publicly available.