Sampling to Distill: Knowledge Transfer from Open-World Data
Yuzheng Wang Zhaoyu Chen Jie Zhang Dingkang Yang Zuhao Ge Yang Liu Siao Liu Yunquan Sun Wenqiang Zhang Lizhe Qi
Data-Free Knowledge Distillation (DFKD) is a novel task that aims to train
high-performance student models using only the teacher network without original
training data. Despite encouraging results, existing DFKD methods rely heavily
on generation modules with high computational costs. Meanwhile, they ignore the
fact that the generated and original data exist domain shifts due to the lack
of supervision information. Moreover, knowledge is transferred through each
example, ignoring the implicit relationship among multiple examples. To this
end, we propose a novel Open-world Data Sampling Distillation (ODSD) method
without a redundant generation process. First, we try to sample open-world data
close to the original data's distribution by an adaptive sampling module. Then,
we introduce a low-noise representation to alleviate the domain shifts and
build a structured relationship of multiple data examples to exploit data
knowledge. Extensive experiments on CIFAR-10, CIFAR-100, NYUv2, and ImageNet
show that our ODSD method achieves state-of-the-art performance. Especially, we
improve 1.50\%-9.59\% accuracy on the ImageNet dataset compared with the
existing results.