Customizing Synthetic Data for Data-Free Student Learning
Shiya Luo Defang Chen Can Wang
Data-free knowledge distillation (DFKD) aims to obtain a lightweight student
model without original training data. Existing works generally synthesize data
from the pre-trained teacher model to replace the original training data for
student learning. To more effectively train the student model, the synthetic
data shall be customized to the current student learning ability. However, this
is ignored in the existing DFKD methods and thus negatively affects the student
training. To address this issue, we propose Customizing Synthetic Data for
Data-Free Student Learning (CSD) in this paper, which achieves adaptive data
synthesis using a self-supervised augmented auxiliary task to estimate the
student learning ability. Specifically, data synthesis is dynamically adjusted
to enlarge the cross entropy between the labels and the predictions from the
self-supervised augmented task, thus generating hard samples for the student
model. The experiments on various datasets and teacher-student models show the
effectiveness of our proposed method. Code is available at:
$\href{https://github.com/luoshiya/CSD}{https://github.com/luoshiya/CSD}$