Can pre-trained models assist in dataset distillation?
Yao Lu Xuguang Chen Yuchen Zhang Jianyang Gu Tianle Zhang Yifan Zhang Xiaoniu Yang Qi Xuan Kai Wang Yang You
Dataset Distillation (DD) is a prominent technique that encapsulates
knowledge from a large-scale original dataset into a small synthetic dataset
for efficient training. Meanwhile, Pre-trained Models (PTMs) function as
knowledge repositories, containing extensive information from the original
dataset. This naturally raises a question: Can PTMs effectively transfer
knowledge to synthetic datasets, guiding DD accurately? To this end, we conduct
preliminary experiments, confirming the contribution of PTMs to DD. Afterwards,
we systematically study different options in PTMs, including initialization
parameters, model architecture, training epoch and domain knowledge, revealing
that: 1) Increasing model diversity enhances the performance of synthetic
datasets; 2) Sub-optimal models can also assist in DD and outperform
well-trained ones in certain cases; 3) Domain-specific PTMs are not mandatory
for DD, but a reasonable domain match is crucial. Finally, by selecting optimal
options, we significantly improve the cross-architecture generalization over
baseline DD methods. We hope our work will facilitate researchers to develop
better DD techniques. Our code is available at
https://github.com/yaolu-zjut/DDInterpreter.