Improved Distribution Matching for Dataset Condensation
Ganlong Zhao Guanbin Li Yipeng Qin Yizhou Yu
Dataset Condensation aims to condense a large dataset into a smaller one
while maintaining its ability to train a well-performing model, thus reducing
the storage cost and training effort in deep learning applications. However,
conventional dataset condensation methods are optimization-oriented and
condense the dataset by performing gradient or parameter matching during model
optimization, which is computationally intensive even on small datasets and
models. In this paper, we propose a novel dataset condensation method based on
distribution matching, which is more efficient and promising. Specifically, we
identify two important shortcomings of naive distribution matching (i.e.,
imbalanced feature numbers and unvalidated embeddings for distance computation)
and address them with three novel techniques (i.e., partitioning and expansion
augmentation, efficient and enriched model sampling, and class-aware
distribution regularization). Our simple yet effective method outperforms most
previous optimization-oriented methods with much fewer computational resources,
thereby scaling data condensation to larger datasets and models. Extensive
experiments demonstrate the effectiveness of our method. Codes are available at
https://github.com/uitrbn/IDM