Dataset Distillation in Latent Space
Yuxuan Duan Jianfu Zhang Liqing Zhang
Dataset distillation (DD) is a newly emerging research area aiming at
alleviating the heavy computational load in training models on large datasets.
It tries to distill a large dataset into a small and condensed one so that
models trained on the distilled dataset can perform comparably with those
trained on the full dataset when performing downstream tasks. Among the
previous works in this area, there are three key problems that hinder the
performance and availability of the existing DD methods: high time complexity,
high space complexity, and low info-compactness. In this work, we
simultaneously attempt to settle these three problems by moving the DD
processes from conventionally used pixel space to latent space. Encoded by a
pretrained generic autoencoder, latent codes in the latent space are naturally
info-compact representations of the original images in much smaller sizes.
After transferring three mainstream DD algorithms to latent space, we
significantly reduce time and space consumption while achieving similar
performance, allowing us to distill high-resolution datasets or target at
greater data ratio that previous methods have failed. Besides, within the same
storage budget, we can also quantitatively deliver more latent codes than
pixel-level images, which further boosts the performance of our methods.