Data Distillation Can Be Like Vodka: Distilling More Times For Better
  Quality
Xuxi Chen Yu Yang Zhangyang Wang Baharan Mirzasoleiman
Dataset distillation aims to minimize the time and memory needed for training
deep networks on large datasets, by creating a small set of synthetic images
that has a similar generalization performance to that of the full dataset.
However, current dataset distillation techniques fall short, showing a notable
performance gap when compared to training on the original data. In this work,
we are the first to argue that using just one synthetic subset for distillation
will not yield optimal generalization performance. This is because the training
dynamics of deep networks drastically change during the training. Hence,
multiple synthetic subsets are required to capture the training dynamics at
different phases of training. To address this issue, we propose Progressive
Dataset Distillation (PDD). PDD synthesizes multiple small sets of synthetic
images, each conditioned on the previous sets, and trains the model on the
cumulative union of these subsets without requiring additional training time.
Our extensive experiments show that PDD can effectively improve the performance
of existing dataset distillation methods by up to 4.3%. In addition, our method
for the first time enable generating considerably larger synthetic datasets.