Importance-Aware Adaptive Dataset Distillation
Guang Li Ren Togo Takahiro Ogawa Miki Haseyama
Herein, we propose a novel dataset distillation method for constructing small
informative datasets that preserve the information of the large original
datasets. The development of deep learning models is enabled by the
availability of large-scale datasets. Despite unprecedented success,
large-scale datasets considerably increase the storage and transmission costs,
resulting in a cumbersome model training process. Moreover, using raw data for
training raises privacy and copyright concerns. To address these issues, a new
task named dataset distillation has been introduced, aiming to synthesize a
compact dataset that retains the essential information from the large original
dataset. State-of-the-art (SOTA) dataset distillation methods have been
proposed by matching gradients or network parameters obtained during training
on real and synthetic datasets. The contribution of different network
parameters to the distillation process varies, and uniformly treating them
leads to degraded distillation performance. Based on this observation, we
propose an importance-aware adaptive dataset distillation (IADD) method that
can improve distillation performance by automatically assigning importance
weights to different network parameters during distillation, thereby
synthesizing more robust distilled datasets. IADD demonstrates superior
performance over other SOTA dataset distillation methods based on parameter
matching on multiple benchmark datasets and outperforms them in terms of
cross-architecture generalization. In addition, the analysis of self-adaptive
weights demonstrates the effectiveness of IADD. Furthermore, the effectiveness
of IADD is validated in a real-world medical application such as COVID-19
detection.