Exploiting Inter-sample and Inter-feature Relations in Dataset
  Distillation
Wenxiao Deng Wenbin Li Tianyu Ding Lei Wang Hongguang Zhang Kuihua Huang Jing Huo Yang Gao
Dataset distillation has emerged as a promising approach in deep learning,
enabling efficient training with small synthetic datasets derived from larger
real ones. Particularly, distribution matching-based distillation methods
attract attention thanks to its effectiveness and low computational cost.
However, these methods face two primary limitations: the dispersed feature
distribution within the same class in synthetic datasets, reducing class
discrimination, and an exclusive focus on mean feature consistency, lacking
precision and comprehensiveness. To address these challenges, we introduce two
novel constraints: a class centralization constraint and a covariance matching
constraint. The class centralization constraint aims to enhance class
discrimination by more closely clustering samples within classes. The
covariance matching constraint seeks to achieve more accurate feature
distribution matching between real and synthetic datasets through local feature
covariance matrices, particularly beneficial when sample sizes are much smaller
than the number of features. Experiments demonstrate notable improvements with
these constraints, yielding performance boosts of up to 6.6% on CIFAR10, 2.9%
on SVHN, 2.5% on CIFAR100, and 2.5% on TinyImageNet, compared to the
state-of-the-art relevant methods. In addition, our method maintains robust
performance in cross-architecture settings, with a maximum performance drop of
1.7% on four architectures. Code is available at
https://github.com/VincenDen/IID.