Boosting the Cross-Architecture Generalization of Dataset Distillation
  through an Empirical Study
Lirui Zhao Yuxin Zhang Mingbao Lin Fei Chao Rongrong Ji
The poor cross-architecture generalization of dataset distillation greatly
weakens its practical significance. This paper attempts to mitigate this issue
through an empirical study, which suggests that the synthetic datasets undergo
an inductive bias towards the distillation model. Therefore, the evaluation
model is strictly confined to having similar architectures of the distillation
model. We propose a novel method of EvaLuation with distillation Feature (ELF),
which utilizes features from intermediate layers of the distillation model for
the cross-architecture evaluation. In this manner, the evaluation model learns
from bias-free knowledge therefore its architecture becomes unfettered while
retaining performance. By performing extensive experiments, we successfully
prove that ELF can well enhance the cross-architecture generalization of
current DD methods. Code of this project is at
\url{https://github.com/Lirui-Zhao/ELF}.