Improve Cross-Architecture Generalization on Dataset Distillation
Binglin Zhou Linhao Zhong Wentao Chen
Dataset distillation, a pragmatic approach in machine learning, aims to
create a smaller synthetic dataset from a larger existing dataset. However,
existing distillation methods primarily adopt a model-based paradigm, where the
synthetic dataset inherits model-specific biases, limiting its generalizability
to alternative models. In response to this constraint, we propose a novel
methodology termed "model pool". This approach involves selecting models from a
diverse model pool based on a specific probability distribution during the data
distillation process. Additionally, we integrate our model pool with the
established knowledge distillation approach and apply knowledge distillation to
the test process of the distilled dataset. Our experimental results validate
the effectiveness of the model pool approach across a range of existing models
while testing, demonstrating superior performance compared to existing
methodologies.