Revisiting Permutation Symmetry for Merging Models between Different
  Datasets
Masanori Yamada Tomoya Yamashita Shin'ya Yamaguchi Daiki Chijiwa
Model merging is a new approach to creating a new model by combining the
weights of different trained models. Previous studies report that model merging
works well for models trained on a single dataset with different random seeds,
while model merging between different datasets is difficult. Merging knowledge
from different datasets has practical significance, but it has not been well
investigated. In this paper, we investigate the properties of merging models
between different datasets. Through theoretical and empirical analyses, we find
that the accuracy of the merged model decreases more significantly as the
datasets diverge more and that the different loss landscapes for each dataset
make model merging between different datasets difficult. We also show that
merged models require datasets for merging in order to achieve a high accuracy.
Furthermore, we show that condensed datasets created by dataset condensation
can be used as substitutes for the original datasets when merging models. We
conduct experiments for model merging between different datasets. When merging
between MNIST and Fashion- MNIST models, the accuracy significantly improves by
28% using the dataset and 25% using the condensed dataset compared with not
using the dataset.