Deep Model Fusion: A Survey
Weishi Li Yong Peng Miao Zhang Liang Ding Han Hu Li Shen
Deep model fusion/merging is an emerging technique that merges the parameters
or predictions of multiple deep learning models into a single one. It combines
the abilities of different models to make up for the biases and errors of a
single model to achieve better performance. However, deep model fusion on
large-scale deep learning models (e.g., LLMs and foundation models) faces
several challenges, including high computational cost, high-dimensional
parameter space, interference between different heterogeneous models, etc.
Although model fusion has attracted widespread attention due to its potential
to solve complex real-world tasks, there is still a lack of complete and
detailed survey research on this technique. Accordingly, in order to understand
the model fusion method better and promote its development, we present a
comprehensive survey to summarize the recent progress. Specifically, we
categorize existing deep model fusion methods as four-fold: (1) "Mode
connectivity", which connects the solutions in weight space via a path of
non-increasing loss, in order to obtain better initialization for model fusion;
(2) "Alignment" matches units between neural networks to create better
conditions for fusion; (3) "Weight average", a classical model fusion method,
averages the weights of multiple models to obtain more accurate results closer
to the optimal solution; (4) "Ensemble learning" combines the outputs of
diverse models, which is a foundational technique for improving the accuracy
and robustness of the final model. In addition, we analyze the challenges faced
by deep model fusion and propose possible research directions for model fusion
in the future. Our review is helpful in deeply understanding the correlation
between different model fusion methods and practical application methods, which
can enlighten the research in the field of deep model fusion.