DPPA: Pruning Method for Large Language Model to Model Merging
Yaochen Zhu Rui Xia Jiajun Zhang
Model merging is to combine fine-tuned models derived from multiple domains,
with the intent of enhancing the model's proficiency across various domains.
The principal concern is the resolution of parameter conflicts. A substantial
amount of existing research remedy this issue during the merging stage, with
the latest study focusing on resolving this issue throughout the pruning stage.
The DARE approach has exhibited promising outcomes when applied to a simplistic
fine-tuned model. However, the efficacy of this method tends to wane when
employed on complex fine-tuned models that show a significant parameter bias
relative to the baseline model. In this paper, we introduce a dual-stage method
termed Dynamic Pruning Partition Amplification (DPPA), devised to tackle the
challenge of merging complex fine-tuned models. Initially, we introduce
Dynamically Pruning (DP), an improved approach based on magnitude pruning,
which aim is to enhance performance at higher pruning rates. Subsequently, we
propose Dynamically Partition Amplification (DPA), a rescaling strategy, is
designed to dynamically amplify parameter partitions in relation to their
significance levels. The experimental results show that our method maintains a
mere 20% of domain-specific parameters and yet delivers a performance
comparable to other methodologies that preserve up to 90% of parameters.
Furthermore, our method displays outstanding performance post-pruning, leading
to a significant improvement of nearly 20% performance in model merging. We
make our code on Github.