MedMerge: Merging Models for Effective Transfer Learning to Medical
  Imaging Tasks
Ibrahim Almakky Santosh Sanjeev Anees Ur Rehman Hashmi Mohammad Areeb Qazi Mohammad Yaqub
Transfer learning has become a powerful tool to initialize deep learning
models to achieve faster convergence and higher performance. This is especially
useful in the medical imaging analysis domain, where data scarcity limits
possible performance gains for deep learning models. Some advancements have
been made in boosting the transfer learning performance gain by merging models
starting from the same initialization. However, in the medical imaging analysis
domain, there is an opportunity in merging models starting from different
initialisations, thus combining the features learnt from different tasks. In
this work, we propose MedMerge, a method whereby the weights of different
models can be merged, and their features can be effectively utilized to boost
performance on a new task. With MedMerge, we learn kernel-level weights that
can later be used to merge the models into a single model, even when starting
from different initializations. Testing on various medical imaging analysis
tasks, we show that our merged model can achieve significant performance gains,
with up to 3% improvement on the F1 score. The code implementation of this work
will be available at www.github.com/BioMedIA-MBZUAI/MedMerge.