DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning
  Models
Nastaran Saadati Minh Pham Nasla Saleem Joshua R. Waite Aditya Balu Zhanhong Jiang Chinmay Hegde Soumik Sarkar
Recent advances in decentralized deep learning algorithms have demonstrated
cutting-edge performance on various tasks with large pre-trained models.
However, a pivotal prerequisite for achieving this level of competitiveness is
the significant communication and computation overheads when updating these
models, which prohibits the applications of them to real-world scenarios. To
address this issue, drawing inspiration from advanced model merging techniques
without requiring additional training, we introduce the Decentralized Iterative
Merging-And-Training (DIMAT) paradigm--a novel decentralized deep learning
framework. Within DIMAT, each agent is trained on their local data and
periodically merged with their neighboring agents using advanced model merging
techniques like activation matching until convergence is achieved. DIMAT
provably converges with the best available rate for nonconvex functions with
various first-order methods, while yielding tighter error bounds compared to
the popular existing approaches. We conduct a comprehensive empirical analysis
to validate DIMAT's superiority over baselines across diverse computer vision
tasks sourced from multiple datasets. Empirical results validate our
theoretical claims by showing that DIMAT attains faster and higher initial gain
in accuracy with independent and identically distributed (IID) and non-IID
data, incurring lower communication overhead. This DIMAT paradigm presents a
new opportunity for the future decentralized learning, enhancing its
adaptability to real-world with sparse and light-weight communication and
computation.