Global Update Tracking: A Decentralized Learning Algorithm for
  Heterogeneous Data
Sai Aparna Aketi Abolfazl Hashemi Kaushik Roy
Decentralized learning enables the training of deep learning models over
large distributed datasets generated at different locations, without the need
for a central server. However, in practical scenarios, the data distribution
across these devices can be significantly different, leading to a degradation
in model performance. In this paper, we focus on designing a decentralized
learning algorithm that is less susceptible to variations in data distribution
across devices. We propose Global Update Tracking (GUT), a novel tracking-based
method that aims to mitigate the impact of heterogeneous data in decentralized
learning without introducing any communication overhead. We demonstrate the
effectiveness of the proposed technique through an exhaustive set of
experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion
MNIST, and ImageNette), model architectures, and network topologies. Our
experiments show that the proposed method achieves state-of-the-art performance
for decentralized learning on heterogeneous data via a $1-6\%$ improvement in
test accuracy compared to other existing techniques.