Averaging Rate Scheduler for Decentralized Learning on Heterogeneous
  Data
Sai Aparna Aketi Sakshi Choudhary Kaushik Roy
State-of-the-art decentralized learning algorithms typically require the data
distribution to be Independent and Identically Distributed (IID). However, in
practical scenarios, the data distribution across the agents can have
significant heterogeneity. In this work, we propose averaging rate scheduling
as a simple yet effective way to reduce the impact of heterogeneity in
decentralized learning. Our experiments illustrate the superiority of the
proposed method (~3% improvement in test accuracy) compared to the conventional
approach of employing a constant averaging rate.