EControl: Fast Distributed Optimization with Compression and Error
  Control
Yuan Gao Rustem Islamov Sebastian Stich
Modern distributed training relies heavily on communication compression to
reduce the communication overhead. In this work, we study algorithms employing
a popular class of contractive compressors in order to reduce communication
overhead. However, the naive implementation often leads to unstable convergence
or even exponential divergence due to the compression bias. Error Compensation
(EC) is an extremely popular mechanism to mitigate the aforementioned issues
during the training of models enhanced by contractive compression operators.
Compared to the effectiveness of EC in the data homogeneous regime, the
understanding of the practicality and theoretical foundations of EC in the data
heterogeneous regime is limited. Existing convergence analyses typically rely
on strong assumptions such as bounded gradients, bounded data heterogeneity, or
large batch accesses, which are often infeasible in modern machine learning
applications. We resolve the majority of current issues by proposing EControl,
a novel mechanism that can regulate error compensation by controlling the
strength of the feedback signal. We prove fast convergence for EControl in
standard strongly convex, general convex, and nonconvex settings without any
additional assumptions on the problem or data heterogeneity. We conduct
extensive numerical evaluations to illustrate the efficacy of our method and
support our theoretical findings.