LoCoDL: Communication-Efficient Distributed Learning with Local Training
  and Compression
Laurent Condat Artavazd Maranjyan Peter Richtárik
In Distributed optimization and Learning, and even more in the modern
framework of federated learning, communication, which is slow and costly, is
critical. We introduce LoCoDL, a communication-efficient algorithm that
leverages the two popular and effective techniques of Local training, which
reduces the communication frequency, and Compression, in which short bitstreams
are sent instead of full-dimensional vectors of floats. LoCoDL works with a
large class of unbiased compressors that includes widely-used sparsification
and quantization methods. LoCoDL provably benefits from local training and
compression and enjoys a doubly-accelerated communication complexity, with
respect to the condition number of the functions and the model dimension, in
the general heterogenous regime with strongly convex functions. This is
confirmed in practice, with LoCoDL outperforming existing algorithms.