DePRL: Achieving Linear Convergence Speedup in Personalized
  Decentralized Learning with Shared Representations
Guojun Xiong Gang Yan Shiqiang Wang Jian Li
Decentralized learning has emerged as an alternative method to the popular
parameter-server framework which suffers from high communication burden,
single-point failure and scalability issues due to the need of a central
server. However, most existing works focus on a single shared model for all
workers regardless of the data heterogeneity problem, rendering the resulting
model performing poorly on individual workers. In this work, we propose a novel
personalized decentralized learning algorithm named DePRL via shared
representations. Our algorithm relies on ideas from representation learning
theory to learn a low-dimensional global representation collaboratively among
all workers in a fully decentralized manner, and a user-specific
low-dimensional local head leading to a personalized solution for each worker.
We show that DePRL achieves, for the first time, a provable linear speedup for
convergence with general non-linear representations (i.e., the convergence rate
is improved linearly with respect to the number of workers). Experimental
results support our theoretical findings showing the superiority of our method
in data heterogeneous environments.