Partially Personalized Federated Learning: Breaking the Curse of Data
  Heterogeneity
Konstantin Mishchenko Rustem Islamov Eduard Gorbunov Samuel Horváth
We present a partially personalized formulation of Federated Learning (FL)
that strikes a balance between the flexibility of personalization and
cooperativeness of global training. In our framework, we split the variables
into global parameters, which are shared across all clients, and individual
local parameters, which are kept private. We prove that under the right split
of parameters, it is possible to find global parameters that allow each client
to fit their data perfectly, and refer to the obtained problem as
overpersonalized. For instance, the shared global parameters can be used to
learn good data representations, whereas the personalized layers are fine-tuned
for a specific client. Moreover, we present a simple algorithm for the
partially personalized formulation that offers significant benefits to all
clients. In particular, it breaks the curse of data heterogeneity in several
settings, such as training with local steps, asynchronous training, and
Byzantine-robust training.