Rethinking the Starting Point: Enhancing Performance and Fairness of
  Federated Learning via Collaborative Pre-Training
Yun-Wei Chu Dong-Jun Han Seyyedali Hosseinalipour Christopher G. Brinton
Most existing federated learning (FL) methodologies have assumed training
begins from a randomly initialized model. Recently, several studies have
empirically demonstrated that leveraging a pre-trained model can offer
advantageous initializations for FL. In this paper, we propose a collaborative
pre-training approach, CoPreFL, which strategically designs a pre-trained model
to serve as a good initialization for any downstream FL task. The key idea of
our pre-training algorithm is a meta-learning procedure which mimics downstream
distributed scenarios, enabling it to adapt to any unforeseen FL task.
CoPreFL's pre-training optimization procedure also strikes a balance between
average performance and fairness, with the aim of addressing these competing
challenges in downstream FL tasks through intelligent initializations.
Extensive experimental results validate that our pre-training method provides a
robust initialization for any unseen downstream FL task, resulting in enhanced
average performance and more equitable predictions.