FedBRB: An Effective Solution to the Small-to-Large Scenario in
  Device-Heterogeneity Federated Learning
Ziyue Xu Mingfeng Xu Tianchi Liao Zibin Zheng Chuan Chen
Recently, the success of large models has demonstrated the importance of
scaling up model size. This has spurred interest in exploring collaborative
training of large-scale models from federated learning perspective. Due to
computational constraints, many institutions struggle to train a large-scale
model locally. Thus, training a larger global model using only smaller local
models has become an important scenario (i.e., the \textbf{small-to-large
scenario}). Although recent device-heterogeneity federated learning approaches
have started to explore this area, they face limitations in fully covering the
parameter space of the global model. In this paper, we propose a method called
\textbf{FedBRB} (\underline{B}lock-wise \underline{R}olling and weighted
\underline{B}roadcast) based on the block concept. FedBRB can uses small local
models to train all blocks of the large global model, and broadcasts the
trained parameters to the entire space for faster information interaction.
Experiments demonstrate FedBRB yields substantial performance gains, achieving
state-of-the-art results in this scenario. Moreover, FedBRB using only minimal
local models can even surpass baselines using larger local models.