A General and Efficient Federated Split Learning with Pre-trained Image
  Transformers for Heterogeneous Data
Yifan Shi Yuhui Zhang Ziyue Huang Xiaofeng Yang Li Shen Wei Chen Xueqian Wang
Federated Split Learning (FSL) is a promising distributed learning paradigm
in practice, which gathers the strengths of both Federated Learning (FL) and
Split Learning (SL) paradigms, to ensure model privacy while diminishing the
resource overhead of each client, especially on large transformer models in a
resource-constrained environment, e.g., Internet of Things (IoT). However,
almost all works merely investigate the performance with simple neural network
models in FSL. Despite the minor efforts focusing on incorporating Vision
Transformers (ViT) as model architectures, they train ViT from scratch, thereby
leading to enormous training overhead in each device with limited resources.
Therefore, in this paper, we harness Pre-trained Image Transformers (PITs) as
the initial model, coined FES-PIT, to accelerate the training process and
improve model robustness. Furthermore, we propose FES-PTZO to hinder the
gradient inversion attack, especially having the capability compatible with
black-box scenarios, where the gradient information is unavailable. Concretely,
FES-PTZO approximates the server gradient by utilizing a zeroth-order (ZO)
optimization, which replaces the backward propagation with just one forward
process. Empirically, we are the first to provide a systematic evaluation of
FSL methods with PITs in real-world datasets, different partial device
participations, and heterogeneous data splits. Our experiments verify the
effectiveness of our algorithms.