Communication-Efficient Split Learning via Adaptive Feature-Wise
  Compression
Yongjeong Oh Jaeho Lee Christopher G. Brinton Yo-Seb Jeon
This paper proposes a novel communication-efficient split learning (SL)
framework, named SplitFC, which reduces the communication overhead required for
transmitting intermediate feature and gradient vectors during the SL training
process. The key idea of SplitFC is to leverage different dispersion degrees
exhibited in the columns of the matrices. SplitFC incorporates two compression
strategies: (i) adaptive feature-wise dropout and (ii) adaptive feature-wise
quantization. In the first strategy, the intermediate feature vectors are
dropped with adaptive dropout probabilities determined based on the standard
deviation of these vectors. Then, by the chain rule, the intermediate gradient
vectors associated with the dropped feature vectors are also dropped. In the
second strategy, the non-dropped intermediate feature and gradient vectors are
quantized using adaptive quantization levels determined based on the ranges of
the vectors. To minimize the quantization error, the optimal quantization
levels of this strategy are derived in a closed-form expression. Simulation
results on the MNIST, CIFAR-10, and CelebA datasets demonstrate that SplitFC
provides more than a 5.6% increase in classification accuracy compared to
state-of-the-art SL frameworks, while they require 320 times less communication
overhead compared to the vanilla SL framework without compression.