Accelerating Distributed Deep Learning using Lossless Homomorphic
  Compression
Haoyu Li Yuchen Xu Jiayi Chen Rohit Dwivedula Wenfei Wu Keqiang He Aditya Akella Daehyeok Kim
As deep neural networks (DNNs) grow in complexity and size, the resultant
increase in communication overhead during distributed training has become a
significant bottleneck, challenging the scalability of distributed training
systems. Existing solutions, while aiming to mitigate this bottleneck through
worker-level compression and in-network aggregation, fall short due to their
inability to efficiently reconcile the trade-offs between compression
effectiveness and computational overhead, hindering overall performance and
scalability. In this paper, we introduce a novel compression algorithm that
effectively merges worker-level compression with in-network aggregation. Our
solution is both homomorphic, allowing for efficient in-network aggregation
without CPU/GPU processing, and lossless, ensuring no compromise on training
accuracy. Theoretically optimal in compression and computational efficiency,
our approach is empirically validated across diverse DNN models such as NCF,
LSTM, VGG19, and BERT-base, showing up to a 6.33$\times$ improvement in
aggregation throughput and a 3.74$\times$ increase in per-iteration training
speed.