Kimad: Adaptive Gradient Compression with Bandwidth Awareness
Jihao Xin Ivan Ilin Shunkang Zhang Marco Canini Peter Richtárik
In distributed training, communication often emerges as a bottleneck. In
response, we introduce Kimad, a solution that offers adaptive gradient
compression. By consistently monitoring bandwidth, Kimad refines compression
ratios to match specific neural network layer requirements. Our exhaustive
tests and proofs confirm Kimad's outstanding performance, establishing it as a
benchmark in adaptive compression for distributed deep learning.