RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient
  Compression on Remote Sensing Image Interpretation
Weiying Xie Zixuan Wang Jitao Ma Daixun Li Yunsong Li
Distributed deep learning has recently been attracting more attention in
remote sensing (RS) applications due to the challenges posed by the increased
amount of open data that are produced daily by Earth observation programs.
However, the high communication costs of sending model updates among multiple
nodes are a significant bottleneck for scalable distributed learning. Gradient
sparsification has been validated as an effective gradient compression (GC)
technique for reducing communication costs and thus accelerating the training
speed. Existing state-of-the-art gradient sparsification methods are mostly
based on the "larger-absolute-more-important" criterion, ignoring the
importance of small gradients, which is generally observed to affect the
performance. Inspired by informative representation of manifold structures from
neighborhood information, we propose a simple yet effective dynamic gradient
compression scheme leveraging neighborhood statistics indicator for RS image
interpretation, termed RS-DGC. We first enhance the interdependence between
gradients by introducing the gradient neighborhood to reduce the effect of
random noise. The key component of RS-DGC is a Neighborhood Statistical
Indicator (NSI), which can quantify the importance of gradients within a
specified neighborhood on each node to sparsify the local gradients before
gradient transmission in each iteration. Further, a layer-wise dynamic
compression scheme is proposed to track the importance changes of each layer in
real time. Extensive downstream tasks validate the superiority of our method in
terms of intelligent interpretation of RS images. For example, we achieve an
accuracy improvement of 0.51% with more than 50 times communication compression
on the NWPU-RESISC45 dataset using VGG-19 network.