Preserving Near-Optimal Gradient Sparsification Cost for Scalable
  Distributed Deep Learning
Daegun Yoon Sangyoon Oh
Communication overhead is a major obstacle to scaling distributed training
systems. Gradient sparsification is a potential optimization approach to reduce
the communication volume without significant loss of model fidelity. However,
existing gradient sparsification methods have low scalability owing to
inefficient design of their algorithms, which raises the communication overhead
significantly. In particular, gradient build-up and inadequate sparsity control
methods degrade the sparsification performance considerably. Moreover,
communication traffic increases drastically owing to workload imbalance of
gradient selection between workers.
  To address these challenges, we propose a novel gradient sparsification
scheme called ExDyna. In ExDyna, the gradient tensor of the model comprises
fined-grained blocks, and contiguous blocks are grouped into non-overlapping
partitions. Each worker selects gradients in its exclusively allocated
partition so that gradient build-up never occurs. To balance the workload of
gradient selection between workers, ExDyna adjusts the topology of partitions
by comparing the workloads of adjacent partitions. In addition, ExDyna supports
online threshold scaling, which estimates the accurate threshold of gradient
selection on-the-fly. Accordingly, ExDyna can satisfy the user-required
sparsity level during a training period regardless of models and datasets.
Therefore, ExDyna can enhance the scalability of distributed training systems
by preserving near-optimal gradient sparsification cost. In experiments, ExDyna
outperformed state-of-the-art sparsifiers in terms of training speed and
sparsification performance while achieving high accuracy.