ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm
  with Adaptive Batch Size for Heterogeneous GPU Clusters
Xin Zhou Ling Chen Houming Wu
As the size of models and datasets grows, it has become increasingly common
to train models in parallel. However, existing distributed stochastic gradient
descent (SGD) algorithms suffer from insufficient utilization of computational
resources and poor convergence in heterogeneous clusters. In this paper, we
propose a delayed synchronous SGD algorithm with adaptive batch size (ABS-SGD)
for heterogeneous GPU clusters. In ABS-SGD, workers perform global
synchronization to accumulate delayed gradients and use the accumulated delayed
gradients to update parameters. While workers are performing global
synchronization for delayed gradients, they perform the computation of the next
batch without specifying batch size in advance, which lasts until the next
global synchronization starts, realizing the full utilization of computational
resources. Since the gradient delay is only one iteration, the stale gradient
problem can be alleviated. We theoretically prove the convergence of ABS-SGD in
heterogeneous clusters. Extensive experiments in three types of heterogeneous
clusters demonstrate that ABS-SGD can make full use of computational resources
and accelerate model convergence: When training ResNet18 network with 4
workers, ABS-SGD increases the convergence speed by 1.30x on average compared
with the best baseline algorithm.