Quantization Avoids Saddle Points in Distributed Optimization
Yanan Bo Yongqiang Wang
Distributed nonconvex optimization underpins key functionalities of numerous
distributed systems, ranging from power systems, smart buildings, cooperative
robots, vehicle networks to sensor networks. Recently, it has also merged as a
promising solution to handle the enormous growth in data and model sizes in
deep learning. A fundamental problem in distributed nonconvex optimization is
avoiding convergence to saddle points, which significantly degrade optimization
accuracy. We discover that the process of quantization, which is necessary for
all digital communications, can be exploited to enable saddle-point avoidance.
More specifically, we propose a stochastic quantization scheme and prove that
it can effectively escape saddle points and ensure convergence to a
second-order stationary point in distributed nonconvex optimization. With an
easily adjustable quantization granularity, the approach allows a user to
control the number of bits sent per iteration and, hence, to aggressively
reduce the communication overhead. Numerical experimental results using
distributed optimization and learning problems on benchmark datasets confirm
the effectiveness of the approach.