SignSGD with Federated Voting
Chanho Park H. Vincent Poor Namyoon Lee
Distributed learning is commonly used for accelerating model training by
harnessing the computational capabilities of multiple-edge devices. However, in
practical applications, the communication delay emerges as a bottleneck due to
the substantial information exchange required between workers and a central
parameter server. SignSGD with majority voting (signSGD-MV) is an effective
distributed learning algorithm that can significantly reduce communication
costs by one-bit quantization. However, due to heterogeneous computational
capabilities, it fails to converge when the mini-batch sizes differ among
workers. To overcome this, we propose a novel signSGD optimizer with
\textit{federated voting} (signSGD-FV). The idea of federated voting is to
exploit learnable weights to perform weighted majority voting. The server
learns the weights assigned to the edge devices in an online fashion based on
their computational capabilities. Subsequently, these weights are employed to
decode the signs of the aggregated local gradients in such a way to minimize
the sign decoding error probability. We provide a unified convergence rate
analysis framework applicable to scenarios where the estimated weights are
known to the parameter server either perfectly or imperfectly. We demonstrate
that the proposed signSGD-FV algorithm has a theoretical convergence guarantee
even when edge devices use heterogeneous mini-batch sizes. Experimental results
show that signSGD-FV outperforms signSGD-MV, exhibiting a faster convergence
rate, especially in heterogeneous mini-batch sizes.