Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex
  Optimization
Siyuan Zhang Nachuan Xiao Xin Liu
In this paper, we concentrate on decentralized optimization problems with
nonconvex and nonsmooth objective functions, especially on the decentralized
training of nonsmooth neural networks. We introduce a unified framework, named
DSM, to analyze the global convergence of decentralized stochastic subgradient
methods. We prove the global convergence of our proposed framework under mild
conditions, by establishing that the generated sequence asymptotically
approximates the trajectories of its associated differential inclusion.
Furthermore, we establish that our proposed framework encompasses a wide range
of existing efficient decentralized subgradient methods, including
decentralized stochastic subgradient descent (DSGD), DSGD with
gradient-tracking technique (DSGD-T), and DSGD with momentum (DSGDm). In
addition, we introduce SignSGD employing the sign map to regularize the update
directions in DSGDm, and show it is enclosed in our proposed framework.
Consequently, our convergence results establish, for the first time, global
convergence of these methods when applied to nonsmooth nonconvex objectives.
Preliminary numerical experiments demonstrate that our proposed framework
yields highly efficient decentralized subgradient methods with convergence
guarantees in the training of nonsmooth neural networks.