Decentralized Learning over Wireless Networks with Broadcast-Based
  Subgraph Sampling
Daniel Pérez Herrera Zheng Chen Erik G. Larsson
This work centers on the communication aspects of decentralized learning over
wireless networks, using consensus-based decentralized stochastic gradient
descent (D-SGD). Considering the actual communication cost or delay caused by
in-network information exchange in an iterative process, our goal is to achieve
fast convergence of the algorithm measured by improvement per transmission
slot. We propose BASS, an efficient communication framework for D-SGD over
wireless networks with broadcast transmission and probabilistic subgraph
sampling. In each iteration, we activate multiple subsets of non-interfering
nodes to broadcast model updates to their neighbors. These subsets are randomly
activated over time, with probabilities reflecting their importance in network
connectivity and subject to a communication cost constraint (e.g., the average
number of transmission slots per iteration). During the consensus update step,
only bi-directional links are effectively preserved to maintain communication
symmetry. In comparison to existing link-based scheduling methods, the inherent
broadcasting nature of wireless channels offers intrinsic advantages in
speeding up convergence of decentralized learning by creating more communicated
links with the same number of transmission slots.