Peer-to-Peer Learning + Consensus with Non-IID Data
Srinivasa Pranav José M. F. Moura
Peer-to-peer deep learning algorithms are enabling distributed edge devices
to collaboratively train deep neural networks without exchanging raw training
data or relying on a central server. Peer-to-Peer Learning (P2PL) and other
algorithms based on Distributed Local-Update Stochastic/mini-batch Gradient
Descent (local DSGD) rely on interleaving epochs of training with distributed
consensus steps. This process leads to model parameter drift/divergence amongst
participating devices in both IID and non-IID settings. We observe that model
drift results in significant oscillations in test performance evaluated after
local training and consensus phases. We then identify factors that amplify
performance oscillations and demonstrate that our novel approach, P2PL with
Affinity, dampens test performance oscillations in non-IID settings without
incurring any additional communication cost.