Vanishing Variance Problem in Fully Decentralized Neural-Network Systems
Yongding Tian Zaid Al-Ars Maksim Kitsak Peter Hofstee
Federated learning and gossip learning are emerging methodologies designed to
mitigate data privacy concerns by retaining training data on client devices and
exclusively sharing locally-trained machine learning (ML) models with others.
The primary distinction between the two lies in their approach to model
aggregation: federated learning employs a centralized parameter server, whereas
gossip learning adopts a fully decentralized mechanism, enabling direct model
exchanges among nodes. This decentralized nature often positions gossip
learning as less efficient compared to federated learning. Both methodologies
involve a critical step: computing a representation of received ML models and
integrating this representation into the existing model. Conventionally, this
representation is derived by averaging the received models, exemplified by the
FedAVG algorithm. Our findings suggest that this averaging approach inherently
introduces a potential delay in model convergence. We identify the underlying
cause and refer to it as the "vanishing variance" problem, where averaging
across uncorrelated ML models undermines the optimal variance established by
the Xavier weight initialization. Unlike federated learning where the central
server ensures model correlation, and unlike traditional gossip learning which
circumvents this problem through model partitioning and sampling, our research
introduces a variance-corrected model averaging algorithm. This novel algorithm
preserves the optimal variance needed during model averaging, irrespective of
network topology or non-IID data distributions. Our extensive simulation
results demonstrate that our approach enables gossip learning to achieve
convergence efficiency comparable to that of federated learning.