A Joint Gradient and Loss Based Clustered Federated Learning Design
Licheng Lin Mingzhe Chen Zhaohui Yang Yusen Wu Yuchen Liu
In this paper, a novel clustered FL framework that enables distributed edge
devices with non-IID data to independently form several clusters in a
distributed manner and implement FL training within each cluster is proposed.
In particular, our designed clustered FL algorithm must overcome two challenges
associated with FL training. First, the server has limited FL training
information (i.e., the parameter server can only obtain the FL model
information of each device) and limited computational power for finding the
differences among a large amount of devices. Second, each device does not have
the data information of other devices for device clustering and can only use
global FL model parameters received from the server and its data information to
determine its cluster identity, which will increase the difficulty of device
clustering. To overcome these two challenges, we propose a joint gradient and
loss based distributed clustering method in which each device determines its
cluster identity considering the gradient similarity and training loss. The
proposed clustering method not only considers how a local FL model of one
device contributes to each cluster but also the direction of gradient descent
thus improving clustering speed. By delegating clustering decisions to edge
devices, each device can fully leverage its private data information to
determine its own cluster identity, thereby reducing clustering overhead and
improving overall clustering performance. Simulation results demonstrate that
our proposed clustered FL algorithm can reduce clustering iterations by up to
99% compared to the existing baseline.