Convergence Visualizer of Decentralized Federated Distillation with
  Reduced Communication Costs
Akihito Taya Yuuki Nishiyama Kaoru Sezaki
Federated learning (FL) achieves collaborative learning without the need for
data sharing, thus preventing privacy leakage. To extend FL into a fully
decentralized algorithm, researchers have applied distributed optimization
algorithms to FL by considering machine learning (ML) tasks as parameter
optimization problems. Conversely, the consensus-based multi-hop federated
distillation (CMFD) proposed in the authors' previous work makes neural network
(NN) models get close with others in a function space rather than in a
parameter space. Hence, this study solves two unresolved challenges of CMFD:
(1) communication cost reduction and (2) visualization of model convergence.
Based on a proposed dynamic communication cost reduction method (DCCR), the
amount of data transferred in a network is reduced; however, with a slight
degradation in the prediction accuracy. In addition, a technique for
visualizing the distance between the NN models in a function space is also
proposed. The technique applies a dimensionality reduction technique by
approximating infinite-dimensional functions as numerical vectors to visualize
the trajectory of how the models change by the distributed learning algorithm.