Importance of Smoothness Induced by Optimizers in FL4ASR: Towards
  Understanding Federated Learning for End-to-End ASR
Sheikh Shams Azam Tatiana Likhomanenko Martin Pelikan Jan "Honza" Silovsky
In this paper, we start by training End-to-End Automatic Speech Recognition
(ASR) models using Federated Learning (FL) and examining the fundamental
considerations that can be pivotal in minimizing the performance gap in terms
of word error rate between models trained using FL versus their centralized
counterpart. Specifically, we study the effect of (i) adaptive optimizers, (ii)
loss characteristics via altering Connectionist Temporal Classification (CTC)
weight, (iii) model initialization through seed start, (iv) carrying over
modeling setup from experiences in centralized training to FL, e.g., pre-layer
or post-layer normalization, and (v) FL-specific hyperparameters, such as
number of local epochs, client sampling size, and learning rate scheduler,
specifically for ASR under heterogeneous data distribution. We shed light on
how some optimizers work better than others via inducing smoothness. We also
summarize the applicability of algorithms, trends, and propose best practices
from prior works in FL (in general) toward End-to-End ASR models.