DYNAMITE: Dynamic Interplay of Mini-Batch Size and Aggregation Frequency
  for Federated Learning with Static and Streaming Dataset
Weijie Liu Xiaoxi Zhang Jingpu Duan Carlee Joe-Wong Zhi Zhou Xu Chen
Federated Learning (FL) is a distributed learning paradigm that can
coordinate heterogeneous edge devices to perform model training without sharing
private data. While prior works have focused on analyzing FL convergence with
respect to hyperparameters like batch size and aggregation frequency, the joint
effects of adjusting these parameters on model performance, training time, and
resource consumption have been overlooked, especially when facing dynamic data
streams and network characteristics. This paper introduces novel analytical
models and optimization algorithms that leverage the interplay between batch
size and aggregation frequency to navigate the trade-offs among convergence,
cost, and completion time for dynamic FL training. We establish a new
convergence bound for training error considering heterogeneous datasets across
devices and derive closed-form solutions for co-optimized batch size and
aggregation frequency that are consistent across all devices. Additionally, we
design an efficient algorithm for assigning different batch configurations
across devices, improving model accuracy and addressing the heterogeneity of
both data and system characteristics. Further, we propose an adaptive control
algorithm that dynamically estimates network states, efficiently samples
appropriate data batches, and effectively adjusts batch sizes and aggregation
frequency on the fly. Extensive experiments demonstrate the superiority of our
offline optimal solutions and online adaptive algorithm.