FLASH: Federated Learning Across Simultaneous Heterogeneities
Xiangyu Chang Sk Miraj Ahmed Srikanth V. Krishnamurthy Basak Guler Ananthram Swami Samet Oymak Amit K. Roy-Chowdhury
The key premise of federated learning (FL) is to train ML models across a
diverse set of data-owners (clients), without exchanging local data. An
overarching challenge to this date is client heterogeneity, which may arise not
only from variations in data distribution, but also in data quality, as well as
compute/communication latency. An integrated view of these diverse and
concurrent sources of heterogeneity is critical; for instance, low-latency
clients may have poor data quality, and vice versa. In this work, we propose
FLASH(Federated Learning Across Simultaneous Heterogeneities), a lightweight
and flexible client selection algorithm that outperforms state-of-the-art FL
frameworks under extensive sources of heterogeneity, by trading-off the
statistical information associated with the client's data quality, data
distribution, and latency. FLASH is the first method, to our knowledge, for
handling all these heterogeneities in a unified manner. To do so, FLASH models
the learning dynamics through contextual multi-armed bandits (CMAB) and
dynamically selects the most promising clients. Through extensive experiments,
we demonstrate that FLASH achieves substantial and consistent improvements over
state-of-the-art baselines -- as much as 10% in absolute accuracy -- thanks to
its unified approach. Importantly, FLASH also outperforms federated aggregation
methods that are designed to handle highly heterogeneous settings and even
enjoys a performance boost when integrated with them.