Federated Learning for distribution skewed data using sample weights
Hung Nguyen Peiyuan Wu Morris Chang
One of the most challenging issues in federated learning is that the data is
often not independent and identically distributed (nonIID). Clients are
expected to contribute the same type of data and drawn from one global
distribution. However, data are often collected in different ways from
different resources. Thus, the data distributions among clients might be
different from the underlying global distribution. This creates a weight
divergence issue and reduces federated learning performance. This work focuses
on improving federated learning performance for skewed data distribution across
clients. The main idea is to adjust the client distribution closer to the
global distribution using sample weights. Thus, the machine learning model
converges faster with higher accuracy. We start from the fundamental concept of
empirical risk minimization and theoretically derive a solution for adjusting
the distribution skewness using sample weights. To determine sample weights, we
implicitly exchange density information by leveraging a neural network-based
density estimation model, MADE. The clients data distribution can then be
adjusted without exposing their raw data. Our experiment results on three
real-world datasets show that the proposed method not only improves federated
learning accuracy but also significantly reduces communication costs compared
to the other experimental methods.