Federated Learning via Input-Output Collaborative Distillation
Xuan Gong Shanglin Li Yuxiang Bao Barry Yao Yawen Huang Ziyan Wu Baochang Zhang Yefeng Zheng David Doermann
Federated learning (FL) is a machine learning paradigm in which distributed
local nodes collaboratively train a central model without sharing individually
held private data. Existing FL methods either iteratively share local model
parameters or deploy co-distillation. However, the former is highly susceptible
to private data leakage, and the latter design relies on the prerequisites of
task-relevant real data. Instead, we propose a data-free FL framework based on
local-to-central collaborative distillation with direct input and output space
exploitation. Our design eliminates any requirement of recursive local
parameter exchange or auxiliary task-relevant data to transfer knowledge,
thereby giving direct privacy control to local users. In particular, to cope
with the inherent data heterogeneity across locals, our technique learns to
distill input on which each local model produces consensual yet unique results
to represent each expertise. Our proposed FL framework achieves notable
privacy-utility trade-offs with extensive experiments on image classification
and segmentation tasks under various real-world heterogeneous federated
learning settings on both natural and medical images.