Fed-CVLC: Compressing Federated Learning Communications with
  Variable-Length Codes
Xiaoxin Su Yipeng Zhou Laizhong Cui John C. S. Lui Jiangchuan Liu
In Federated Learning (FL) paradigm, a parameter server (PS) concurrently
communicates with distributed participating clients for model collection,
update aggregation, and model distribution over multiple rounds, without
touching private data owned by individual clients. FL is appealing in
preserving data privacy; yet the communication between the PS and scattered
clients can be a severe bottleneck. Model compression algorithms, such as
quantization and sparsification, have been suggested but they generally assume
a fixed code length, which does not reflect the heterogeneity and variability
of model updates. In this paper, through both analysis and experiments, we show
strong evidences that variable-length is beneficial for compression in FL. We
accordingly present Fed-CVLC (Federated Learning Compression with
Variable-Length Codes), which fine-tunes the code length in response of the
dynamics of model updates. We develop optimal tuning strategy that minimizes
the loss function (equivalent to maximizing the model utility) subject to the
budget for communication. We further demonstrate that Fed-CVLC is indeed a
general compression design that bridges quantization and sparsification, with
greater flexibility. Extensive experiments have been conducted with public
datasets to demonstrate that Fed-CVLC remarkably outperforms state-of-the-art
baselines, improving model utility by 1.50%-5.44%, or shrinking communication
traffic by 16.67%-41.61%.