Compressed Federated Reinforcement Learning with a Generative Model
Ali Beikmohammadi Sarit Khirirat Sindri Magnússon
Reinforcement learning has recently gained unprecedented popularity, yet it
still grapples with sample inefficiency. Addressing this challenge, federated
reinforcement learning (FedRL) has emerged, wherein agents collaboratively
learn a single policy by aggregating local estimations. However, this
aggregation step incurs significant communication costs. In this paper, we
propose CompFedRL, a communication-efficient FedRL approach incorporating both
\textit{periodic aggregation} and (direct/error-feedback) compression
mechanisms. Specifically, we consider compressed federated $Q$-learning with a
generative model setup, where a central server learns an optimal $Q$-function
by periodically aggregating compressed $Q$-estimates from local agents. For the
first time, we characterize the impact of these two mechanisms (which have
remained elusive) by providing a finite-time analysis of our algorithm,
demonstrating strong convergence behaviors when utilizing either direct or
error-feedback compression. Our bounds indicate improved solution accuracy
concerning the number of agents and other federated hyperparameters while
simultaneously reducing communication costs. To corroborate our theory, we also
conduct in-depth numerical experiments to verify our findings, considering
Top-$K$ and Sparsified-$K$ sparsification operators.