Federated Natural Policy Gradient Methods for Multi-task Reinforcement
  Learning
Tong Yang Shicong Cen Yuting Wei Yuxin Chen Yuejie Chi
Federated reinforcement learning (RL) enables collaborative decision making
of multiple distributed agents without sharing local data trajectories. In this
work, we consider a multi-task setting, in which each agent has its own private
reward function corresponding to different tasks, while sharing the same
transition kernel of the environment. Focusing on infinite-horizon tabular
Markov decision processes, the goal is to learn a globally optimal policy that
maximizes the sum of the discounted total rewards of all the agents in a
decentralized manner, where each agent only communicates with its neighbors
over some prescribed graph topology. We develop federated vanilla and
entropy-regularized natural policy gradient (NPG) methods under softmax
parameterization, where gradient tracking is applied to the global Q-function
to mitigate the impact of imperfect information sharing. We establish
non-asymptotic global convergence guarantees under exact policy evaluation,
which are nearly independent of the size of the state-action space and
illuminate the impacts of network size and connectivity. To the best of our
knowledge, this is the first time that global convergence is established for
federated multi-task RL using policy optimization. Moreover, the convergence
behavior of the proposed algorithms is robust against inexactness of policy
evaluation.