Provable Policy Gradient Methods for Average-Reward Markov Potential
  Games
Min Cheng Ruida Zhou P. R. Kumar Chao Tian
We study Markov potential games under the infinite horizon average reward
criterion. Most previous studies have been for discounted rewards. We prove
that both algorithms based on independent policy gradient and independent
natural policy gradient converge globally to a Nash equilibrium for the average
reward criterion. To set the stage for gradient-based methods, we first
establish that the average reward is a smooth function of policies and provide
sensitivity bounds for the differential value functions, under certain
conditions on ergodicity and the second largest eigenvalue of the underlying
Markov decision process (MDP). We prove that three algorithms, policy gradient,
proximal-Q, and natural policy gradient (NPG), converge to an $\epsilon$-Nash
equilibrium with time complexity $O(\frac{1}{\epsilon^2})$, given a
gradient/differential Q function oracle. When policy gradients have to be
estimated, we propose an algorithm with
$\tilde{O}(\frac{1}{\min_{s,a}\pi(a|s)\delta})$ sample complexity to achieve
$\delta$ approximation error w.r.t~the $\ell_2$ norm. Equipped with the
estimator, we derive the first sample complexity analysis for a policy gradient
ascent algorithm, featuring a sample complexity of $\tilde{O}(1/\epsilon^5)$.
Simulation studies are presented.