Asynchronous Decentralized Q-Learning: Two Timescale Analysis By
  Persistence
Bora Yongacoglu Gürdal Arslan Serdar Yüksel
Non-stationarity is a fundamental challenge in multi-agent reinforcement
learning (MARL), where agents update their behaviour as they learn. Many
theoretical advances in MARL avoid the challenge of non-stationarity by
coordinating the policy updates of agents in various ways, including
synchronizing times at which agents are allowed to revise their policies.
Synchronization enables analysis of many MARL algorithms via multi-timescale
methods, but such synchrony is infeasible in many decentralized applications.
In this paper, we study an asynchronous variant of the decentralized Q-learning
algorithm, a recent MARL algorithm for stochastic games. We provide sufficient
conditions under which the asynchronous algorithm drives play to equilibrium
with high probability. Our solution utilizes constant learning rates in the
Q-factor update, which we show to be critical for relaxing the synchrony
assumptions of earlier work. Our analysis also applies to asynchronous
generalizations of a number of other algorithms from the regret testing
tradition, whose performance is analyzed by multi-timescale methods that study
Markov chains obtained via policy update dynamics. This work extends the
applicability of the decentralized Q-learning algorithm and its relatives to
settings in which parameters are selected in an independent manner, and tames
non-stationarity without imposing the coordination assumptions of prior work.