A Note on Stability in Asynchronous Stochastic Approximation without
  Communication Delays
Huizhen Yu Yi Wan Richard S. Sutton
In this paper, we study asynchronous stochastic approximation algorithms
without communication delays. Our main contribution is a stability proof for
these algorithms that extends a method of Borkar and Meyn by accommodating more
general noise conditions. We also derive convergence results from this
stability result and discuss their application in important average-reward
reinforcement learning problems.