Convergence of SARSA with linear function approximation: The random
  horizon case
Lina Palmborg
The reinforcement learning algorithm SARSA combined with linear function
approximation has been shown to converge for infinite horizon discounted Markov
decision problems (MDPs). In this paper, we investigate the convergence of the
algorithm for random horizon MDPs, which has not previously been shown. We
show, similar to earlier results for infinite horizon discounted MDPs, that if
the behaviour policy is $\varepsilon$-soft and Lipschitz continuous with
respect to the weight vector of the linear function approximation, with small
enough Lipschitz constant, then the algorithm will converge with probability
one when considering a random horizon MDP.