Approximate information state based convergence analysis of recurrent
  Q-learning
Erfan Seyedsalehi Nima Akbarzadeh Amit Sinha Aditya Mahajan
In spite of the large literature on reinforcement learning (RL) algorithms
for partially observable Markov decision processes (POMDPs), a complete
theoretical understanding is still lacking. In a partially observable setting,
the history of data available to the agent increases over time so most
practical algorithms either truncate the history to a finite window or compress
it using a recurrent neural network leading to an agent state that is
non-Markovian. In this paper, it is shown that in spite of the lack of the
Markov property, recurrent Q-learning (RQL) converges in the tabular setting.
Moreover, it is shown that the quality of the converged limit depends on the
quality of the representation which is quantified in terms of what is known as
an approximate information state (AIS). Based on this characterization of the
approximation error, a variant of RQL with AIS losses is presented. This
variant performs better than a strong baseline for RQL that does not use AIS
losses. It is demonstrated that there is a strong correlation between the
performance of RQL over time and the loss associated with the AIS
representation.