Markov Decision Processes with Noisy State Observation
Amirhossein Afsharrad Sanjay Lall
This paper addresses the challenge of a particular class of noisy state
observations in Markov Decision Processes (MDPs), a common issue in various
real-world applications. We focus on modeling this uncertainty through a
confusion matrix that captures the probabilities of misidentifying the true
state. Our primary goal is to estimate the inherent measurement noise, and to
this end, we propose two novel algorithmic approaches. The first, the method of
second-order repetitive actions, is designed for efficient noise estimation
within a finite time window, providing identifiable conditions for system
analysis. The second approach comprises a family of Bayesian algorithms, which
we thoroughly analyze and compare in terms of performance and limitations. We
substantiate our theoretical findings with simulations, demonstrating the
effectiveness of our methods in different scenarios, particularly highlighting
their behavior in environments with varying stationary distributions. Our work
advances the understanding of reinforcement learning in noisy environments,
offering robust techniques for more accurate state estimation in MDPs.