POMDP inference and robust solution via deep reinforcement learning: An
  application to railway optimal maintenance
Giacomo Arcieri Cyprien Hoelzl Oliver Schwery Daniel Straub Konstantinos G. Papakonstantinou Eleni Chatzi
Partially Observable Markov Decision Processes (POMDPs) can model complex
sequential decision-making problems under stochastic and uncertain
environments. A main reason hindering their broad adoption in real-world
applications is the lack of availability of a suitable POMDP model or a
simulator thereof. Available solution algorithms, such as Reinforcement
Learning (RL), require the knowledge of the transition dynamics and the
observation generating process, which are often unknown and non-trivial to
infer. In this work, we propose a combined framework for inference and robust
solution of POMDPs via deep RL. First, all transition and observation model
parameters are jointly inferred via Markov Chain Monte Carlo sampling of a
hidden Markov model, which is conditioned on actions, in order to recover full
posterior distributions from the available data. The POMDP with uncertain
parameters is then solved via deep RL techniques with the parameter
distributions incorporated into the solution via domain randomization, in order
to develop solutions that are robust to model uncertainty. As a further
contribution, we compare the use of transformers and long short-term memory
networks, which constitute model-free RL solutions, with a
model-based/model-free hybrid approach. We apply these methods to the
real-world problem of optimal maintenance planning for railway assets.