Bayesian Risk-Averse Q-Learning with Streaming Observations
Yuhao Wang Enlu Zhou
We consider a robust reinforcement learning problem, where a learning agent
learns from a simulated training environment. To account for the model
mis-specification between this training environment and the real environment
due to lack of data, we adopt a formulation of Bayesian risk MDP (BRMDP) with
infinite horizon, which uses Bayesian posterior to estimate the transition
model and impose a risk functional to account for the model uncertainty.
Observations from the real environment that is out of the agent's control
arrive periodically and are utilized by the agent to update the Bayesian
posterior to reduce model uncertainty. We theoretically demonstrate that BRMDP
balances the trade-off between robustness and conservativeness, and we further
develop a multi-stage Bayesian risk-averse Q-learning algorithm to solve BRMDP
with streaming observations from real environment. The proposed algorithm
learns a risk-averse yet optimal policy that depends on the availability of
real-world observations. We provide a theoretical guarantee of strong
convergence for the proposed algorithm.