On-Robot Bayesian Reinforcement Learning for POMDPs
Hai Nguyen Sammie Katt Yuchen Xiao Christopher Amato
Robot learning is often difficult due to the expense of gathering data. The
need for large amounts of data can, and should, be tackled with effective
algorithms and leveraging expert information on robot dynamics. Bayesian
reinforcement learning (BRL), thanks to its sample efficiency and ability to
exploit prior knowledge, is uniquely positioned as such a solution method.
Unfortunately, the application of BRL has been limited due to the difficulties
of representing expert knowledge as well as solving the subsequent inference
problem. This paper advances BRL for robotics by proposing a specialized
framework for physical systems. In particular, we capture this knowledge in a
factored representation, then demonstrate the posterior factorizes in a similar
shape, and ultimately formalize the model in a Bayesian framework. We then
introduce a sample-based online solution method, based on Monte-Carlo tree
search and particle filtering, specialized to solve the resulting model. This
approach can, for example, utilize typical low-level robot simulators and
handle uncertainty over unknown dynamics of the environment. We empirically
demonstrate its efficiency by performing on-robot learning in two human-robot
interaction tasks with uncertainty about human behavior, achieving near-optimal
performance after only a handful of real-world episodes. A video of learned
policies is at https://youtu.be/H9xp60ngOes.