Active Exploration in Bayesian Model-based Reinforcement Learning for
  Robot Manipulation
Carlos Plou Ana C. Murillo Ruben Martinez-Cantin
Efficiently tackling multiple tasks within complex environment, such as those
found in robot manipulation, remains an ongoing challenge in robotics and an
opportunity for data-driven solutions, such as reinforcement learning (RL).
Model-based RL, by building a dynamic model of the robot, enables data reuse
and transfer learning between tasks with the same robot and similar
environment. Furthermore, data gathering in robotics is expensive and we must
rely on data efficient approaches such as model-based RL, where policy learning
is mostly conducted on cheaper simulations based on the learned model.
Therefore, the quality of the model is fundamental for the performance of the
posterior tasks. In this work, we focus on improving the quality of the model
and maintaining the data efficiency by performing active learning of the
dynamic model during a preliminary exploration phase based on maximize
information gathering. We employ Bayesian neural network models to represent,
in a probabilistic way, both the belief and information encoded in the dynamic
model during exploration. With our presented strategies we manage to actively
estimate the novelty of each transition, using this as the exploration reward.
In this work, we compare several Bayesian inference methods for neural
networks, some of which have never been used in a robotics context, and
evaluate them in a realistic robot manipulation setup. Our experiments show the
advantages of our Bayesian model-based RL approach, with similar quality in the
results than relevant alternatives with much lower requirements regarding robot
execution steps. Unlike related previous studies that focused the validation
solely on toy problems, our research takes a step towards more realistic
setups, tackling robotic arm end-tasks.