Nonprehensile Planar Manipulation through Reinforcement Learning with
  Multimodal Categorical Exploration
Juan Del Aguila Ferrandis João Moura Sethu Vijayakumar
Developing robot controllers capable of achieving dexterous nonprehensile
manipulation, such as pushing an object on a table, is challenging. The
underactuated and hybrid-dynamics nature of the problem, further complicated by
the uncertainty resulting from the frictional interactions, requires
sophisticated control behaviors. Reinforcement Learning (RL) is a powerful
framework for developing such robot controllers. However, previous RL
literature addressing the nonprehensile pushing task achieves low accuracy,
non-smooth trajectories, and only simple motions, i.e. without rotation of the
manipulated object. We conjecture that previously used unimodal exploration
strategies fail to capture the inherent hybrid-dynamics of the task, arising
from the different possible contact interaction modes between the robot and the
object, such as sticking, sliding, and separation. In this work, we propose a
multimodal exploration approach through categorical distributions, which
enables us to train planar pushing RL policies for arbitrary starting and
target object poses, i.e. positions and orientations, and with improved
accuracy. We show that the learned policies are robust to external disturbances
and observation noise, and scale to tasks with multiple pushers. Furthermore,
we validate the transferability of the learned policies, trained entirely in
simulation, to a physical robot hardware using the KUKA iiwa robot arm. See our
supplemental video: https://youtu.be/vTdva1mgrk4.