Multi-Agent Generative Adversarial Interactive Self-Imitation Learning
  for AUV Formation Control and Obstacle Avoidance
Zheng Fang Tianhao Chen Dong Jiang Zheng Zhang Guangliang Li
Multiple autonomous underwater vehicles (multi-AUV) can cooperatively
accomplish tasks that a single AUV cannot complete. Recently, multi-agent
reinforcement learning has been introduced to control of multi-AUV. However,
designing efficient reward functions for various tasks of multi-AUV control is
difficult or even impractical. Multi-agent generative adversarial imitation
learning (MAGAIL) allows multi-AUV to learn from expert demonstration instead
of pre-defined reward functions, but suffers from the deficiency of requiring
optimal demonstrations and not surpassing provided expert demonstrations. This
paper builds upon the MAGAIL algorithm by proposing multi-agent generative
adversarial interactive self-imitation learning (MAGAISIL), which can
facilitate AUVs to learn policies by gradually replacing the provided
sub-optimal demonstrations with self-generated good trajectories selected by a
human trainer. Our experimental results in a multi-AUV formation control and
obstacle avoidance task on the Gazebo platform with AUV simulator of our lab
show that AUVs trained via MAGAISIL can surpass the provided sub-optimal expert
demonstrations and reach a performance close to or even better than MAGAIL with
optimal demonstrations. Further results indicate that AUVs' policies trained
via MAGAISIL can adapt to complex and different tasks as well as MAGAIL
learning from optimal demonstrations.