Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable
  Environments
Xiong-Hui Chen Junyin Ye Hang Zhao Yi-Chen Li Haoran Shi Yu-Yan Xu Zhihao Ye Si-Hang Yang Anqi Huang Kai Xu Zongzhang Zhang Yang Yu
Imitation learning (IL) enables agents to mimic expert behaviors. Most
previous IL techniques focus on precisely imitating one policy through mass
demonstrations. However, in many applications, what humans require is the
ability to perform various tasks directly through a few demonstrations of
corresponding tasks, where the agent would meet many unexpected changes when
deployed. In this scenario, the agent is expected to not only imitate the
demonstration but also adapt to unforeseen environmental changes.
  This motivates us to propose a new topic called imitator learning (ItorL),
which aims to derive an imitator module that can on-the-fly reconstruct the
imitation policies based on very limited expert demonstrations for different
unseen tasks, without any extra adjustment. In this work, we focus on imitator
learning based on only one expert demonstration. To solve ItorL, we propose
Demo-Attention Actor-Critic (DAAC), which integrates IL into a
reinforcement-learning paradigm that can regularize policies' behaviors in
unexpected situations. Besides, for autonomous imitation policy building, we
design a demonstration-based attention architecture for imitator policy that
can effectively output imitated actions by adaptively tracing the suitable
states in demonstrations. We develop a new navigation benchmark and a robot
environment for \topic~and show that DAAC~outperforms previous imitation
methods \textit{with large margins} both on seen and unseen tasks.