Expert Proximity as Surrogate Rewards for Single Demonstration Imitation
  Learning
Chia-Cheng Chiang Li-Cheng Lan Wei-Fang Sun Chien Feng Cho-Jui Hsieh Chun-Yi Lee
In this paper, we focus on single-demonstration imitation learning (IL), a
practical approach for real-world applications where obtaining numerous expert
demonstrations is costly or infeasible. In contrast to typical IL settings with
multiple demonstrations, single-demonstration IL involves an agent having
access to only one expert trajectory. We highlight the issue of sparse reward
signals in this setting and propose to mitigate this issue through our proposed
Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed
to address reward sparsity by introducing a denser surrogate reward function
that considers environmental dynamics. This surrogate reward function
encourages the agent to navigate towards states that are proximal to expert
states. In practice, TDIL trains a transition discriminator to differentiate
between valid and non-valid transitions in a given environment to compute the
surrogate rewards. The experiments demonstrate that TDIL outperforms existing
IL approaches and achieves expert-level performance in the single-demonstration
IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit
Door" environment.