Adversarial Imitation Learning On Aggregated Data
Pierre Le Pelletier de Woillemont Rémi Labory Vincent Corruble
Inverse Reinforcement Learning (IRL) learns an optimal policy, given some
expert demonstrations, thus avoiding the need for the tedious process of
specifying a suitable reward function. However, current methods are constrained
by at least one of the following requirements. The first one is the need to
fully solve a forward Reinforcement Learning (RL) problem in the inner loop of
the algorithm, which might be prohibitively expensive in many complex
environments. The second one is the need for full trajectories from the
experts, which might not be easily available. The third one is the assumption
that the expert data is homogeneous rather than a collection from various
experts or possibly alternative solutions to the same task. Such constraints
make IRL approaches either not scalable or not usable on certain existing
systems. In this work we propose an approach which removes these requirements
through a dynamic, adaptive method called Adversarial Imitation Learning on
Aggregated Data (AILAD). It learns conjointly both a non linear reward function
and the associated optimal policy using an adversarial framework. The reward
learner only uses aggregated data. Moreover, it generates diverse behaviors
producing a distribution over the aggregated data matching that of the experts.