Accelerating Inverse Reinforcement Learning with Expert Bootstrapping
David Wu Sanjiban Choudhury
Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL)
search over candidate reward functions and solve a reinforcement learning
problem in the inner loop. This creates a rather strange inversion where a
harder problem, reinforcement learning, is in the inner loop of a presumably
easier problem, imitation learning. In this work, we show that better
utilization of expert demonstrations can reduce the need for hard exploration
in the inner RL loop, hence accelerating learning. Specifically, we propose two
simple recipes: (1) placing expert transitions into the replay buffer of the
inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner
about high reward states instead of forcing the learner to discover them
through extensive exploration, and (2) using expert actions in Q value
bootstrapping in order to improve the target Q value estimates and more
accurately describe high value expert states. Our methods show significant
gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks,
speeding up recovery to 70\% of deterministic expert performance by 2.13x on
HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.