Align Your Intents: Offline Imitation Learning via Optimal Transport
Maksim Bobrin Nazar Buzun Dmitrii Krylov Dmitry V. Dylov
Offline reinforcement learning (RL) addresses the problem of sequential
decision-making by learning optimal policy through pre-collected data, without
interacting with the environment. As yet, it has remained somewhat impractical,
because one rarely knows the reward explicitly and it is hard to distill it
retrospectively. Here, we show that an imitating agent can still learn the
desired behavior merely from observing the expert, despite the absence of
explicit rewards or action labels. In our method, AILOT (Aligned Imitation
Learning via Optimal Transport), we involve special representation of states in
a form of intents that incorporate pairwise spatial distances within the data.
Given such representations, we define intrinsic reward function via optimal
transport distance between the expert's and the agent's trajectories. We report
that AILOT outperforms state-of-the art offline imitation learning algorithms
on D4RL benchmarks and improves the performance of other offline RL algorithms
in the sparse-reward tasks.