Offline Imitation Learning by Controlling the Effective Planning Horizon
Hee-Jun Ahn Seong-Woong Shim Byung-Jun Lee
In offline imitation learning (IL), we generally assume only a handful of
expert trajectories and a supplementary offline dataset from suboptimal
behaviors to learn the expert policy. While it is now common to minimize the
divergence between state-action visitation distributions so that the agent also
considers the future consequences of an action, a sampling error in an offline
dataset may lead to erroneous estimates of state-action visitations in the
offline case. In this paper, we investigate the effect of controlling the
effective planning horizon (i.e., reducing the discount factor) as opposed to
imposing an explicit regularizer, as previously studied. Unfortunately, it
turns out that the existing algorithms suffer from magnified approximation
errors when the effective planning horizon is shortened, which results in a
significant degradation in performance. We analyze the main cause of the
problem and provide the right remedies to correct the algorithm. We show that
the corrected algorithm improves on popular imitation learning benchmarks by
controlling the effective planning horizon rather than an explicit
regularization.