Exploring Gradient Explosion in Generative Adversarial Imitation
  Learning: A Probabilistic Perspective
Wanying Wang Yichen Zhu Yirui Zhou Chaomin Shen Jian Tang Zhiyuan Xu Yaxin Peng Yangchun Zhang
Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone
approach in imitation learning. This paper investigates the gradient explosion
in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with
stochastic policy (ST-GAIL). We begin with the observation that the training
can be highly unstable for DE-GAIL at the beginning of the training phase and
end up divergence. Conversely, the ST-GAIL training trajectory remains
consistent, reliably converging. To shed light on these disparities, we provide
an explanation from a theoretical perspective. By establishing a probabilistic
lower bound for GAIL, we demonstrate that gradient explosion is an inevitable
outcome for DE-GAIL due to occasionally large expert-imitator policy disparity,
whereas ST-GAIL does not have the issue with it. To substantiate our assertion,
we illustrate how modifications in the reward function can mitigate the
gradient explosion challenge. Finally, we propose CREDO, a simple yet effective
strategy that clips the reward function during the training phase, allowing the
GAIL to enjoy high data efficiency and stable trainability.