Rethinking Adversarial Inverse Reinforcement Learning: From the Angles
  of Policy Imitation and Transferable Reward Recovery
Yangchun Zhang Yirui Zhou
Adversarial inverse reinforcement learning (AIRL) stands as a cornerstone
approach in imitation learning. This paper rethinks the two different angles of
AIRL: policy imitation and transferable reward recovery. We begin with
substituting the built-in algorithm in AIRL with soft actor-critic (SAC) during
the policy optimization process to enhance sample efficiency, thanks to the
off-policy formulation of SAC and identifiable Markov decision process (MDP)
models with respect to AIRL. It indeed exhibits a significant improvement in
policy imitation but accidentally brings drawbacks to transferable reward
recovery. To learn this issue, we illustrate that the SAC algorithm itself is
not feasible to disentangle the reward function comprehensively during the AIRL
training process, and propose a hybrid framework, PPO-AIRL + SAC, for
satisfactory transfer effect. Additionally, we analyze the capability of
environments to extract disentangled rewards from an algebraic theory
perspective.