ODICE: Revealing the Mystery of Distribution Correction Estimation via
  Orthogonal-gradient Update
Liyuan Mao Haoran Xu Weinan Zhang Xianyuan Zhan
In this study, we investigate the DIstribution Correction Estimation (DICE)
methods, an important line of work in offline reinforcement learning (RL) and
imitation learning (IL). DICE-based methods impose state-action-level behavior
constraint, which is an ideal choice for offline learning. However, they
typically perform much worse than current state-of-the-art (SOTA) methods that
solely use action-level behavior constraint. After revisiting DICE-based
methods, we find there exist two gradient terms when learning the value
function using true-gradient update: forward gradient (taken on the current
state) and backward gradient (taken on the next state). Using forward gradient
bears a large similarity to many offline RL methods, and thus can be regarded
as applying action-level constraint. However, directly adding the backward
gradient may degenerate or cancel out its effect if these two gradients have
conflicting directions. To resolve this issue, we propose a simple yet
effective modification that projects the backward gradient onto the normal
plane of the forward gradient, resulting in an orthogonal-gradient update, a
new learning rule for DICE-based methods. We conduct thorough theoretical
analyses and find that the projected backward gradient brings state-level
behavior regularization, which reveals the mystery of DICE-based methods: the
value learning objective does try to impose state-action-level constraint, but
needs to be used in a corrected way. Through toy examples and extensive
experiments on complex offline RL and IL tasks, we demonstrate that DICE-based
methods using orthogonal-gradient updates (O-DICE) achieve SOTA performance and
great robustness.